Skip to content

Commit 1ebbe64

Browse files
mergify[bot]gizas
mergify[bot]
and
gizas
authored
[8.x](backport #40301) Disable kubeadm for filebeat (both filbeat.input and filbeat.autodiscover) (#40889)
* Disable kubeadm for filebeat (both filbeat.input and filbeat.autodiscover) (#40301) * disable kubeadm for filebeat autodiscover * Update filebeat/_meta/test/docs/01_playground/filebeat.yaml Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co> * Update libbeat/processors/add_kubernetes_metadata/docs/add_kubernetes_metadata.asciidoc Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co> * Update dev-tools/kubernetes/filebeat/Dockerfile.debug Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co> * fixing docs and adding mericbeat references * fixing docs and adding mericbeat references * Update filebeat/_meta/config/filebeat.autodiscover.reference.yml.tmpl Co-authored-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> --------- Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co> Co-authored-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> (cherry picked from commit 89cd840) # Conflicts: # metricbeat/module/kubernetes/util/kubernetes.go * fixing worng backport comments --------- Co-authored-by: Andrew Gizas <andreas.gkizas@elastic.co>
1 parent ae3fa91 commit 1ebbe64

File tree

18 files changed

+182
-8
lines changed

18 files changed

+182
-8
lines changed

‎CHANGELOG.next.asciidoc

+1
Original file line numberDiff line numberDiff line change
@@ -302,6 +302,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
302302
- Disable event normalization for netflow input {pull}40635[40635]
303303
- Allow attribute selection in the Active Directory entity analytics provider. {issue}40482[40482] {pull}40662[40662]
304304
- Improve error quality when CEL program does not correctly return an events array. {pull}40580[40580]
305+
- Add `use_kubeadm` config option for filebeat (both filbeat.input and autodiscovery) in order to toggle kubeadm-config api requests {pull}40301[40301]
305306

306307
*Auditbeat*
307308

‎dev-tools/kubernetes/filebeat/Dockerfile.debug

+1-1
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ ENV PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/go/bin:/usr/local/go/bin
44

55
ENV CGO_ENABLED=0
66

7-
RUN go install github.com/go-delve/delve/cmd/dlv@v1.21.2
7+
RUN go install github.com/go-delve/delve/cmd/dlv@latest
88

99
COPY build/filebeat-debugger /usr/share/filebeat/filebeat-debugger
1010

‎dev-tools/kubernetes/heartbeat/Dockerfile.debug

+1-1
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ ENV PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/go/bin:/usr/local/go/bin
44

55
ENV CGO_ENABLED=0
66

7-
RUN go install github.com/go-delve/delve/cmd/dlv@v1.21.2
7+
RUN go install github.com/go-delve/delve/cmd/dlv@latest
88

99
COPY build/heartbeat-debugger /usr/share/heartbeat/heartbeat-debugger
1010

‎dev-tools/kubernetes/metricbeat/Dockerfile.debug

+1-1
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ ENV PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/go/bin:/usr/local/go/bin
44

55
ENV CGO_ENABLED=0
66

7-
RUN go install github.com/go-delve/delve/cmd/dlv@v1.21.2
7+
RUN go install github.com/go-delve/delve/cmd/dlv@latest
88

99
COPY build/metricbeat-debugger /usr/share/metricbeat/metricbeat-debugger
1010

‎filebeat/_meta/config/filebeat.autodiscover.reference.yml.tmpl

+24
Original file line numberDiff line numberDiff line change
@@ -14,3 +14,27 @@
1414
# - type: container
1515
# paths:
1616
# - /var/log/containers/*.log
17+
18+
#Example: for kubernetes container logs autodiscovery
19+
# filebeat.autodiscover:
20+
# providers:
21+
# - type: kubernetes
22+
# node: ${NODE_NAME}
23+
# hints.enabled: true
24+
# # By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
25+
# use_kubeadm: true
26+
# hints.default_config:
27+
# type: filestream
28+
# id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
29+
# paths:
30+
# - /var/log/containers/*-${data.kubernetes.container.id}.log
31+
# parsers:
32+
# - container: ~
33+
# prospector:
34+
# scanner:
35+
# fingerprint.enabled: true
36+
# symlinks: true
37+
# file_identity.fingerprint: ~
38+
39+
#By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
40+
# use_kubeadm: true

‎filebeat/_meta/test/docs/01_playground/filebeat.yaml

+23-5
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,20 @@ metadata:
99
data:
1010
filebeat.yml: |-
1111
filebeat.inputs:
12-
- type: container
12+
- type: filestream
13+
id: kubernetes-container-logs
1314
paths:
1415
- /var/log/containers/*.log
16+
parsers:
17+
- container: ~
18+
prospector:
19+
scanner:
20+
fingerprint.enabled: true
21+
symlinks: true
22+
file_identity.fingerprint: ~
1523
processors:
1624
- add_kubernetes_metadata:
25+
#use_kubeadm: true
1726
host: ${NODE_NAME}
1827
matchers:
1928
- logs_path:
@@ -25,10 +34,19 @@ data:
2534
# - type: kubernetes
2635
# node: ${NODE_NAME}
2736
# hints.enabled: true
37+
# #use_kubeadm: true
2838
# hints.default_config:
29-
# type: container
39+
# type: filestream
40+
# id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
3041
# paths:
31-
# - /var/log/containers/*${data.kubernetes.container.id}.log
42+
# - /var/log/containers/*-${data.kubernetes.container.id}.log
43+
# parsers:
44+
# - container: ~
45+
# prospector:
46+
# scanner:
47+
# fingerprint.enabled: true
48+
# symlinks: true
49+
# file_identity.fingerprint: ~
3250
3351
processors:
3452
- add_cloud_metadata:
@@ -64,12 +82,12 @@ spec:
6482
dnsPolicy: ClusterFirstWithHostNet
6583
containers:
6684
- name: filebeat
67-
image: docker.elastic.co/beats/filebeat:8.0.0-SNAPSHOT
85+
image: docker.elastic.co/beats/filebeat:8.16.0-SNAPSHOT
6886
command: [ "sleep" ]
6987
args: [ "infinity" ]
7088
env:
7189
- name: ELASTICSEARCH_HOST
72-
value: elasticsearch.default.svc.cluster.local
90+
value: elasticsearch
7391
- name: ELASTICSEARCH_PORT
7492
value: "9200"
7593
- name: ELASTICSEARCH_USERNAME

‎filebeat/filebeat.reference.yml

+24
Original file line numberDiff line numberDiff line change
@@ -1219,6 +1219,30 @@ filebeat.inputs:
12191219
# paths:
12201220
# - /var/log/containers/*.log
12211221

1222+
#Example: for kubernetes container logs autodiscovery
1223+
# filebeat.autodiscover:
1224+
# providers:
1225+
# - type: kubernetes
1226+
# node: ${NODE_NAME}
1227+
# hints.enabled: true
1228+
# # By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
1229+
# use_kubeadm: true
1230+
# hints.default_config:
1231+
# type: filestream
1232+
# id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
1233+
# paths:
1234+
# - /var/log/containers/*-${data.kubernetes.container.id}.log
1235+
# parsers:
1236+
# - container: ~
1237+
# prospector:
1238+
# scanner:
1239+
# fingerprint.enabled: true
1240+
# symlinks: true
1241+
# file_identity.fingerprint: ~
1242+
1243+
#By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
1244+
# use_kubeadm: true
1245+
12221246
# ========================== Filebeat global options ===========================
12231247

12241248
# Registry data path. If a relative path is used, it is considered relative to the

‎libbeat/autodiscover/providers/kubernetes/config.go

+2
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ var AllSupportedHints = []string{"enabled", "module", "metricsets", "hosts", "pe
3737
type Config struct {
3838
KubeConfig string `config:"kube_config"`
3939
KubeClientOptions kubernetes.KubeClientOptions `config:"kube_client_options"`
40+
KubeAdm bool `config:"use_kubeadm"`
4041

4142
Namespace string `config:"namespace"`
4243
SyncPeriod time.Duration `config:"sync_period"`
@@ -71,6 +72,7 @@ var DefaultCleanupTimeout time.Duration = 0
7172
func defaultConfig() *Config {
7273
return &Config{
7374
SyncPeriod: 10 * time.Minute,
75+
KubeAdm: true,
7476
Resource: "pod",
7577
CleanupTimeout: DefaultCleanupTimeout,
7678
Prefix: "co.elastic",

‎libbeat/autodiscover/providers/kubernetes/pod.go

+9
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,15 @@ func NewPodEventer(uuid uuid.UUID, cfg *conf.C, client k8s.Interface, publish fu
9797
}
9898

9999
metaConf := config.AddResourceMetadata
100+
// We initialise the use_kubeadm variable based on modules KubeAdm base configuration
101+
err = metaConf.Namespace.SetBool("use_kubeadm", -1, config.KubeAdm)
102+
if err != nil {
103+
logger.Errorf("couldn't set kubeadm variable for namespace due to error %+v", err)
104+
}
105+
err = metaConf.Node.SetBool("use_kubeadm", -1, config.KubeAdm)
106+
if err != nil {
107+
logger.Errorf("couldn't set kubeadm variable for node due to error %+v", err)
108+
}
100109

101110
if metaConf.Node.Enabled() || config.Hints.Enabled() {
102111
options := kubernetes.WatchOptions{

‎libbeat/autodiscover/providers/kubernetes/service.go

+5
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,11 @@ func NewServiceEventer(uuid uuid.UUID, cfg *conf.C, client k8s.Interface, publis
7171
var namespaceWatcher kubernetes.Watcher
7272

7373
metaConf := config.AddResourceMetadata
74+
// We initialise the use_kubeadm variable based on modules KubeAdm base configuration
75+
err = metaConf.Namespace.SetBool("use_kubeadm", -1, config.KubeAdm)
76+
if err != nil {
77+
logger.Errorf("couldn't set kubeadm variable for namespace due to error %+v", err)
78+
}
7479

7580
if metaConf.Namespace.Enabled() || config.Hints.Enabled() {
7681
namespaceWatcher, err = kubernetes.NewNamedWatcher("namespace", client, &kubernetes.Namespace{}, kubernetes.WatchOptions{

‎libbeat/processors/add_kubernetes_metadata/config.go

+1
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ import (
2929
type kubeAnnotatorConfig struct {
3030
KubeConfig string `config:"kube_config"`
3131
KubeClientOptions kubernetes.KubeClientOptions `config:"kube_client_options"`
32+
KubeAdm bool `config:"use_kubeadm"`
3233
Node string `config:"node"`
3334
Scope string `config:"scope"`
3435
Namespace string `config:"namespace"`

‎libbeat/processors/add_kubernetes_metadata/docs/add_kubernetes_metadata.asciidoc

+1
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,7 @@ Example:
152152
-------------------------------------------------------------------------------------
153153
`kube_config`:: (Optional) Use given config file as configuration for Kubernetes
154154
client. It defaults to `KUBECONFIG` environment variable if present.
155+
`use_kubeadm`:: (Optional) Default true. By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
155156
`kube_client_options`:: (Optional) Additional options can be configured for Kubernetes
156157
client. Currently client QPS and burst are supported, if not set Kubernetes client's
157158
https://pkg.go.dev/k8s.io/client-go/rest#pkg-constants[default QPS and burst] will be used.

‎libbeat/processors/add_kubernetes_metadata/kubernetes.go

+9
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,15 @@ func (k *kubernetesAnnotator) init(config kubeAnnotatorConfig, cfg *config.C) {
147147
k.initOnce.Do(func() {
148148
var replicaSetWatcher, jobWatcher, namespaceWatcher, nodeWatcher kubernetes.Watcher
149149

150+
// We initialise the use_kubeadm variable based on modules KubeAdm base configuration
151+
err := config.AddResourceMetadata.Namespace.SetBool("use_kubeadm", -1, config.KubeAdm)
152+
if err != nil {
153+
k.log.Errorf("couldn't set kubeadm variable for namespace due to error %+v", err)
154+
}
155+
err = config.AddResourceMetadata.Node.SetBool("use_kubeadm", -1, config.KubeAdm)
156+
if err != nil {
157+
k.log.Errorf("couldn't set kubeadm variable for node due to error %+v", err)
158+
}
150159
client, err := kubernetes.GetKubernetesClient(config.KubeConfig, config.KubeClientOptions)
151160
if err != nil {
152161
if kubernetes.IsInCluster(config.KubeConfig) {

‎metricbeat/docs/modules/kubernetes.asciidoc

+14
Original file line numberDiff line numberDiff line change
@@ -235,6 +235,8 @@ metricbeat.modules:
235235
# If kube_config is not set, KUBECONFIG environment variable will be checked
236236
# and if not present it will fall back to InCluster
237237
#kube_config: ~/.kube/config
238+
#By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
239+
use_kubeadm: true
238240
#include_labels: []
239241
#exclude_labels: []
240242
#include_annotations: []
@@ -286,6 +288,8 @@ metricbeat.modules:
286288
# If kube_config is not set, KUBECONFIG environment variable will be checked
287289
# and if not present it will fall back to InCluster
288290
#kube_config: ~/.kube/config
291+
#By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
292+
use_kubeadm: true
289293
#include_labels: []
290294
#exclude_labels: []
291295
#include_annotations: []
@@ -326,6 +330,8 @@ metricbeat.modules:
326330
# If kube_config is not set, KUBECONFIG environment variable will be checked
327331
# and if not present it will fall back to InCluster
328332
#kube_config: ~/.kube/config
333+
#By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
334+
use_kubeadm: true
329335
# Set the namespace to watch for events
330336
#namespace: staging
331337
# Set the sync period of the watchers
@@ -346,6 +352,8 @@ metricbeat.modules:
346352
ssl.certificate_authorities:
347353
- /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
348354
period: 30s
355+
#By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
356+
use_kubeadm: true
349357
350358
# Kubernetes proxy server
351359
# (when running metricbeat locally at hosts or as a daemonset + host network)
@@ -355,6 +363,8 @@ metricbeat.modules:
355363
- proxy
356364
hosts: ["localhost:10249"]
357365
period: 10s
366+
#By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
367+
use_kubeadm: true
358368
359369
# Kubernetes controller manager
360370
# (URL and deployment method should be adapted to match the controller manager deployment / service / endpoint)
@@ -364,6 +374,8 @@ metricbeat.modules:
364374
- controllermanager
365375
hosts: ["http://localhost:10252"]
366376
period: 10s
377+
#By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
378+
use_kubeadm: true
367379
368380
# Kubernetes scheduler
369381
# (URL and deployment method should be adapted to match scheduler deployment / service / endpoint)
@@ -373,6 +385,8 @@ metricbeat.modules:
373385
- scheduler
374386
hosts: ["localhost:10251"]
375387
period: 10s
388+
#By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
389+
use_kubeadm: true
376390
----
377391

378392
This module supports TLS connections when using `ssl` config field, as described in <<configuration-ssl>>.

‎metricbeat/metricbeat.reference.yml

+14
Original file line numberDiff line numberDiff line change
@@ -522,6 +522,8 @@ metricbeat.modules:
522522
# If kube_config is not set, KUBECONFIG environment variable will be checked
523523
# and if not present it will fall back to InCluster
524524
#kube_config: ~/.kube/config
525+
#By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
526+
use_kubeadm: true
525527
#include_labels: []
526528
#exclude_labels: []
527529
#include_annotations: []
@@ -573,6 +575,8 @@ metricbeat.modules:
573575
# If kube_config is not set, KUBECONFIG environment variable will be checked
574576
# and if not present it will fall back to InCluster
575577
#kube_config: ~/.kube/config
578+
#By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
579+
use_kubeadm: true
576580
#include_labels: []
577581
#exclude_labels: []
578582
#include_annotations: []
@@ -613,6 +617,8 @@ metricbeat.modules:
613617
# If kube_config is not set, KUBECONFIG environment variable will be checked
614618
# and if not present it will fall back to InCluster
615619
#kube_config: ~/.kube/config
620+
#By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
621+
use_kubeadm: true
616622
# Set the namespace to watch for events
617623
#namespace: staging
618624
# Set the sync period of the watchers
@@ -633,6 +639,8 @@ metricbeat.modules:
633639
ssl.certificate_authorities:
634640
- /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
635641
period: 30s
642+
#By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
643+
use_kubeadm: true
636644

637645
# Kubernetes proxy server
638646
# (when running metricbeat locally at hosts or as a daemonset + host network)
@@ -642,6 +650,8 @@ metricbeat.modules:
642650
- proxy
643651
hosts: ["localhost:10249"]
644652
period: 10s
653+
#By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
654+
use_kubeadm: true
645655

646656
# Kubernetes controller manager
647657
# (URL and deployment method should be adapted to match the controller manager deployment / service / endpoint)
@@ -651,6 +661,8 @@ metricbeat.modules:
651661
- controllermanager
652662
hosts: ["http://localhost:10252"]
653663
period: 10s
664+
#By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
665+
use_kubeadm: true
654666

655667
# Kubernetes scheduler
656668
# (URL and deployment method should be adapted to match scheduler deployment / service / endpoint)
@@ -660,6 +672,8 @@ metricbeat.modules:
660672
- scheduler
661673
hosts: ["localhost:10251"]
662674
period: 10s
675+
#By default requests to kubeadm config map are made in order to enrich cluster name by requesting /api/v1/namespaces/kube-system/configmaps/kubeadm-config API endpoint.
676+
use_kubeadm: true
663677

664678
#--------------------------------- KVM Module ---------------------------------
665679
- module: kvm

0 commit comments

Comments
 (0)