Releases: fluent/fluent-bit
Fluent Bit 2.0.2
https://fluentbit.io/announcements/v2.0.2/
This is a hotfix release that solve a corner case in metrics processing.
Fluent Bit 2.0.0
News
Fluent Bit v2.0.0 is the stable release!, new changes on this version:
Logs, Metrics, and Traces
Fluent Bit has always been agnostic of the data that its processes and moves around; one of the major use cases has been around Log Processing. Recently we introduced functionality to integrate Metrics and now with Fluent Bit v2, we are formally announcing the support for Traces.
As a vendor-agnostic solution, Fluent Bit natively integrates with protocols and other ecosystems, so having Metrics and Traces support means that we fully integrate with systems like Prometheus and OpenTelemetry.
Metrics
As part of the metrics collectors (inputs) and outputs supported for Metrics, allow us to collect and deliver metrics in a smooth way, a common example of these components are:
| Input | Description |
|---|---|
| node_exporter_metrics | A plugin that implements a subset of host metrics collection like the known external tool Prometheus Node Exporter. |
| nginx_metrics | Scrape Nginx metrics endpoints. It supports the OSS and Nginx+ enterprise editions. |
| windows_exporter_metrics | Collect system metrics from a Windows system, for it support CPU collection. |
| fluentbit_metrics | Collect internal Fluent Bit metrics and ingest them into the pipeline. |
| Output | Description |
|---|---|
| prometheus_exporter | Expose metrics in an HTTP endpoint in Prometheus text format. This mechanism is commonly used when you want to scrape metrics available by Fluent Bit by other a third part solution like Prometheus Server. |
| prometheus_remote_write | Deliver metrics to a remote endpoint by using the Prometheus Remote Write protocol. |
| splunk | New support for Splunk Metrics (HEC) |
| Influxdb | Send metrics to InfluxDB database |
One of the biggest addition on this are is the support for OpenTelemetry metrics, in the input and output side, more details below.
OpenTelemetry
OpenTelemetry is a framework, spec and protocol to unify the aspects of Telemetry collection and delivery. One of the major use cases in production is for Traces; Metrics and Logs support has been recently added.
In Fluent Bit, we are announcing full support for OpenTelemetry, where now we can receive and send telemetry data by using OpenTelemetry protocol:
OpenTelemetry input plugin
This new plugin, can receive OpenTelemetry Logs, Metrics and Traces, all of them are supported through this new implementation that allow easily integration with applications instrumented with OpenTelemetry SDKs.
OpenTelemetry output plugin
This new plugin allows to deliver any Log, Metric or Trace collected to a remote endpoint that supports OpenTelemetry, it could be any vendor platform or the OpenTelemetry collector.
Performance
The new threaded mechanism allows input plugins to run in a separate thread which helps to desaturate the main pipeline and move certain loads to a different CPU core. This feature must be enabled manually in the configuration of each input plugin by adding the threaded property, e.g:
[INPUT]
name tail
path /var/log/containers/*.log
tag kube.*
threaded on
Full OpenSSL support / deprecate mbedTLS
With this new version, all the network transport layer that needs security enabled will use OpenSSL instead of mbedTLS: networking & crypto functions. From now on, mbedTLS is officially deprecated.
All interfaces and plugins that were using mbedTLS has been ported to use the new crypto and networking layer we have implemented.
TLS for input plugins
Using Fluent Bit as an aggregator is a common use case, but one component was missing: TLS support for ingestion (Transport Layer Security), which for many users it was a big blocker; the same problem has been reported by many Fluentd users looking forward to migrating to Fluent Bit.
In this new version, we are announcing full-native TLS capabilities for input plugins.
Plugins Changes
Loki (output)
In previous versions, the Loki project did not support out-of-order records, for performance reasons, Fluent Bit delivers messages in parallel which could cause exceptions on the transmission since some messages might not arrive in the order expected by Loki; to solve this problem, we used to force our Loki connector to just send one Chunk at a time, so the order was preserved.
Since newer versions of Loki no longer have this restriction, we are removing the restriction too from our side and allowing concurrency again, which is a boost in performance. No changes are needed.
Note: this improvement might be a breaking change if you are using an older version of Loki. If you are using **Loki >= 2.4 ** you are good to go.
To learn more about this Loki restriction and the enhancements, you can refer to the following blog post from Grafana team:
https://grafana.com/blog/2021/12/03/new-feature-in-loki-2.4-no-more-ordering-constraint/
Splunk + Metrics (output)
The current Splunk output connector now supports the delivery of Metrics. No changes in your configuration are needed, if you send metrics records to a Splunk output plugin, the data will be converted and delivered as expected by the Splunk HEC endpoint.
Developers Experience
There are many cases where a user would like to extend the functionality that Fluent Bit provides, usually for specific business reasons.
For customizations, we currently support plugins that can be written in C, filters in Lua, or output plugins in Golang. But understanding our user’s needs and the restrictions of this limited functionality, we are going beyond that.
Fluent Bit v2 comes with support to integrate more plugin types with Golang and WebAssembly. The following table describes the supported languages by plugin type.
| Language | Input / Source | Filter | Output / Destination |
|---|---|---|---|
| C | Yes | Yes | Yes |
| Golang | Yes | —- | Yes |
| WebAssembly | Yes | Yes | —- |
TAP
One of the common questions from our users has been “how can I see the data flowing through a pipeline?”, usually, this comes up when there is a need to perform troubleshooting without stopping the agent.
TAP is a very advanced functionality that allows inspecting the data that flows through a pipeline, it can be enabled on runtime by just using a simple HTTP call.
More details about this functionality can be seen here in the official documentation:
Internal Metrics
Historically, a common way to extract internal metrics from Fluent Bit has been by enabling the built-in HTTP service which exposes the following endpoints in the REST API:
| Endpoint | Description |
|---|---|
| /api/v1/metrics | JSON metrics for records and byte sizes associated. |
| /api/v1/metrics/prometheus | Same metrics as /api/v1/metrics but formatted in Prometheus Text format. |
| /api/v1/storage | JSON metrics for the storage component which expose metrics for Chunks. |
One of the restrictions of this interface, is that those metrics are not part of the pipeline, which means, you can only access them remotely. But what about if you want to send your metrics to a destination like Prometheus, InfluxDB or other ?.
During the release cycle of the previous series of Fluent Bit, we introduced a new mechanism to collect and process internal metrics through a new input plugin plugin called fluentbit_metrics. This new mechanism allows to expose or send the metrics to a destination in many ways like:
- Prometheus Exporter
- Prometheus Remote Write
- InfluxDB
- OpenTelemetry
- Standard output interface (stdout)
New Storage Metrics
As mentioned, the Storage metrics originally were exposed in a different endpoint which forced the user to scrape another endpoint and also faced the restriction that storage metrics were only exposed in JSON.
Starting from Fluent Bit v2, fluentbit_metrics exposes seven new metrics for the Storage layer:
| Metrics name | Description |
|---|---|
fluentbit_input_storage_overlimit |
It takes a value of 1 if the input plugin instance is over the limit imposed by a mem_buf_limit configuration. Otherwise is just set to 0. |
| `fluentbit_input_storage_m... |
Fluent Bit 1.9.9
Fluent Bit 1.9.8
Fluent Bit 1.9.7
Fluent Bit 1.9.6
https://fluentbit.io/announcements/v1.9.6/
What's Changed
- throttle: Fixed use after free by @ptsneves in #4792
- include: fluent-bit: flb_input: fix some typos in comments by @lecaros in #5609
- out_datadog: switch noisy info level to debug level by @patrick-stephens in #5596
- datadog: migrate to v2 API logs endpoint by @truthbk in #4798
- tests: runtime: in_syslog: Add missing FLB_HAVE_UNIX_SOCKET clause by @cosmo0920 in #5647
- out_syslog: fix TLS handling in config and code logic (fix #5643) by @edsiper in #5646
- in_dummy_thread: remove unused vars by @nokute78 in #5642
- filter_throttle: add mutex to prevent race issue by @nokute78 in #5635
- parser: free allocated types by @DavidKorczynski in #5608
- lib: c-ares: upgrade to v1.18.1 by @edsiper in #5663
- filter_multiline: use flb_time API to get timestamp in millisecond by @nokute78 in #5638
- fstore: Suppress log cb warn by @nokute78 in #5667
- in_opentelemetry: endpoints change and new traces serialization by @edsiper in #5678
- exec: fix name of oneshot parameter by @chrisburr in #5127
- build: Fix building error for c-ares library on macOS by @cosmo0920 in #5683
- out_stdout: handle error case by @DavidKorczynski in #5603
- record_accessor: fix possible null-deref by @DavidKorczynski in #5101
- tests: internal: fuzzers: fix index error by @DavidKorczynski in #5602
- input: return error code from cb_collect by @DavidKorczynski in #5605
- filter_expect: support 'result_key' property and add test code by @nokute78 in #5668
- in_netif: add missing return type by @nokute78 in #5639
- ml: clear last recently used parser to match next parser(#5524) by @nokute78 in #5564
- tests: runtime: in_tail: add some test cases by @nokute78 in #5456
- filter_modify: add error check for flb_strndup(#5103) by @nokute78 in #5129
- tests: internal: upstream: add unit test for flb_upstream by @nokute78 in #4756
- out_forward: support 'unix_path' by @nokute78 in #4715
- filter_kubernetes: Add ability to change kubelet_host by @tumd in #5291
- out_cloudwatch_logs: support record_accessor by @PettitWesley in #5633
- s3: update S3 PutObjectSize to 1GB by @zwj102030 in #5688
New Contributors
- @sar009 made their first contribution in #5399
- @Dentrax made their first contribution in #5606
- @truthbk made their first contribution in #4798
- @chrisburr made their first contribution in #5127
- @tumd made their first contribution in #5291
- @zwj102030 made their first contribution in #5688
Full Changelog: v1.9.5...v1.9.6
Contributors
Assets 2
Fluent Bit 1.9.5
Fluent Bit 1.9.4
https://fluentbit.io/announcements/v1.9.4/
What's Changed
- out_azure: fix typo which generates segv by @strima in #5379
- lib: monkey: sync changes for headers path by @edsiper in #5409
- cpack: build: msi: Support MSI installer with CPack WiX generator by @cosmo0920 in #5367
- out_cloudwatch_logs: set 1 worker as default by @PettitWesley in #5417
- in_systemd: fix 'db.sync' property (#5426) by @nokute78 in #5429
- aws: add debug logs to check batch sizes by @PettitWesley in #5428
- tests: runtime: remove warnings by @nokute78 in #5383
- tests: runtime: in_syslog: add test code by @nokute78 in #5388
- in_systemd: Don't insert rows if flb_sqldb_query fails by @danlenar in #5373
- pack: modify to determine how many JSON messages are OK(#5336) by @nokute78 in #5412
- record_accessor: add append API, flb_ra_append_kv_pair by @nokute78 in #5390
- http_client: add 'flb_http_get_header' to get the value of request header by @nokute78 in #5065
- packaging: add options to parameterise server/key for testing by @patrick-stephens in #5444
- out_stackdriver: Initialize the metadata_server value to NULL by @m-deepakraja in #5436
- fstore: ensure the chunk is up while modifying it by @leonardo-albertovich in #5469
- in_opentelemetry: Fixed the event type so OTel metrics by @leonardo-albertovich in #5470
- out_stackdriver: free metadata_server when 'METADATA_SERVER' is not set by @nokute78 in #5453
- time: add strict event time type checking by @nokute78 in #5455
- filter_kubernetes: add option kube_token_ttl (#4352) by @novegit in #4487
- filter_lua: Fix record split in mpack implementation by @tarruda in #5503
- tests: runtime: out_syslog: add test code by @nokute78 in #5510
- in_forward: support unix_perm by @nokute78 in #5511
- out_http: Implement "body_key" and "headers_key" options by @tarruda in #5439
- utils: seek atmark from tail (#5530) by @nokute78 in #5534
New Contributors
- @strima made their first contribution in #5379
- @m-deepakraja made their first contribution in #5436
Full Changelog: v1.9.3...v1.9.4
