v4.15.0

For installation instructions check out the getting started guide.

Added

• Flag --skip-env-var-check added to the lint subcommand, this disables the new linting behaviour where environment variable interpolations without defaults throw linting errors when the variable is not defined.
• The kafka_franz input now supports explicit partitions in the field topics.
• The kafka_franz input now supports batching.
• New metadata Bloblang function for batch-aware structured metadata queries.
• Go API: Running the Benthos CLI with a context set with a deadline now triggers graceful termination before the deadline is reached.
• Go API: New public/service/servicetest package added for functions useful for testing custom Benthos builds.
• New lru and ttlru in-memory caches.

Fixed

• Provide msgpack plugins through public/components/msgpack.
• The kafka_franz input should no longer commit offsets one behind the next during partition yielding.
• The streams mode HTTP API should no longer route requests to /streams/<stream-ID> to the /streams handler. This issue was introduced in v4.14.0.

The full change log can be found here.

v4.14.0

For installation instructions check out the getting started guide.

Added

• The -e/--env-file cli flag can now be specified multiple times.
• New studio pull cli subcommand for running Benthos Studio session deployments.
• Metadata field kafka_tombstone_message added to the kafka and kafka_franz inputs.
• Method SetEnvVarLookupFunc added to the stream builder API.
• The discord input and output now use the official chat client API and no longer rely on poll-based HTTP requests, this should result in more efficient and less erroneous behaviour.
• New bloblang timestamp methods ts_add_iso8601 and ts_sub_iso8601.
• All SQL components now support the trino driver.
• New input codec csv-safe.
• Added base64rawurl scheme to both the encode and decode Bloblang methods.
• New find_by and find_all_by Bloblang methods.
• New skipbom input codec.
• New javascript processor.

Fixed

• The find_all bloblang method no longer produces results that are of an unknown type.
• The find_all and find Bloblang methods no longer fail when the value argument is a field reference.
• Endpoints specified by HTTP server components using both the general http server block or their own custom server addresses should now be treated as path prefixes. This corrects a behavioural change that was introduced when both respective server options were updated to support path parameters.
• Prevented a panic caused when using the encrypt_aes and decrypt_aes Bloblang methods with a mismatched key/iv lengths.
• The snowpipe field of the snowflake_put output can now be omitted from the config without raising an error.
• Batch-aware processors such as mapping and mutation should now report correct error metrics.
• Running benthos blobl server should no longer panic when a mapping with variable read/writes is executed in parallel.
• Speculative fix for the cloudwatch metrics exporter rejecting metrics due to minimum field size of 1, PutMetricDataInput.MetricData[0].Dimensions[0].Value.
• The snowflake_put output now prevents silent failures under certain conditions. Details here.
• Reduced the amount of pre-compilation of Bloblang based linting rules for documentation fields, this should dramatically improve the start up time of Benthos (~1s down to ~200ms).
• Environment variable interpolations with an empty fallback (${FOO:}) are now valid.
• Fixed an issue where the mongodb output wasn't using bulk send requests according to batching policies.
• The amqp_1 input now falls back to accessing Message.Value when the data is empty.

Changed

• When a config contains environment variable interpolations without a default value (i.e. ${FOO}), if that environment variable is not defined a linting error will be emitted. Shutting down due to linting errors can be disabled with the --chilled cli flag, and variables can be specified with an empty default value (${FOO:}) in order to make the previous behaviour explicit and prevent the new linting error.
• The find and find_all Bloblang methods no longer support query arguments as they were incompatible with supporting value arguments. For query based arguments use the new find_by and find_all_by methods.

The full change log can be found here.

v4.13.0

For installation instructions check out the getting started guide.

Added

• New nats_kv processor, input and output.
• Field partition added to the kafka_franz output, allowing for manual partitioning.

Fixed

• The broker output with the pattern fan_out_sequential will no longer abandon in-flight requests that are error blocked until the full shutdown timeout has occurred.
• The broker input no longer reports itself as unavailable when a child input has intentionally closed.
• Config unit tests that check for structured data should no longer fail in all cases.
• The http_server input with a custom address now supports path variables.

The full change log can be found here.

v4.12.1

For installation instructions check out the getting started guide.

Fixed

• Fixed a regression bug in the nats components where panics occur during a flood of messages. This issue was introduced in v4.12.0 (45f785a).

The full change log can be found here.

v4.12.0

For installation instructions check out the getting started guide.

Added

• Format csv:x added to the unarchive processor.
• Field max_buffer added to the aws_s3 input.
• Field open_message_type added to the websocket input.
• The experimental --watcher cli flag now takes into account file deletions and new files that match wildcard patterns.
• Field dump_request_log_level added to HTTP components.
• New couchbase cache implementation.
• New compress and decompress Bloblang methods.
• Field endpoint added to the gcp_pubsub input and output.
• Fields file_name, file_extension and request_id added to the snowflake_put output.
• Add interpolation support to the path field of the snowflake_put output.
• Add ZSTD compression support to the compression field of the snowflake_put output.
• New Bloblang method concat.
• New redis ratelimit.
• The socket_server input now supports tls as a network type.
• New bloblang function timestamp_unix_milli.
• New bloblang method ts_unix_milli.
• JWT based HTTP authentication now supports EdDSA.
• New flow_control fields added to the gcp_pubsub output.
• Added bloblang methods sign_jwt_hs256, sign_jwt_hs384 and sign_jwt_hs512
• New bloblang methods parse_jwt_hs256, parse_jwt_hs384, parse_jwt_hs512.
• The open_telemetry_collector tracer now automatically sets the service.name and service.version tags if they are not configured by the user.
• New bloblang string methods trim_prefix and trim_suffix.

Fixed

• Fixed an issue where messages caught in a retry loop from inputs that do not support nacks (generate, kafka, file, etc) could be retried in their post-mutation form from the switch output rather than the original copy of the message.
• The sqlite buffer should no longer print Failed to ack buffer message logs during graceful termination.
• The default value of the conn_max_idle field has been changed from 0 to 2 for all sql_* components in accordance
  to the database/sql docs.
• The parse_csv bloblang method with parse_header_row set to false no longer produces rows that are of an unknown type.
• Fixed a bug where the oracle driver for the sql_* components was returning timestamps which were getting marshalled into an empty JSON object instead of a string.
• The aws_sqs input no longer backs off on subsequent empty requests when long polling is enabled.
• It's now possible to mock resources within the main test target file in config unit tests.
• Unit test linting no longer incorrectly expects the json_contains predicate to contain a string value only.
• Config component initialisation errors should no longer show nested path annotations.
• Prevented panics from the jq processor when querying invalid types.
• The jaeger tracer no longer emits the service.version tag automatically if the user sets the service.name tag explicitly.
• The int64(), int32(), uint64() and uint32() bloblang methods can now infer the number base as documented here.
• The mapping and mutation processors should provide metrics and tracing events again.
• Fixed a data race in the redis_streams input.
• Upgraded the Redis components to github.com/redis/go-redis/v9.

The full change log can be found here.

v4.11.0

For installation instructions check out the getting started guide.

Added

• Field default_encoding added to the parquet_encode processor.
• Field client_session_keep_alive added to the snowflake_put output.
• Bloblang now supports metadata access via @foo syntax, which also supports arbitrary values.
• TLS client certs now support both PKCS#1 and PKCS#8 encrypted keys.
• New redis_script processor.
• New wasm processor.
• Fields marked as secrets will no longer be printed with benthos echo or debug HTTP endpoints.
• Add no_indent parameter to the format_json bloblang method.
• New format_xml bloblang method.
• New batched higher level input type.
• The gcp_pubsub input now supports optionally creating subscriptions.
• New sqlite buffer.
• Bloblang now has int64, int32, uint64 and uint32 methods for casting explicit integer types.
• Field application_properties_map added to the amqp1 output.
• Param parse_header_row, delimiter and lazy_quotes added to the parse_csv bloblang method.
• Field delete_on_finish added to the csv input.
• Metadata fields header, path, mod_time_unix and mod_time added to the csv input.
• New couchbase processor.
• Field max_attempts added to the nsq input.
• Messages consumed by the nsq input are now enriched with metadata.
• New Bloblang method parse_url.

Fixed

• Fixed a regression bug in the mongodb processor where message errors were not set any more. This issue was introduced in v4.7.0 (64eb72).
• The avro-ocf:marshaler=json input codec now omits unexpected logical type fields.
• Fixed a bug in the sql_insert output (see commit c6a71e9) where transaction-based drivers (clickhouse and oracle) would fail to roll back an in-progress transaction if any of the messages caused an error.
• The resource input should no longer block the first layer of graceful termination.

Changed

• The catch method now defines the context of argument mappings to be the string of the caught error. In previous cases the context was undocumented, vague and would often bind to the outer context. It's still possible to reference this outer context by capturing the error (e.g. .catch(_ -> this)).
• Field interpolations that fail due to mapping errors will no longer produce placeholder values and will instead provide proper errors that result in nacks or retries similar to other issues.

The full change log can be found here.

v4.11.0-rc1

For installation instructions check out the getting started guide.

Added

• Field default_encoding added to the parquet_encode processor.
• Field client_session_keep_alive added to the snowflake_put output.
• Bloblang now supports metadata access via @foo syntax, which also supports arbitrary values.
• TLS client certs now support both PKCS#1 and PKCS#8 encrypted keys.
• New redis_script processor.
• New wasm processor.
• Fields marked as secrets will no longer be printed with benthos echo or debug HTTP endpoints.
• Add no_indent parameter to the format_json bloblang method.
• New format_xml bloblang method.
• New batched higher level input type.
• The gcp_pubsub input now supports optionally creating subscriptions.
• New sqlite buffer.
• Bloblang now has int64, int32, uint64 and uint32 methods for casting explicit integer types.
• Field application_properties_map added to the amqp1 output.
• Param parse_header_row, delimiter and lazy_quotes added to the parse_csv bloblang method.
• Field delete_on_finish added to the csv input.
• Metadata fields header, path, mod_time_unix and mod_time added to the csv input.
• New couchbase processor.
• Field max_attempts added to the nsq input.
• Messages consumed by the nsq input are now enriched with metadata.
• New Bloblang method parse_url.

Fixed

• Fixed a regression bug in the mongodb processor where message errors were not set any more. This issue was introduced in v4.7.0 (64eb72).
• The avro-ocf:marshaler=json input codec now omits unexpected logical type fields.
• Fixed a bug in the sql_insert output (see commit c6a71e9) where transaction-based drivers (clickhouse and oracle) would fail to roll back an in-progress transaction if any of the messages caused an error.
• The resource input should no longer block the first layer of graceful termination.

Changed

• The catch method now defines the context of argument mappings to be the string of the caught error. In previous cases the context was undocumented, vague and would often bind to the outer context. It's still possible to reference this outer context by capturing the error (e.g. .catch(_ -> this)).
• Field interpolations that fail due to mapping errors will no longer produce placeholder values and will instead provide proper errors that result in nacks or retries similar to other issues.

The full change log can be found here.

v4.10.0

For installation instructions check out the getting started guide.

Added

• The nats_jetstream input now adds a range of useful metadata information to messages.
• Field transaction_type added to the azure_table_storage output, which deprecates the previous insert_type field and supports interpolation functions.
• Field logged_batch added to the cassandra output.
• All sql components now support Snowflake.
• New azure_table_storage input.
• New sql_raw input.
• New tracing_id bloblang function.
• New with bloblang method.
• Field multi_header added to the kafka and kafka_franz inputs.
• New cassandra input.
• New base64_encode and base64_decode functions for the awk processor.
• Param use_number added to the parse_json bloblang method.
• Fields init_statement and init_files added to all sql components.
• New find and find_all bloblang array methods.

Fixed

• The gcp_cloud_storage output no longer ignores errors when closing a written file, this was masking issues when the target bucket was invalid.
• Upgraded the kafka_franz input and output to use github.com/twmb/franz-go@v1.9.0 since some bug fixes were made recently.
• Fixed an issue where a read_until child input with processors affiliated would block graceful termination.
• The --labels linting option no longer flags resource components.

The full change log can be found here.

v4.9.1

For installation instructions check out the getting started guide.

Added

• Go API: A new BatchError type added for distinguishing errors of a given batch.

Fixed

• Rolled back kafka input and output underlying sarama client library to fix a regression introduced in 4.9.0 😅 where invalid configuration (Consumer.Group.Rebalance.GroupStrategies and Consumer.Group.Rebalance.Strategy cannot be set at the same time) errors would prevent consumption under certain configurations. We've decided to roll back rather than upgrade as a breaking API change was introduced that could cause issues for Go API importers (more info here: IBM/sarama#2358).

The full change log can be found here.

v4.9.0

For installation instructions check out the getting started guide.

Added

• New parquet input for reading a batch of Parquet files from disk.
• Field max_in_flight added to the redis_list input.

Fixed

• Upgraded kafka input and output underlying sarama client library to fix a regression introduced in 4.7.0 where The requested offset is outside the range of offsets maintained by the server for the given topic/partition errors would prevent consumption of partitions.
• The cassandra output now inserts logged batches of data rather than the less efficient (and unnecessary) unlogged form.

The full change log can be found here.