Get compression working with aws_s3 sink (files uncompressed)

[mblarsen]

I'm having a hard time getting compression to work with the aws_s3 sink when using batch. Batching works just fine, but the uploaded file is always uncompressed.

This is my configuration

sinks:
  amplitude_backup_s3:
    batch:
      max_bytes: 5242880
      timeout_secs: 300
    bucket: my-bucket
    buffer:
      max_size: 268435488
      type: disk
      when_full: block
    compression: gzip
    encoding:
      codec: json
    filename_append_uuid: false
    filename_extension: json.gz
    key_prefix: "year=%Y/month=%m/day=%d/ts="
    inputs:
      - amplitude_backup_source
    region: eu-central-1
    storage_class: INTELLIGENT_TIERING
    type: aws_s3

sources:
  amplitude_backup_source:
    address: 0.0.0.0:8080
    encoding: json
    type: http
    path: /2/httpapi

I'm using timberio/vector:latest-alpine docker image so I don't know the exact vector version. But fairly recent I would guess.

[jszwedko]

Hi @mblarsen !

How are you verifying that the file is compressed or uncompressed? Note as mentioned on https://vector.dev/docs/reference/configuration/sinks/aws_s3/#compression browsers and some libraries and CLI clients will transparently decompress when downloading. I suggest verifying the size as listed in S3 vs. the size when you download it to verify it is compressed (the content-encoding attribute in S3 should also list the compression).

[mblarsen]

Yes, I did consider that fact. I downloaded the file using aws s3 cp to make sure my OS didn't mess with me.

Also usually if you browser/os does the decompression upon download usually they remove the zip extension as well.

After download (using aws cli) I run file to inspect the file type (of course cat would reveal the same)

§ file '/ts=1735814209.json.gz'
ts=1735814209.json.gz: JSON data

[mblarsen]

My apologies -- upon trying the approach again (deleting previous attempts) it works as expected. Now I see 10kb vs 90kb

§ ls -la test.json.gz
.rw-r--r-- me staff 10 KB Fri Jan  3 15:56:26 2025  test.json.gz

§ file test.json.gz
test.json.gz: gzip compressed data, original size modulo 2^32 92775

§ open test.json.gz
(decompresses on mac)

§ ls -la test.json
.rw-r--r-- me staff 91 KB Fri Jan  3 16:09:35 2025  test.json