enhancement(aws_s3 source): Logs processed S3 objects #22083
Merged
+49
−5
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Unlike the `aws_sqs` source type, the sqs messaage itself is not the source of events. This logs the bucket and key for files that were ingested via vector in order to have a better audit log. If acknowledgements are enabled, it also logs when they are successfully processed.
github-actions
bot
added
the
domain: sources
jszwedko
changed the title
pront
approved these changes
Jan 6, 2025
pront
reviewed
Jan 6, 2025
pront
reviewed
Jan 9, 2025
src/sources/aws_s3/sqs.rs
Outdated
| @@ -215,7 +215,11 @@ pub enum ProcessingError { | |||
| #[snafu(display("Unsupported S3 event version: {}.", version,))] | |||
| UnsupportedS3EventVersion { version: semver::Version }, | |||
| #[snafu(display("Sink reported an error sending events"))] | |||
There was a problem hiding this comment.
This looks much better now. You can also use these fields in #[snafu(display... like UnsupportedS3EventVersion does above.
There was a problem hiding this comment.
Thank you very much. I've incorporated these into the snafu string.
I appreciate all the handholding you've been providing.
There was a problem hiding this comment.
Thank you for your contribution!
If all checks pass, this will be added to the merge queue.
auto-merge was automatically disabled
January 10, 2025 13:01
Head branch was pushed to by a user without write access
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds log events for each object processed from S3 to add an audit trail. This is similar to the events that are logged for the
filesource (started watching, stopped watching).If acknowledgements on the sink are enabled, then this will also log completion of the object.
Unlike the
aws_sqssource, which converts sqs messages to logs, theaws_s3source consumes objects that usually contain many logs.Change Type
Is this a breaking change?
How did you test this PR?
Deployed to our test and dev environments, with pipelines that have acknowledgements enabled and others that have it disabled. Reviewed logging.
Does this PR include user facing changes?
Checklist
Cargo.lock), pleaserun
dd-rust-license-tool writeto regenerate the license inventory and commit the changes (if any). More details here.References