AWS source auth.assume_role isn't working

[RoeiSagi]

Hey All,
I'm having issues with configurating source and use the aut.assume_role.
I have the following configuration:
[sources.demo]
type = "aws_sqs"
queue_url = "queueURL"
auth.region = "us-east-1"
auth.assume_role = "arn:aws:iam::AccountID:role/RoleName"

But I receive:
vector::internal_events::aws_sqs: Failed to fetch SQS events. error=failed to construct request error_code="failed_fetching_sqs_events" .
Any one has encountered this ? The same worked with KeyID and Secret Key.

[jszwedko]

Hi @RoeiSagi ,

I'd suggest running with the VECTOR_LOG environment variable set to aws_config=debug. That should get you some more useful logs for debugging this sort of issue.

[bkonicek-calm]

I am seeing this same issue, on Helm chart 0.36.1, appVersion 0.41.1 deployed to an EKS cluster. The role I am assuming is in a different AWS account, with permissions to sqs:ReceiveMessage and sqs:DeleteMessage on the topic.

I've set logLevel: "aws_config=debug" in the chart values and get these log entries before the pod restarts

+ aggregator-vector-0 › vector
aggregator-vector-0 vector 2024-10-11T18:11:51.808678Z DEBUG aws_config::fs_util: loaded home directory src="HOME"
aggregator-vector-0 vector 2024-10-11T18:11:51.808720Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
aggregator-vector-0 vector 2024-10-11T18:11:51.808735Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
aggregator-vector-0 vector 2024-10-11T18:11:51.808751Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file not found path=~/.aws/config
aggregator-vector-0 vector 2024-10-11T18:11:51.808767Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/config") size=0
aggregator-vector-0 vector 2024-10-11T18:11:51.808781Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
aggregator-vector-0 vector 2024-10-11T18:11:51.808790Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
aggregator-vector-0 vector 2024-10-11T18:11:51.808803Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
aggregator-vector-0 vector 2024-10-11T18:11:51.808814Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/credentials") size=0
aggregator-vector-0 vector 2024-10-11T18:11:51.875064Z DEBUG aws_config::fs_util: loaded home directory src="HOME"
aggregator-vector-0 vector 2024-10-11T18:11:51.875088Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
aggregator-vector-0 vector 2024-10-11T18:11:51.875097Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
aggregator-vector-0 vector 2024-10-11T18:11:51.875113Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file not found path=~/.aws/config
aggregator-vector-0 vector 2024-10-11T18:11:51.875119Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/config") size=0
aggregator-vector-0 vector 2024-10-11T18:11:51.875128Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
aggregator-vector-0 vector 2024-10-11T18:11:51.875134Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
aggregator-vector-0 vector 2024-10-11T18:11:51.875142Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
aggregator-vector-0 vector 2024-10-11T18:11:51.875147Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/credentials") size=0
aggregator-vector-0 vector 2024-10-11T18:11:51.880087Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials
aggregator-vector-0 vector 2024-10-11T18:11:51.880359Z DEBUG assume_role:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Environment context=the credential provider was not enabled: environment variable not set (CredentialsNotLoaded(CredentialsNotLoaded { source: Some("environment variable not set") }))
aggregator-vector-0 vector 2024-10-11T18:11:51.880405Z DEBUG assume_role:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}: aws_config::fs_util: loaded home directory src="HOME"
aggregator-vector-0 vector 2024-10-11T18:11:51.880416Z DEBUG assume_role:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
aggregator-vector-0 vector 2024-10-11T18:11:51.880425Z DEBUG assume_role:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
aggregator-vector-0 vector 2024-10-11T18:11:51.880441Z DEBUG assume_role:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file not found path=~/.aws/config
aggregator-vector-0 vector 2024-10-11T18:11:51.880454Z DEBUG assume_role:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/config") size=0
aggregator-vector-0 vector 2024-10-11T18:11:51.880468Z DEBUG assume_role:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
aggregator-vector-0 vector 2024-10-11T18:11:51.880476Z DEBUG assume_role:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
aggregator-vector-0 vector 2024-10-11T18:11:51.880486Z DEBUG assume_role:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
aggregator-vector-0 vector 2024-10-11T18:11:51.880492Z DEBUG assume_role:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/credentials") size=0
aggregator-vector-0 vector 2024-10-11T18:11:51.880504Z DEBUG assume_role:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Profile context=the credential provider was not enabled: No profiles were defined (CredentialsNotLoaded(CredentialsNotLoaded { source: Some(NoProfilesDefined) }))
aggregator-vector-0 vector 2024-10-11T18:11:51.936366Z DEBUG assume_role:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: loaded credentials provider=WebIdentityToken
aggregator-vector-0 vector 2024-10-11T18:11:51.985169Z DEBUG assume_role: aws_config::sts::assume_role: obtained assumed credentials access_key_id=Some("REDACTED")

It looks like it's correctly assuming my role in the other account, but then something is causing the request to SQS to fail. I'm not seeing any Access Denied logs in CloudTrail to help troubleshoot this either.

[jszwedko]

I don't see that error in the logs you provided 🤔 Are there more logs?

[bkonicek-calm]

I haven't even gotten to the point of uploading to S3 to test it. When I update my Helm chart to setup the source/sink for aws_s3 it logs these lines and then after ~30-60 seconds the pod terminates and restarts. It seems to be related to the liveness/readiness probes. I disabled them, and it stopped crashlooping, but then I tried uploading a file to S3 and got the same error (with debugging disabled). When I turn debugging back on and try uploading another file I don't see any errors.

aggregator-vector-0 vector 2024-10-11T19:08:03.107723Z ERROR source{component_kind="source" component_id=saas_audit_logs component_type=aws_s3}: vector::internal_events::aws_sqs: Failed to fetch SQS events. error=service error error_code="failed_fetching_sqs_events" error_type="request_failed" stage="receiving" internal_log_rate_limit=true

[bkonicek-calm]

I just want to confirm my architecture here to make sure I'm not missing anything

1. S3 bucket setup to notify SQS topic in Account A
2. IAM role in Account A with permissions to getobject from S3 bucket, and sqs:* from the SQS topic. Trust relationship allowing Vector role in Account B to sts:AssumeRole
3. Vector installed in EKS cluster in account B
4. Vector uses IRSA role with permissions to assume IAM role in Account A

Vector config snippet:

sources:
  saas_audit_logs:
    type: aws_s3
    region: us-east-1
    compression: auto
    auth:
      assume_role: arn:aws:iam::ACCOUNT-A:role/vector-cross-account-role
    sqs:
      queue_url: https://sqs.us-east-1.amazonaws.com/ACCOUNT-A/vector-queue
sinks:
  stdout:
    type: console
    inputs: [datadog_agent, fluent, vector, saas_audit_logs]
    encoding:
      codec: json

[jszwedko]

Interesting, I would have expected to see some AWS debug lines around the error log you shared:

aggregator-vector-0 vector 2024-10-11T19:08:03.107723Z ERROR source{component_kind="source" component_id=saas_audit_logs component_type=aws_s3}: vector::internal_events::aws_sqs: Failed to fetch SQS events. error=service error error_code="failed_fetching_sqs_events" error_type="request_failed" stage="receiving" internal_log_rate_limit=true

The logs you provided before don't seem to include the error log here. I would also expect to see the AWS response which would include more details about what the failure was.

[bkonicek-calm]

I think I found the problem - we're using a customer managed KMS key on our SQS topic, and the role we are assuming was missing some permissions to properly kms:Decrypt and kms:DescribeKey on the KMS key. After updating the allowed IAM actions this now seems to be working.