Add support for k8s projected service account token for OAUTH (kafka)

[creedasaurus]

Feature/Idea

Support file-based OIDC tokens that periodically get refreshed for OIDC-capable APIs (in my case, Kafka/Schema-registry) similar to how the kubernetes client works with projected volumes to mount service-account tokens.

Background

I am working with a 3rd party kafka provider and realized that I could use the projected service account token that EKS automatically mounts in a pod with a specified service account, which is a JWT managed by the AWS OIDC provider, to authenticate my services to the Kafka cluster. It seemed like a pretty great solution for short-lived auto-rotated auth tokens (yay security). I did have to create a small custom AuthenticateCallbackHandler(sry. Its Java.). But they added this OIDC support along with a FileTokenRetriever in the Kafka java client some time back. I modified it slightly to add a cache and re-read the file periodically, since the file get's refreshed.

Implementation

File Cache resource

Initially, I saw that the Sarama Kafka component could use a cache resource to provide a token, so I tested out making a modified version (had to use Bento for the fork, for some reason dependencies were failing with connect on my machine) to try loading the token from a filepath in the pod (in my EKS case, /var/run/secrets/eks.amazonaws.com/serviceaccount/token). It worked great, but I then realized that the file cache doesn't have expiry or anything, and is described as mostly for debugging purposes.

Dedicated mechanism

After trying that out, I thought maybe a better solution would be a more OIDC specific resource, that maybe had a better understanding of the refresh requirements or something. Maybe this wouldn't necessarily be a "component" per se, but a common set of functions that components could use to add a new form of auth if they wanted, e.g.:

input:
  label: "kafka_1"
  kafka_franz:
    seed_brokers:
      - "some.broker:9092"
    topics:
      - "test-topic"
    sasl:
      - mechanism: "OAUTHBEARER"
        oidc:
          path: "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
          extensions:
            logicalCluster: lkc-abcdefg
            identityPoolId: pool-abcdefg

Thoughts?

Anyway, I am not a whiz when it comes to oauth really. I'm a little new to it. However, when I figured out I could have a form of short-lived credentials, that my application didn't necessarily have to manage the rotation of, and getting all the benefits of a "workload identity" role, rather than doling out long-lived key/secrets for each service with different scopes, etc. I thought this was amazing and would be a great benefit for Benthos/Connect.

I don't think this would be very difficult to implement. And there is prior art in the form of the kubernetes client itself. Would this be something worth adding as an internal component?

Also, am I even thinking about this the right way? Again, I have little experience in OIDC, etc. I have used AWS IAM for a long time though, and want the same role-based workload identity. This seemed to work well, but I'm really not even sure it's the "correct" way to do things. I could only imagine the k8s api was pretty well vetted though. Anyway. Would love any feedback or linking in someone with good OIDC knowledge. For now I'll be plumbing in static vault secrets that aren't rotated regularly 😬

Links

• k8s client BearerAuthRoundTripper
  • example of datadog-agent using it. This is authenticating with the kubernetes API, of course, but as far as I can tell, it's using the same kind of mechanism any OIDC provider/consumer would use, right?
• kafka oauth token retriever uses file if the token endpoint URL is a file path
  • except that this implementation doesn't cache and re-read the token periodically
• My own fork of Bento to test out the file-path (and fixing the extensions property) token cache
  • side-note: I couldn't build connect on my machine for some reason. I'll probably file an issue for that later
  • This wasn't even totally necessary except that the cache provider implementation for sarama didn't include the extensions (PR incoming for that specifically).