What would be an ideal way to deploy benthos to Kubernetes for microservices?

[Sayandeep36]

Hi we have been able to deploy benthos to kubernetes cluster with one workflow, however, our observations while deployment was we cannot add replica sets.

We have a lot of benthos workflows which can be considered as microservices and we would like to deploy all of them to kubernetes keeping the considerations of scaling up/pod failure and recovery mechanisms.

Is it recommended to use streams to deploy all workflows into one big benthos service or the ideal way would be deploy multiple benthos services corresponding to each microservice/workflows so that we can run the kubernetes pods smoothly.

We have done a lot of hit and trial during our deployment process, any help and suggestions will be welcomed.

Thanks in advance.

[mihaitodor]

Hey @Sayandeep36, please find some answers inline:

however, our observations while deployment was we cannot add replica sets.

A Kubernetes Deployment manages ReplicaSets. You need to interact with the Deployment if you wish to change anything that's owned by it, including scaling up and down, rolling out a new version, etc. The Deployment can apply the changes in a seamless way if configured correctly. The docs here and here should clarify the differences and offer some guidance on which one to use.

Is it recommended to use streams to deploy all workflows into one big benthos service or the ideal way would be deploy multiple benthos services corresponding to each microservice/workflows so that we can run the kubernetes pods smoothly.

It depends. I know that's not what you want to hear, but each approach has its own advantages and disadvantages.

Using Streams Mode lets you run multiple streams in the same Benthos process which makes it easy to, say, have a cluster of N Benthos instances running in Streams Mode with some load balancer in front of it. Then you can just POST new streams to this cluster through the load balancer and a random Benthos instance will pick up the stream and run it. That's all great for workflows which are finite and aren't resource-intensive (think about reading some small json / csv file from S3 and writing it to a database). However, for perpetual workflows, you'll have to come up with a solution to delete them from the right pod when they need to be stopped, which gets messy. Also, you can end up with imbalanced workloads, where you can have a single pod take up way too many streams (can be addressed by using a smarter load balancer). Also, if the workflows are rather arbitrary, it would be hard to know what the memory limit for these pods needs to be.

Using one stream / Benthos pod offers the most flexibility, but then, depending on your scale, you'll have to worry more about resource usage and the limits you'll need to configure. Also, you'll likely need to invest more time in a good orchestrator, maybe something like Flyte or Airflow or any other alternative (there are many). You can even consider building your own custom Kubernetes Operator which can do the job quite well. One thing to keep in mind is that, in this case, one-size-fits-all resource limits is likely too naive. At the very least, you'll want to have t-shirt size flavours and some hint that your orchestrator can use to figure out which config to use for the pod. You can find some prior art in the Benthos Captain repo and there might be other projects out there that I'm not aware of.