[Question] Syncronize replicas of eventmesh runtime

[kharlamovart]

Search before asking

• I had searched in the issues and found no similar issues.

Question

I am deploying event mesh in a kubernetes cluster, and I have a question I couldn't find in documentation.
For example, I have 3 replicas of eventmesh runtime, and there's a consumer subscribed to one of them. If this pod will fail, subscription will be lost and consumer won't get new messages from topic until he will resubscribe. Can I do something to not lose subscriptions, maybe there's a way to synchronize subscriptions between different replicas of event mesh runtime?

[github-actions]

Welcome to the Apache EventMesh community!!
We are glad that you are contributing by opening this issue. :D

Please make sure to include all the relevant context.
We will be here shortly.

If you are interested in contributing to our project, please let us know!
You can check out our contributing guide on contributing to EventMesh.

Want to get closer to the community?

WeChat Assistant	WeChat Public Account	Slack
		Join Slack Chat

Mailing Lists:

Name	Description	Subscribe	Unsubscribe	Archive
Users	User support and questions mailing list	Subscribe	Unsubscribe	Mail Archives
Development	Development related discussions	Subscribe	Unsubscribe	Mail Archives
Commits	All commits to repositories	Subscribe	Unsubscribe	Mail Archives
Issues	Issues or PRs comments and reviews	Subscribe	Unsubscribe	Mail Archives

[Pil0tXia]

You can use nginx to reverse proxy multiple EventMesh Runtime nodes. In the new architecture being developed for EventMesh, a traffic proxy layer will be introduced, at which point nginx will no longer be needed.

[kharlamovart]

But even if I use reverse proxy, subscription request will be recieved only by one of the nodes, and if this node fails, subscription will be lost. And if request for subscription will be recieved by all nodes, consumer will recieve same message from topic multiple times (at least it's how I understand it). So is there a workaround to not lose subscription for topic if one of nodes fails?

[Pil0tXia]

@kharlamovart

You can check org.apache.eventmesh.client.http.AbstractHttpClient#selectEventMesh for HTTP mode and org.apache.eventmesh.runtime.core.protocol.tcp.client.rebalance.EventMeshRebalanceService for TCP mode. They implement some of the failover capabilities

[kharlamovart]

I've checked listed source file for http mode, and as far as I understand, it allows for consumer to select a node if multiple ip addresses are given. It's not quite what I'm looking for. In my situation, I have multiple pods with eventmesh-runtime running, so I will have only one ip address to connect, which represents service connected to multiple eventmesh runtimes. In current way of deployment, if I will sent a subscribe request, it will be recieved by one of the pods, and if this pod will fail, the subscription will be lost. Heartbeat request won't fail, because 1) it will be routed to a different pod ;2) failed pod will be instantly restarted. In the end, subscribtion will be lost, and consumer won't know about it. So I guess that using multiple eventmesh runtimes with single data source doesnt't make sense

[Pil0tXia]

Yes, the issue you mentioned does exist in version 1.10.0. While the runtime synchronizes subscription information from other nodes in the Meta, it does not automatically switch streams for the client. In the Function architecture that has already been merged into the master branch, we have introduced a traffic proxy layer to address this problem.

[kharlamovart]

Can you give me more information about "Meta", is it some kind of shared storage? I didn't see it in documentation, maybe thos also can help me in solving my problem.

[Pil0tXia]

Meta is where cluster metadata is stored. It can be a registry like Nacos or some kind of distributed consistency protocol.

[kharlamovart]

Ok, thanks for your answers!