Proposal: Adding messaging publish(send) and process wrapper API and matched instrumentation.

[Cirilla-zmh]

Is your feature request related to a problem? Please describe.

As we know, there are various message-based communication frameworks used by developers, such as Kafka clients, RabbitMQ, Apache EMQ, and even Redisson. The differences in their designs of message model, transmission, and APIs are so significant that we can hardly instrument them all with OTel Java Instrumentation in a seamless manner, unless users manually instrument them with the OpenTelemetry API.

However, many users want to trace their messages with minimal code changes and dependencies. Vendors providing OTel instrumentation distributions may need to capture the entire message transmission process in order to aggregate the data into metrics or build standard tools (dashboards, alerts, etc.). Therefore, we should consider offering a standardized instrumented solution for all of these scenarios.

Describe the solution you'd like

According to our research, there are four types of issues that complicate the instrumentation of messaging frameworks. From difficult to simple, we can list the following:

1. There is no appropriate tracing context carrier for the message model, such as frameworks implementing MQTT v3 or those with custom message models.
2. There is no suitable framework method to inject tracing context into the message body. (In fact, we haven't yet found such tracking points in Java, but they do appear in the Sarama framework for Golang.)
3. There is custom consumption logic, such as that used in Kafka MySQL sinks.
4. There is no appropriate framework method for identifying user process behavior in systems like Kafka clients, RabbitMQ clients, and Spring Batch Kafka Listeners (which return messages directly to users).

If applications are built with frameworks in above modes, we cannot capture all messaging operations only with OTel Java Instrumentation, especially the most significant operations - "publish" and "process".

Our proposal is that, since we can't always find suitable tracking points, why not provide a standard API or even annotations to identify these two processes, like this (this is just a preliminary idea):

public class DemoConsumer {
  private static Consumer consumer;

  public static void main(args []) {
    List<Message> msgList = consumer.recieve();
    msgList.forEach(
      MesssagingHelper.processWrap(
        m -> System.out.println(m.getBody().asString()), textMapCarrier, attributesGetter));
  }
}

Adding such a standard API, rather than adopting a universal approach with the OTel API, makes sense for the following reasons:

1. We can automatically create the sending and processing spans, facilitating the propagation of the tracing context without requiring user code implementation.
2. We can add auto instrumentation to these processes in a way that conforms to OTel semantics, clearly informing developers at the code level about which interfaces need to be implemented to obtain signal data, without requiring them to understand all the roles and principles of the OTel API in advance.
3. We can treat these operations similarly to other operations created in OTel Java Instrumentation, such as adding metrics aggregation logic or building reliable alerts, without worrying about whether developers might overlook critical information in spans created manually.

Describe alternatives you've considered

No response

Additional context

In fact, we believe this is a common issue that affects more than just Java. We hope to begin our efforts in Java first and then expand to other languages. If you have any other suggestions, welcome to join the discussion. ;)

[breedx-splk]

I agree that this feels like it is not a java-specific concern. This might be better brought up in the spec?

[Cirilla-zmh]

I agree that this feels like it is not a java-specific concern. This might be better brought up in the spec?

@breedx-splk Thank you for your reminder! We would like to discuss the feasibility of implementing this in Java first, taking a bottom-up approach, before advocating in the spec. Since I am not sufficiently familiar with the implementations in other languages, I can discuss this issue in the spec if you think it is necessary.

Additionally, we currently have a relatively urgent need for this type of API in Java, so we aim to develop a prototype first. Before the spec accepts this proposal, if the relevant implementation cannot appear in the OTel Java repository, could it be implemented in the OTel Java Contrib repository first? cc @trask

[trask]

thanks @Cirilla-zmh! this sounds like:

#1947 (comment)

provide a simple library/guidance for users who are using poll

[trask]

could it be implemented in the OTel Java Contrib repository first?

this makes sense to me 👍

[Cirilla-zmh]

Here's a prototype: https://github.com/open-telemetry/opentelemetry-java-contrib/pull/1803/files