Unable to Capture upstream_addr and upstream_response_time in Datadog Metrics

[JDarzan]

I’m currently using Kong Self-Managed Gateway Hybrid Enterprise (via Konnect) to proxy traffic and sending metrics to Datadog via the Datadog plugin. My goal is to track individual upstream targets within an upstream service to diagnose performance bottlenecks.
I have multiple targets behind a single upstream (e.g., 4 instances of a GraphQL service on port 3000). Right now, the Datadog plugin aggregates metrics at the upstream level, but I need per-target visibility. If one of these 4 instances is performing poorly, I have no way of identifying it because I only see the total latency of the upstream group, not the individual targets.

Current Status:
I can successfully log upstream_addr and upstream_response_time in Kong’s access logs
This confirms that Kong knows the exact upstream target handling the request.

187.73.227.93 - - [18/Mar/2025:02:08:25 +0000] "POST /graphql HTTP/1.1" 200 129 "-" "curl/8.1.2" kong_request_id: "5b99234bd472a16366c3b04ebe331f1d" upstream_addr: "51.xxx.xxx.xxx:3000" upstream_response_time: "0.591"

Setting in kong.conf:

nginx_http_log_format = kong_upstream_log '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" kong_request_id: "$kong_request_id" upstream_addr: "$upstream_addr" upstream_response_time: "$upstream_response_time"'
proxy_access_log=logs/access.log kong_upstream_log

I try in plugin datadog set this tags:

"tags": [
  "app:kong",
  "upstream_service:service.name",
  "upstream_route:route.name",
  "upstream_consumer:consumer.name",
  "upstream_host:host",
  "upstream_status:status"
]

However, there is no way to tag metrics with upstream_addr (the actual upstream target).

Modified Datadog plugin to try sending upstream_addr as a tag:
This did not work because the Datadog plugin only supports predefined tags.

Tried injecting X-Upstream-Addr header via a Post-Function plugin:
I used a Kong Functions (Post-Plugins) function to set a response header:

local function set_upstream_addr()
    local upstream_addr = ngx.var.upstream_addr or "unknown"
    kong.response.set_header("X-Upstream-Addr", upstream_addr)
end
set_upstream_addr()

Result: The header was set, but the Datadog plugin still could not capture it.

Tried using a Pre-Function plugin to store upstream_addr before sending the request:
Attempted to store the upstream target in kong.ctx.shared, but the value was unavailable in the logging phase.

How can I modify the Datadog plugin (or another approach) to capture upstream_addr and upstream_response_time as tags in Datadog?

Is there a way to override the Datadog plugin’s tag structure to include upstream_addr dynamically?
Or is there an alternative method to track individual upstream targets while keeping Kong’s built-in Datadog integration?