5xx errors in load generator

[agardnerIT]

Bug Report

helm list -A
my-otel-demo    ......   deployed    opentelemetry-demo-0.33.4     1.12.0

Symptom

Lots and lots of 5xx errors.

Failed to load resource: the server responded with a status of 503 (Service Unavailable)
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: the server responded with a status of 503 (Service Unavailable)

What is the expected behavior?

No 5xx errors

What do you expect to see?

What is the actual behavior?

Please describe the actual behavior experienced.

Reproduce

Could you provide the minimum required steps to resolve the issue you're seeing?

We will close this issue if:

• The steps you provided are complex.
• If we can not reproduce the behavior you're reporting.

Additional Context

kubectl -n <REDACTED> describe pod/my-otel-demo-loadgenerator-867c949bd7-dcgtt
Name:             my-otel-demo-loadgenerator-867c949bd7-dcgtt
Namespace:        <REDACTED>
Priority:         0
Service Account:  my-otel-demo
Start Time:       Thu, 28 Nov 2024 14:16:51 +1000
Labels:           app.kubernetes.io/component=loadgenerator
                  app.kubernetes.io/instance=my-otel-demo
                  app.kubernetes.io/name=my-otel-demo-loadgenerator
                  opentelemetry.io/name=my-otel-demo-loadgenerator
                  pod-template-hash=867c949bd7
Annotations:      <none>
Status:           Running
IP:               <REDACTED>
IPs:
  IP:           <REDACTED>
Controlled By:  ReplicaSet/my-otel-demo-loadgenerator-867c949bd7
Containers:
  loadgenerator:
    Container ID:   containerd://344df4921a5b3b84577b3a85bed54f9c6db1ef1547c5f46a47c6798fbc24912b
    Image:          ghcr.io/open-telemetry/demo:1.12.0-loadgenerator
    Image ID:       ghcr.io/open-telemetry/demo@sha256:85c9935ff31b7ab575903fbd0b56a3161ec13e508966df25dc68fcfe7af5ec98
    Port:           8089/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 28 Nov 2024 14:16:52 +1000
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  1500Mi
    Requests:
      memory:  1500Mi
    Environment:
      OTEL_SERVICE_NAME:                                   (v1:metadata.labels['app.kubernetes.io/component'])
      OTEL_COLLECTOR_NAME:                                my-otel-demo-otelcol
      OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE:  cumulative
      LOCUST_WEB_PORT:                                    8089
      LOCUST_USERS:                                       10
      LOCUST_SPAWN_RATE:                                  1
      LOCUST_HOST:                                        http://my-otel-demo-frontendproxy.<REDACTED>.svc.cluster.local:8080
      LOCUST_HEADLESS:                                    false
      LOCUST_AUTOSTART:                                   true
      LOCUST_BROWSER_TRAFFIC_ENABLED:                     true
      PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION:             python
      FLAGD_HOST:                                         my-otel-demo-flagd
      FLAGD_PORT:                                         8013
      OTEL_EXPORTER_OTLP_ENDPOINT:                        http://$(OTEL_COLLECTOR_NAME):4317
      OTEL_RESOURCE_ATTRIBUTES:                           service.name=$(OTEL_SERVICE_NAME),service.namespace=opentelemetry-demo,service.version=1.12.0
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qzcxc (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  kube-api-access-qzcxc:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  4m14s  default-scheduler  Successfully assigned <REDACTED>/my-otel-demo-loadgenerator-867c949bd7-dcgtt to <REDACTED>
  Normal  Pulled     4m14s  kubelet            Container image "ghcr.io/open-telemetry/demo:1.12.0-loadgenerator" already present on machine
  Normal  Created    4m14s  kubelet            Created container loadgenerator
  Normal  Started    4m14s  kubelet            Started container loadgenerator

kubectl -n <REDACTED> get svc
NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                     AGE
my-otel-demo-adservice               ClusterIP   <REDACTED>   <none>        8080/TCP                                                                    169m
my-otel-demo-cartservice             ClusterIP   <REDACTED>    <none>        8080/TCP                                                                    169m
my-otel-demo-checkoutservice         ClusterIP   <REDACTED>     <none>        8080/TCP                                                                    169m
my-otel-demo-currencyservice         ClusterIP   <REDACTED>    <none>        8080/TCP                                                                    169m
my-otel-demo-emailservice            ClusterIP   <REDACTED>    <none>        8080/TCP                                                                    169m
my-otel-demo-flagd                   ClusterIP   <REDACTED>     <none>        8013/TCP,4000/TCP                                                           169m
my-otel-demo-frontend                ClusterIP   <REDACTED>   <none>        8080/TCP                                                                    169m
my-otel-demo-frontendproxy           ClusterIP   <REDACTED>    <none>        8080/TCP                                                                    169m
my-otel-demo-imageprovider           ClusterIP   <REDACTED>    <none>        8081/TCP                                                                    169m
my-otel-demo-kafka                   ClusterIP   <REDACTED>     <none>        9092/TCP,9093/TCP                                                           169m
my-otel-demo-loadgenerator           ClusterIP   <REDACTED>     <none>        8089/TCP                                                                    169m
my-otel-demo-otelcol                 ClusterIP   <REDACTED>   <none>        6831/UDP,14250/TCP,14268/TCP,8888/TCP,4317/TCP,4318/TCP,9464/TCP,9411/TCP   169m
my-otel-demo-paymentservice          ClusterIP   <REDACTED>    <none>        8080/TCP                                                                    169m
my-otel-demo-productcatalogservice   ClusterIP   <REDACTED>   <none>        8080/TCP                                                                    169m
my-otel-demo-quoteservice            ClusterIP   <REDACTED>   <none>        8080/TCP                                                                    169m
my-otel-demo-recommendationservice   ClusterIP   <REDACTED>     <none>        8080/TCP                                                                    169m
my-otel-demo-shippingservice         ClusterIP   <REDACTED>    <none>        8080/TCP                                                                    169m
my-otel-demo-valkey                  ClusterIP  <REDACTED>      <none>        6379/TCP                                                                    169m

curl from on-cluster pod

It works:

$ kubectl -n <REDACTED> run mycurlpod --image=curlimages/curl -i --tty -- sh
~ $ curl http://my-otel-demo-frontendproxy.<REDACTED>.svc.cluster.local:8080
<!DOCTYPE html><html><head><meta charSet="utf-8"/><meta name.....

[julianocosta89]

@agardnerIT are all pods running?

[agardnerIT]

I've turned some off (like OpenSearch, Jaeger, Grafana and accounting service because it keeps crashing) but other than that, yes they are.

components:
  grafana:
    enabled: false
  opensearch:
    enabled: false
  jaeger:
    enabled: false
  prometheus:
    enabled: false
  accountingService:
    enabled: false

$ kubectl -n <REDACTED> get pods
NAME                                                        READY   STATUS      RESTARTS        AGE
my-otel-demo-adservice-6f4f57766f-nq8bj                     1/1     Running     0               6h27m
my-otel-demo-cartservice-6bfc654788-l64ft                   1/1     Running     0               6h27m
my-otel-demo-checkoutservice-5cdc66f5cc-kdckm               1/1     Running     0               6h27m
my-otel-demo-currencyservice-cfd644bbf-hcljk                1/1     Running     0               6h27m
my-otel-demo-emailservice-5955c8ddfd-mf9h2                  1/1     Running     0               6h27m
my-otel-demo-flagd-5b9f48f7b5-ft7rf                         2/2     Running     15              6h27m
my-otel-demo-frauddetectionservice-6895d4c998-5c5d4         1/1     Running     0               6h27m
my-otel-demo-frontend-65c644ffdf-ntvsq                      1/1     Running     0               3h50m
my-otel-demo-frontendproxy-fbb8588cf-c8s2m                  1/1     Running     0               3h50m
my-otel-demo-imageprovider-65dd7698c9-bkfk7                 1/1     Running     0               6h27m
my-otel-demo-kafka-6678c45b5c-6x588                         1/1     Running     1 (59m ago)     6h27m
my-otel-demo-loadgenerator-867c949bd7-b5bsn                 1/1     Running     0               3h24m
my-otel-demo-otelcol-7cb964855d-sd2p9                       1/1     Running     0               3h55m
my-otel-demo-paymentservice-69fb7df989-mnfwd                1/1     Running     0               6h27m
my-otel-demo-productcatalogservice-6f4b457bfd-82t4v         1/1     Running     0               3h52m
my-otel-demo-quoteservice-7d4f6f9666-2ppx2                  1/1     Running     0               6h27m
my-otel-demo-recommendationservice-9f4496497-72hs8          1/1     Running     0               6h27m
my-otel-demo-shippingservice-695d794b8d-4wrkc               1/1     Running     0               6h27m
my-otel-demo-valkey-6cf4dcccbf-nxbcv                        1/1     Running     0               6h27m
mycurlpod                                                   1/1     Running     1 (3h12m ago)   3h18m

[julianocosta89]

@puckpuck any ideas here?
I'm starting to suspect of the PR #1785.

Have you redeployed your running demo in K8s after that was merged?

[agardnerIT]

Have you redeployed your running demo in K8s after that was merged? Have I?

I deployed this today.

[julianocosta89]

Have you redeployed your running demo in K8s after that was merged? Have I?

I deployed this today.

Sorry @agardnerIT, this question was for @puckpuck 😅 .
He has the demo running also using Helm.

Locally, we have one issue with accountingservice, but yours is new.

[agardnerIT]

Side note: It would be good to document which services can be disabled and what effects (if any) that has on the core demo / usecases.

For example, I've turned off my accounting service - but I have no idea whether that matters to the "core system" or not.

Perhaps a new column / table on this page like:

service	core service	technology
accountingservice	✅	.NET
grafana	🔴	...

Note: Turning off core services may effect the demo system. Non-core services are "safe" to disable and cause minimal disruption to the core usecases for the demo.

[cbos]

I (probably) have the same problem, but not with Kubernetes, but with local docker.

When I open http://localhost:8080 I get this:
upstream connect error or disconnect/reset before headers. reset reason: connection timeout

http://localhost:8080/jaeger/ui works fine, http://localhost:8080/feature/ works fine as well.
http://localhost:8080/grafana does not work
And the normal shop does not work.

Next to the standard setup, I have https://github.com/cbos/observability-toolkit coupled with the demo setup.
In the logging in Loki I see this:

2024-11-29 14:06:28.036 | [2024-11-29T13:06:28.036Z] "POST /api/checkout HTTP/1.1" 503 UF upstream_reset_before_response_started{connection_timeout} - "-" 385 91 4999 - "-" "python-requests/2.31.0" "4f9ddd89-e990-9549-9c85-77562419236d" "frontend-proxy:8080" "52.140.110.196:8080" frontend - 172.22.0.26:8080 172.22.0.25:44032 - - 

This is a weird ip address where it tries to connect to: 52.140.110.196

This is what I have in the Envoy proxy (http://localhost:10000/clusters):

flagservice::observability_name::flagservice
flagservice::default_priority::max_connections::1024
flagservice::default_priority::max_pending_requests::1024
flagservice::default_priority::max_requests::1024
flagservice::default_priority::max_retries::3
flagservice::high_priority::max_connections::1024
flagservice::high_priority::max_pending_requests::1024
flagservice::high_priority::max_requests::1024
flagservice::high_priority::max_retries::3
flagservice::added_via_api::false
flagservice::172.22.0.4:8013::cx_active::0
flagservice::172.22.0.4:8013::cx_connect_fail::0
flagservice::172.22.0.4:8013::cx_total::0
flagservice::172.22.0.4:8013::rq_active::0
flagservice::172.22.0.4:8013::rq_error::0
flagservice::172.22.0.4:8013::rq_success::0
flagservice::172.22.0.4:8013::rq_timeout::0
flagservice::172.22.0.4:8013::rq_total::0
flagservice::172.22.0.4:8013::hostname::flagd
flagservice::172.22.0.4:8013::health_flags::healthy
flagservice::172.22.0.4:8013::weight::1
flagservice::172.22.0.4:8013::region::
flagservice::172.22.0.4:8013::zone::
flagservice::172.22.0.4:8013::sub_zone::
flagservice::172.22.0.4:8013::canary::false
flagservice::172.22.0.4:8013::priority::0
flagservice::172.22.0.4:8013::success_rate::-1
flagservice::172.22.0.4:8013::local_origin_success_rate::-1
frontend::observability_name::frontend
frontend::default_priority::max_connections::1024
frontend::default_priority::max_pending_requests::1024
frontend::default_priority::max_requests::1024
frontend::default_priority::max_retries::3
frontend::high_priority::max_connections::1024
frontend::high_priority::max_pending_requests::1024
frontend::high_priority::max_requests::1024
frontend::high_priority::max_retries::3
frontend::added_via_api::false
frontend::52.140.110.196:8080::cx_active::3
frontend::52.140.110.196:8080::cx_connect_fail::425
frontend::52.140.110.196:8080::cx_total::428
frontend::52.140.110.196:8080::rq_active::0
frontend::52.140.110.196:8080::rq_error::480
frontend::52.140.110.196:8080::rq_success::0
frontend::52.140.110.196:8080::rq_timeout::0
frontend::52.140.110.196:8080::rq_total::0
frontend::52.140.110.196:8080::hostname::frontend
frontend::52.140.110.196:8080::health_flags::healthy
frontend::52.140.110.196:8080::weight::1
frontend::52.140.110.196:8080::region::
frontend::52.140.110.196:8080::zone::
frontend::52.140.110.196:8080::sub_zone::
frontend::52.140.110.196:8080::canary::false
frontend::52.140.110.196:8080::priority::0
frontend::52.140.110.196:8080::success_rate::-1
frontend::52.140.110.196:8080::local_origin_success_rate::-1
opentelemetry_collector_grpc::observability_name::opentelemetry_collector_grpc
opentelemetry_collector_grpc::default_priority::max_connections::1024
opentelemetry_collector_grpc::default_priority::max_pending_requests::1024
opentelemetry_collector_grpc::default_priority::max_requests::1024
opentelemetry_collector_grpc::default_priority::max_retries::3
opentelemetry_collector_grpc::high_priority::max_connections::1024
opentelemetry_collector_grpc::high_priority::max_pending_requests::1024
opentelemetry_collector_grpc::high_priority::max_requests::1024
opentelemetry_collector_grpc::high_priority::max_retries::3
opentelemetry_collector_grpc::added_via_api::false
opentelemetry_collector_grpc::172.22.0.9:4317::cx_active::10
opentelemetry_collector_grpc::172.22.0.9:4317::cx_connect_fail::0
opentelemetry_collector_grpc::172.22.0.9:4317::cx_total::10
opentelemetry_collector_grpc::172.22.0.9:4317::rq_active::0
opentelemetry_collector_grpc::172.22.0.9:4317::rq_error::0
opentelemetry_collector_grpc::172.22.0.9:4317::rq_success::791
opentelemetry_collector_grpc::172.22.0.9:4317::rq_timeout::0
opentelemetry_collector_grpc::172.22.0.9:4317::rq_total::791
opentelemetry_collector_grpc::172.22.0.9:4317::hostname::otelcol
opentelemetry_collector_grpc::172.22.0.9:4317::health_flags::healthy
opentelemetry_collector_grpc::172.22.0.9:4317::weight::1
opentelemetry_collector_grpc::172.22.0.9:4317::region::
opentelemetry_collector_grpc::172.22.0.9:4317::zone::
opentelemetry_collector_grpc::172.22.0.9:4317::sub_zone::
opentelemetry_collector_grpc::172.22.0.9:4317::canary::false
opentelemetry_collector_grpc::172.22.0.9:4317::priority::0
opentelemetry_collector_grpc::172.22.0.9:4317::success_rate::-1
opentelemetry_collector_grpc::172.22.0.9:4317::local_origin_success_rate::-1
imageprovider::observability_name::imageprovider
imageprovider::default_priority::max_connections::1024
imageprovider::default_priority::max_pending_requests::1024
imageprovider::default_priority::max_requests::1024
imageprovider::default_priority::max_retries::3
imageprovider::high_priority::max_connections::1024
imageprovider::high_priority::max_pending_requests::1024
imageprovider::high_priority::max_requests::1024
imageprovider::high_priority::max_retries::3
imageprovider::added_via_api::false
imageprovider::172.22.0.12:8081::cx_active::0
imageprovider::172.22.0.12:8081::cx_connect_fail::0
imageprovider::172.22.0.12:8081::cx_total::0
imageprovider::172.22.0.12:8081::rq_active::0
imageprovider::172.22.0.12:8081::rq_error::0
imageprovider::172.22.0.12:8081::rq_success::0
imageprovider::172.22.0.12:8081::rq_timeout::0
imageprovider::172.22.0.12:8081::rq_total::0
imageprovider::172.22.0.12:8081::hostname::imageprovider
imageprovider::172.22.0.12:8081::health_flags::healthy
imageprovider::172.22.0.12:8081::weight::1
imageprovider::172.22.0.12:8081::region::
imageprovider::172.22.0.12:8081::zone::
imageprovider::172.22.0.12:8081::sub_zone::
imageprovider::172.22.0.12:8081::canary::false
imageprovider::172.22.0.12:8081::priority::0
imageprovider::172.22.0.12:8081::success_rate::-1
imageprovider::172.22.0.12:8081::local_origin_success_rate::-1
jaeger::observability_name::jaeger
jaeger::default_priority::max_connections::1024
jaeger::default_priority::max_pending_requests::1024
jaeger::default_priority::max_requests::1024
jaeger::default_priority::max_retries::3
jaeger::high_priority::max_connections::1024
jaeger::high_priority::max_pending_requests::1024
jaeger::high_priority::max_requests::1024
jaeger::high_priority::max_retries::3
jaeger::added_via_api::false
jaeger::172.22.0.3:16686::cx_active::2
jaeger::172.22.0.3:16686::cx_connect_fail::0
jaeger::172.22.0.3:16686::cx_total::2
jaeger::172.22.0.3:16686::rq_active::0
jaeger::172.22.0.3:16686::rq_error::0
jaeger::172.22.0.3:16686::rq_success::6
jaeger::172.22.0.3:16686::rq_timeout::0
jaeger::172.22.0.3:16686::rq_total::6
jaeger::172.22.0.3:16686::hostname::jaeger
jaeger::172.22.0.3:16686::health_flags::healthy
jaeger::172.22.0.3:16686::weight::1
jaeger::172.22.0.3:16686::region::
jaeger::172.22.0.3:16686::zone::
jaeger::172.22.0.3:16686::sub_zone::
jaeger::172.22.0.3:16686::canary::false
jaeger::172.22.0.3:16686::priority::0
jaeger::172.22.0.3:16686::success_rate::-1
jaeger::172.22.0.3:16686::local_origin_success_rate::-1
loadgen::observability_name::loadgen
loadgen::default_priority::max_connections::1024
loadgen::default_priority::max_pending_requests::1024
loadgen::default_priority::max_requests::1024
loadgen::default_priority::max_retries::3
loadgen::high_priority::max_connections::1024
loadgen::high_priority::max_pending_requests::1024
loadgen::high_priority::max_requests::1024
loadgen::high_priority::max_retries::3
loadgen::added_via_api::false
loadgen::20.61.97.129:8089::cx_active::0
loadgen::20.61.97.129:8089::cx_connect_fail::0
loadgen::20.61.97.129:8089::cx_total::0
loadgen::20.61.97.129:8089::rq_active::0
loadgen::20.61.97.129:8089::rq_error::0
loadgen::20.61.97.129:8089::rq_success::0
loadgen::20.61.97.129:8089::rq_timeout::0
loadgen::20.61.97.129:8089::rq_total::0
loadgen::20.61.97.129:8089::hostname::loadgenerator
loadgen::20.61.97.129:8089::health_flags::healthy
loadgen::20.61.97.129:8089::weight::1
loadgen::20.61.97.129:8089::region::
loadgen::20.61.97.129:8089::zone::
loadgen::20.61.97.129:8089::sub_zone::
loadgen::20.61.97.129:8089::canary::false
loadgen::20.61.97.129:8089::priority::0
loadgen::20.61.97.129:8089::success_rate::-1
loadgen::20.61.97.129:8089::local_origin_success_rate::-1
grafana::observability_name::grafana
grafana::default_priority::max_connections::1024
grafana::default_priority::max_pending_requests::1024
grafana::default_priority::max_requests::1024
grafana::default_priority::max_retries::3
grafana::high_priority::max_connections::1024
grafana::high_priority::max_pending_requests::1024
grafana::high_priority::max_requests::1024
grafana::high_priority::max_retries::3
grafana::added_via_api::false
grafana::20.189.171.131:3000::cx_active::0
grafana::20.189.171.131:3000::cx_connect_fail::0
grafana::20.189.171.131:3000::cx_total::0
grafana::20.189.171.131:3000::rq_active::0
grafana::20.189.171.131:3000::rq_error::0
grafana::20.189.171.131:3000::rq_success::0
grafana::20.189.171.131:3000::rq_timeout::0
grafana::20.189.171.131:3000::rq_total::0
grafana::20.189.171.131:3000::hostname::grafana
grafana::20.189.171.131:3000::health_flags::healthy
grafana::20.189.171.131:3000::weight::1
grafana::20.189.171.131:3000::region::
grafana::20.189.171.131:3000::zone::
grafana::20.189.171.131:3000::sub_zone::
grafana::20.189.171.131:3000::canary::false
grafana::20.189.171.131:3000::priority::0
grafana::20.189.171.131:3000::success_rate::-1
grafana::20.189.171.131:3000::local_origin_success_rate::-1
flagdui::observability_name::flagdui
flagdui::default_priority::max_connections::1024
flagdui::default_priority::max_pending_requests::1024
flagdui::default_priority::max_requests::1024
flagdui::default_priority::max_retries::3
flagdui::high_priority::max_connections::1024
flagdui::high_priority::max_pending_requests::1024
flagdui::high_priority::max_requests::1024
flagdui::high_priority::max_retries::3
flagdui::added_via_api::false
flagdui::172.22.0.10:4000::cx_active::0
flagdui::172.22.0.10:4000::cx_connect_fail::0
flagdui::172.22.0.10:4000::cx_total::8
flagdui::172.22.0.10:4000::rq_active::0
flagdui::172.22.0.10:4000::rq_error::0
flagdui::172.22.0.10:4000::rq_success::14
flagdui::172.22.0.10:4000::rq_timeout::0
flagdui::172.22.0.10:4000::rq_total::14
flagdui::172.22.0.10:4000::hostname::flagdui
flagdui::172.22.0.10:4000::health_flags::healthy
flagdui::172.22.0.10:4000::weight::1
flagdui::172.22.0.10:4000::region::
flagdui::172.22.0.10:4000::zone::
flagdui::172.22.0.10:4000::sub_zone::
flagdui::172.22.0.10:4000::canary::false
flagdui::172.22.0.10:4000::priority::0
flagdui::172.22.0.10:4000::success_rate::-1
flagdui::172.22.0.10:4000::local_origin_success_rate::-1
opentelemetry_collector_http::observability_name::opentelemetry_collector_http
opentelemetry_collector_http::default_priority::max_connections::1024
opentelemetry_collector_http::default_priority::max_pending_requests::1024
opentelemetry_collector_http::default_priority::max_requests::1024
opentelemetry_collector_http::default_priority::max_retries::3
opentelemetry_collector_http::high_priority::max_connections::1024
opentelemetry_collector_http::high_priority::max_pending_requests::1024
opentelemetry_collector_http::high_priority::max_requests::1024
opentelemetry_collector_http::high_priority::max_retries::3
opentelemetry_collector_http::added_via_api::false
opentelemetry_collector_http::172.22.0.9:4318::cx_active::0
opentelemetry_collector_http::172.22.0.9:4318::cx_connect_fail::0
opentelemetry_collector_http::172.22.0.9:4318::cx_total::0
opentelemetry_collector_http::172.22.0.9:4318::rq_active::0
opentelemetry_collector_http::172.22.0.9:4318::rq_error::0
opentelemetry_collector_http::172.22.0.9:4318::rq_success::0
opentelemetry_collector_http::172.22.0.9:4318::rq_timeout::0
opentelemetry_collector_http::172.22.0.9:4318::rq_total::0
opentelemetry_collector_http::172.22.0.9:4318::hostname::otelcol
opentelemetry_collector_http::172.22.0.9:4318::health_flags::healthy
opentelemetry_collector_http::172.22.0.9:4318::weight::1
opentelemetry_collector_http::172.22.0.9:4318::region::
opentelemetry_collector_http::172.22.0.9:4318::zone::
opentelemetry_collector_http::172.22.0.9:4318::sub_zone::
opentelemetry_collector_http::172.22.0.9:4318::canary::false
opentelemetry_collector_http::172.22.0.9:4318::priority::0
opentelemetry_collector_http::172.22.0.9:4318::success_rate::-1
opentelemetry_collector_http::172.22.0.9:4318::local_origin_success_rate::-1

Some of the IP addresses are not resolved to local addresses, but remote/public addresses.

[cbos]

Changing src/frontendproxy/Dockerfile fixed my problem.
I changed the FROM to envoyproxy/envoy:v1.31-latest

#1768 changed this version from 1.30 to 1.32. cc: @puckpuck
The release notes of Envoy proxy mention something about internal_address_config.
https://www.envoyproxy.io/docs/envoy/v1.32.0/version_history/v1.32/v1.32.0.html

I don't know all details, but at least this fixed my problem, my frontend (and other services) are now resolved to local addresses:

frontend::observability_name::frontend
frontend::default_priority::max_connections::1024
frontend::default_priority::max_pending_requests::1024
frontend::default_priority::max_requests::1024
frontend::default_priority::max_retries::3
frontend::high_priority::max_connections::1024
frontend::high_priority::max_pending_requests::1024
frontend::high_priority::max_requests::1024
frontend::high_priority::max_retries::3
frontend::added_via_api::false
frontend::172.22.0.24:8080::cx_active::21
frontend::172.22.0.24:8080::cx_connect_fail::0
frontend::172.22.0.24:8080::cx_total::61
frontend::172.22.0.24:8080::rq_active::0
frontend::172.22.0.24:8080::rq_error::0
frontend::172.22.0.24:8080::rq_success::1185
frontend::172.22.0.24:8080::rq_timeout::0
frontend::172.22.0.24:8080::rq_total::1185
frontend::172.22.0.24:8080::hostname::frontend
frontend::172.22.0.24:8080::health_flags::healthy
frontend::172.22.0.24:8080::weight::1
frontend::172.22.0.24:8080::region::
frontend::172.22.0.24:8080::zone::
frontend::172.22.0.24:8080::sub_zone::
frontend::172.22.0.24:8080::canary::false
frontend::172.22.0.24:8080::priority::0
frontend::172.22.0.24:8080::success_rate::-1
frontend::172.22.0.24:8080::local_origin_success_rate::-1

[julianocosta89]

@agardnerIT and @cbos in what distro of k8s are you running this?

I've tried out today with a local minikube and everything started fine 🤔

[puckpuck]

I can confirm I still see this issue. It takes a few minutes to start to happen, but once it does after every approx 2 mins and 15 seconds, I see 155 new entries to the log with the following output:

Failed to load resource: net::ERR_CONNECTION_REFUSED

This statement is not written to the OTLP logs emitted by the load generator. I only see it in the service's console output logs. I can also replicate this in local docker compose and K8s. Though I can see these errors in console logs, I can't spot anything in the load generator that would indicate an issue is happening.

[Danouchka]

Hi I have same issue on GKE but I created my Elastic OTEL demo (2.0.1) in a different namespace than default.
At load generator level i get this

Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: net::ERR_INCOMPLETE_CHUNKED_ENCODING
Failed to load resource: net::ERR_INCOMPLETE_CHUNKED_ENCODING
Failed to load resource: net::ERR_INCOMPLETE_CHUNKED_ENCODING
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: net::ERR_CONNECTION_REFUSED
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)
Failed to load resource: the server responded with a status of 503 (Service Unavailable)
Failed to load resource: the server responded with a status of 503 (Service Unavailable)
Failed to load resource: the server responded with a status of 503 (Service Unavailable)
Failed to load resource: the server responded with a status of 503 (Service Unavailable)
Failed to load resource: the server responded with a status of 503 (Service Unavailable)
Failed to load resource: the server responded with a status of 503 (Service Unavailable)
Failed to load resource: the server responded with a status of 503 (Service Unavailable)
Failed to load resource: the server responded with a status of 503 (Service Unavailable)
Failed to load resource: the server responded with a status of 503 (Service Unavailable)
Failed to load resource: the server responded with a status of 503 (Service Unavailable)
Failed to load resource: the server responded with a status of 503 (Service Unavailable)

My pods are as follow

NAME READY STATUS RESTARTS AGE
accounting-576b6d49b4-rxhkz 1/1 Running 0 29m
ad-9655d5bff-gtd4k 1/1 Running 0 29m
cart-65f9c57d58-jpggk 1/1 Running 0 29m
checkout-544b795d6b-7klk2 1/1 Running 0 29m
currency-5696f8b8df-r9t94 1/1 Running 0 29m
email-58d79785cb-d8mcr 1/1 Running 0 29m
flagd-867874f7fb-9sthc 2/2 Running 0 29m
fraud-detection-64dc657547-bbvsk 1/1 Running 0 29m
frontend-7d9897464b-8q78z 1/1 Running 1 (2m24s ago) 29m
frontend-proxy-86cccf5487-7blll 1/1 Running 0 29m
image-provider-6bc44fd969-xzmcg 1/1 Running 0 29m
kafka-565b9877d-t8d79 1/1 Running 0 29m
load-generator-589d59b657-6wmpb 1/1 Running 0 29m
otel-collector-69b4974788-5qvg2 1/1 Running 3 (8m5s ago) 29m
payment-857ff844b7-4wfml 1/1 Running 0 29m
product-catalog-fb9dd6645-zptzt 1/1 Running 0 29m
quote-69b49fd86-hj8f5 1/1 Running 0 29m
recommendation-6677d47997-znf6h 1/1 Running 0 29m
shipping-5956dc5697-6xgbs 1/1 Running 0 29m
valkey-cart-5f9855dbf9-wg7ll 1/1 Running 0 29m

My services are as follow

dan_abitbol@cloudshell:~/opentelemetry-demo-edot/kubernetes/elastic-helm (elastic-sa)$ kubectl get svc -n opentelemetry-demo-edot -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
ad ClusterIP 10.44.2.53 8080/TCP 30m opentelemetry.io/name=ad
cart ClusterIP 10.44.12.22 8080/TCP 30m opentelemetry.io/name=cart
checkout ClusterIP 10.44.9.54 8080/TCP 30m opentelemetry.io/name=checkout
currency ClusterIP 10.44.11.57 8080/TCP 30m opentelemetry.io/name=currency
email ClusterIP 10.44.10.214 8080/TCP 30m opentelemetry.io/name=email
flagd ClusterIP 10.44.13.25 8013/TCP,4000/TCP 30m opentelemetry.io/name=flagd
frontend ClusterIP 10.44.14.190 8080/TCP 30m opentelemetry.io/name=frontend
frontend-proxy ClusterIP 10.44.12.246 8080/TCP 30m opentelemetry.io/name=frontend-proxy
image-provider ClusterIP 10.44.6.156 8081/TCP 30m opentelemetry.io/name=image-provider
kafka ClusterIP 10.44.5.113 9092/TCP,9093/TCP 30m opentelemetry.io/name=kafka
load-generator ClusterIP 10.44.3.174 8089/TCP 30m opentelemetry.io/name=load-generator
otel-collector ClusterIP 10.44.8.61 6831/UDP,14250/TCP,14268/TCP,8888/TCP,4317/TCP,4318/TCP,9464/TCP,9411/TCP 30m app.kubernetes.io/instance=my-otel-demo-edot,app.kubernetes.io/name=opentelemetry-collector,component=standalone-collector
payment ClusterIP 10.44.15.3 8080/TCP 30m opentelemetry.io/name=payment
product-catalog ClusterIP 10.44.4.139 8080/TCP 30m opentelemetry.io/name=product-catalog
quote ClusterIP 10.44.8.37 8080/TCP 30m opentelemetry.io/name=quote
recommendation ClusterIP 10.44.1.86 8080/TCP 30m opentelemetry.io/name=recommendation
shipping ClusterIP 10.44.13.193 8080/TCP 30m opentelemetry.io/name=shipping
valkey-cart ClusterIP 10.44.3.59 6379/TCP 30m opentelemetry.io/name=valkey-cart

[puckpuck]

I did some investigation on this. The Logs entries are coming from the Playwright browser. Commenting out the lines of code that emit the brower's console output stops the messages from appearing. This is the code on lines 191 and 203 of locustfile.py

                page.on("console", lambda msg: print(msg.text))

I don't know why the error is getting reported, but this is where it is being reported from.

[Danouchka]

Actually it's more than an error because downstream, we see no transactions

[puckpuck]

I can continue to see traces in Jaeger for the frontend-web service with an attribute of app.synthetic_request = true well after the errors are reported in the load-generator logs. Outside of an error getting logged, I don't see any specific ill effects from this.