All counter metrics reports 0 while the xxx_summary_us_count is not 0

[chunyanlv]

Hi, I am running a model in Triton Inference Sever and trying to get the metrics for performance monitoring.
I am now dealing with a case that all the below counter metrics are reporting 0, they are all not increasing.

• nv_inference_request_failure
• nv_inference_request_success
• nv_inference_count
• nv_inference_exec_count
• nv_inference_pending_request_count (gauge)

However the summary metrics like nv_inference_compute_input_summary_us_count, nv_inference_compute_infer_summary_us_count and nv_inference_compute_output_summary_us_count are reporting correctly, those xxx_summary_us_count are increasing correctly.

Here is the configuration for metrics:

--allow-metrics=true \
--allow-cpu-metrics=true \
--allow-gpu-metrics=true \
--metrics-config counter_latencies=false \
--metrics-config summary_latencies=true \

Current version of the triton inference server that I used is 2.51.0.

Can someone help with this? What could be wrong?

[geraldstanje]

hi, are you able to see histogram metrics? try --metrics-config histogram_latencies=true

[chunyanlv]

Hi, @geraldstanje thanks for the reply!
The current version of the triton inference server that I used is 2.51.0, it is not clear this version supports the histogram metric: https://docs.nvidia.com/deeplearning/triton-inference-server/archives/triton-inference-server-2510/user-guide/docs/user_guide/metrics.html#latencies

[geraldstanje]

@chunyanlv What you mean by its not clear? Do you see a histrogram in the /metrics endpoint? Whats your command to startup triton? Can you try with verson 2.55.0?

[chunyanlv]

Hi, @geraldstanje

We can't upgrade it to 2.55.0. I don't see any histrogram in the current /metrics endpoint.

Here is the command of setting up triton:

tritonserver --model-repository=xxx/model-store
--model-control-mode=explicit
--exit-on-error=false
--log-format=ISO8601
--log-verbose=1
--log-info=true
--log-warning=true
--log-error=true
--allow-metrics=true
--allow-cpu-metrics=true
--allow-gpu-metrics=true
--metrics-config counter_latencies=false
--metrics-config summary_latencies=true
--http-port 7000
--grpc-port 7001
--metrics-port 7002

[chunyanlv]

turns out the python_backend doesn't emit count metrics.

[chunyanlv]

but how to close this issue?

[geraldstanje]

Why close?