Latency metrics design proposal

[ktock]

By @vkuzniet (#358)

Currently stargz snapshotter emits only 2 metrics: layer_fetched_size and layer_size. In addition to those metrics, I suggest adding the following:

• Image and layer metrics
  • Prefetch bytes per layer (or prefetch ratio)
• Quantifying accuracy of prefetch files
  • Number of cache hits
  • Total on-demand bytes served out of cache
  • Number of on-demand file fetches (aka cache miss)
  • Total bytes fetched on-demand
  • Latency between mounting a layer and last on-demand fetch
• Latency Metrics
  • Prefetch
    • Time to download prefetch data
    • Time to decompress prefetch data
    • Total time to complete prefetch
  • Background fetch
    • Time to download background fetch data
    • Time to decompress background fetch data
    • Total time to complete background fetch
  • Mount
    • Time to deserialize json TOC
    • Total time to mount Latency metrics design proposal #358
    • Total wait time for all mounts to complete
  • Remote registry access latency Latency metrics design proposal #358
  • Decompression latency
  • Total request latency for on-demand reads from the container
• Count Metrics
  • Number of retries to download data
  • Count of concurrent remote registry requests
  • Number of times a background task was preemptively cancelled