Opensearch Dashboard Issues

[tastyfrankfurt]

Hey Team,

We are using OpenSearch to monitor our servers and using the host metrics but we are unable to correlate the metric events because events per devices always resolve to gauge.value or counter.value, below is an example of the current log format,
{
"gauge": {
"value": 741896192.0
},
"kind": "absolute",
"name": "filesystem_free_bytes",
"namespace": "host",
"tags": {
"collector": "filesystem",
"device": "/dev/sda2",
"filesystem": "ext4",
"host": "demo.example.local",
"mountpoint": "/boot"
},
"timestamp": "2021-07-16T03:26:40.545460831Z"
}

What we would like to do is have it as follows to work within the constraints of Kibana/Opensearch Dashboard limitations,
{
"kind": "absolute",
"filesystem_free_bytes": 741896192.0
"namespace": "host",
"tags": {
"collector": "filesystem",
"device": "/dev/sda2",
"filesystem": "ext4",
"host": "demo01.example.local",
"mountpoint": "/boot"
},
"timestamp": "2021-07-16T03:26:40.545460831Z"
}
Is there a way to represent this data this way in VRL (we have tried but have not had any success) or is this a feature request we would need to submit.

Thanks

Tasty

[happycode0]

hi Team,

we have the same issue ... all other metric agents (metric beat, fluent bit) have a single field key/value for any metric value which work well with Kibana / elastic visualization ... or ... Opensearch Dashboard Visualization

what i'm after to remap any of the metric events ..
from 2 fields
"gauge.value": 741896192.0
"name": "filesystem_free_bytes",

to 1 field
"filesystem_free_bytes": 741896192.0

need to do that on all metrics of any host_metric source ...

the reason this is important to create any visualization with multi value logic ... can't be done as all values for any metric called gauge.value or count.value ... also try to fix this on the source ...

[spencergilbert]

This sounds like it should ultimately be handled by how metrics are sent via the elasticsearch sink today, but those should be accessible through VRL today (cc @jdrouet).

That said if you have a limited set of metrics you need to use before any deeper changes can be made to the sink.

Transforms:

transforms:
  metric_to_log:
    type: metric_to_log
    inputs: ["host_metrics"]
  remap:
    type: remap
    inputs: ["metric_to_log"]
    source: |
      if .name == "filesystem_free_bytes" {
        .filesystem_free_bytes = del(.gauge.value)
        del(.gauge)
        del(.name)
      }

Output:

{
  "filesystem_free_bytes": 287128838144,
  "host": "COMP-C02F514NML87",
  "kind": "absolute",
  "namespace": "host",
  "tags": {
    "collector": "filesystem",
    "device": "/dev/disk1s2",
    "filesystem": "apfs",
    "mountpoint": "/System/Volumes/Data"
  },
  "timestamp": "2021-07-16T13:11:13.464724Z"
}

It would certainly be awkward to have to do for a large number of metrics, but it could get your dashboards usable today.

Do either of you have documentation on how ES/Kibana uses metrics in their dashboards/visuals?

[tastyfrankfurt]

Hey @spencergilbert,

Yes the above was the same resolution we came too as well, from my perspective I see possible solutions (as you are the developer will definitely be guided by yourself :-)),

1. We cherry pick the ones we need to work with and then add the configuration as per above for each stat type we need. This gets us by in both short and long term but the configuration starts to grow on the client.
2. Vector supports a toggle to convert between current json schema to the adjusted json schema to support elasticsearch/opensearch.
3. A function is created that allows us to rename a key name based on an existing value from another key/value pair.

In regards to how we utilise OpenSearch to demonstrate this behaviour, the following steps are taken using the proposed vrl remap,

1. Create a visualisation using a TSVB and a Top N and Table, use the top 20 values
   https://www.elastic.co/guide/en/kibana/current/tsvb.html
2. On the options tab, filter the metric events we want to look at, for instance, namespace: "host" and name: ("memory_total_bytes or memory_used_bytes") and (tags.device: "/dev/mapper/vg00-" or tags.mountpoint: "/mnt/disk") and select to return last available value.
3. Create an aggregation based on values returned.
4. (Steps 4 and 5 is where the problem starts under the current schema), Reference variable1 to use the value stored under key memory_used_bytes(with the current/default schema this would be gauge.value)
5. Reference variable2 to use the value stored under key memory_total_bytes(with the current/default schema this would be gauge.value)
6. Perform math function to calculate the ratio between the 2 entities.
7. Use Group by to group per the tags.mountpoint value.

Once the above is written we add this to a dashboard that filters based on hostname and then we are above to see the percentage of free space. This would be the same for CPU/RAM etc. Also not this is one use case in this manner and I'm sure there will be more going forward. Let us know if you need screen shots on the tsvb to demonstrate the behaviour.

Hopefully you can see that by having all key/values pairs referring to gauge.value we are unable to differentiate between the what the value refers to within the Kibana TSVB visualisation.

Thanks sorry bit of detail, took be a bit of time to understand where we were stuck also.

Tasty

[spencergilbert]

I think we actually have plans for both 2 and 3, in the upcoming quarter - if I remember correctly. We want to be able to support schemas for sources and sinks to ensure interoperability with other systems. And supporting dynamic field's has been something we've wanted to support in VRL for a little bit

[happycode0]

Thanks spencert for your quick response, for us trying to migrate from from other agents like fluentbit or metricbeat ... to vector ... using same visualization is a key
for our current setup as OpenSearch or ElasticSearch both the same ...

• we ingesting all host metric in single index called host-metric
• all metrics come in separate DOCs and thats fine...
• each document have a filed called tag.collector with value CPU, DISK, Memory ...etc thats also great
• each document have one kind of metric ... ex: filesystem_free_bytes that information stored over 2 fields which is the issue here
  "gauge.value": 741896192.0
  "name": "filesystem_free_bytes"
  the limitation we found with the current schema, each visualization can only work with single value ... as all fields that hold a value called the same either gauge.value or counter.value and to differentiate between one to another we have to attache a filter with field "name": "filesystem_free_bytes" for disk free .... or another "name": "network_receive_bytes_total" ..etc that also fine if we also show a single value per visualization ...

but we can't do any visualization with multiple value logic like bucket script equation, which is fundamental for our dashboards..
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-bucket-script-aggregation.html

another ex:
as for now ..there are no percentage disk space in vector .. and that is fine .. if we can use search-aggregations-pipeline-bucket-script-aggregation which work on multiple value ... like
1] over the index host-metric
2] using tag.collector: filesystem as filter
3] first value aggregation is filesystem_total_bytes
4] second value aggregation is filesystem_used_bytes
5] bucket script equation " 100 * var.filesystem_used_bytes / var.filesystem_total_bytes"
6] group by field "tags.mountpoint"

but for now i can't choose any filed in 3] or 4] as fields that hold value called the same name gauge.value or counter.value

this is only one example .. we have many dashboards with many different logic ..

Really appreciate your work and quick response ...

as of solution .... i attempted with no success to rename gauge.value or counter.value field key to be the same value of the field name with no luck ...

[tastyfrankfurt]

@spencergilbert sorry been offline for a couple of days, we will wait until options 2 and 3 are available in the next quater to see how it looks, for now we will cherry pick the variables giving us grief and relabel them as per the provided examples. Again many thanks for your assistance on this :-)