datadog_logs sink throws warning when host is remapped

[adam-bartlett-sp]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

When we remap hostname/host values, the datadog_logs sink throws this warning constantly.

WARN vector::sinks::datadog::logs::sink: Semantic meaning is defined, but the event path already exists. Renaming to not overwrite. meaning="host" renamed="_RESERVED_host" internal_log_rate_limit=true

Configuration

sources:
  journals: 
    type: journald
    journal_directory: /mnt/journal
    current_boot_only: true
transforms:
  journals_add_host:
    type: remap
    inputs: 
      - journals
    source: "del(._CAP_EFFECTIVE)\ndel(.CODE_FILE)\ndel(.host)\n.hostname = \"(redacted)\"\n.host = \"(redacted)\""
  metrics_add_host:
sinks:
  ddg_logs:
    type: datadog_logs
    inputs:
      - journals_add_host
    default_api_key: xxxx

Version

0.43.1

Debug Output

https://gist.githubusercontent.com/adam-bartlett-sp/5ac0241d77c36d25a319ecbbe16a9551/raw/4e16cf2becd16d9b2d6fcf7fa75adb3273b5a8b6/gistfile1.txt

Example Data

No response

Additional Context

No response

References

No response

[pront]

Hi @adam-bartlett-sp, this is hitting this branch because .host is a reserved attribute used by the sink:

vector/src/sinks/datadog/logs/sink.rs

Lines 153 to 165 in 68ee4f2

	// if an existing attribute exists here already, move it so to not overwrite it.
	// yes, technically the rename path could exist, but technically that could always be the case.
	if log.contains(desired_path) {
	let rename_attr = format!("_RESERVED_{}", meaning);
	let rename_path = event_path!(rename_attr.as_str());
	warn!(
	message = "Semantic meaning is defined, but the event path already exists. Renaming to not overwrite.",
	meaning = meaning,
	renamed = &rename_attr,
	internal_log_rate_limit = true,
	);
	log.rename_key(desired_path, rename_path);
	}

The journald source is populating host (and not hostname as other sources do) and the datadog_logs sink wants to use that field. In an effort to avoid data loss we move the existing value to _RESERVED_host and populate host.

---

The mappings are defined here:

vector/src/common/datadog.rs

Lines 21 to 31 in 68ee4f2

	/// Mapping of the semantic meaning of well known Datadog reserved attributes
	/// to the field name that Datadog intake expects.
	// https://docs.datadoghq.com/logs/log_configuration/attributes_naming_convention/?s=severity#reserved-attributes
	pub const DD_RESERVED_SEMANTIC_ATTRS: [(&str, &str); 6] = [
	(meaning::SEVERITY, "status"), // status is intentionally semantically defined as severity
	(meaning::TIMESTAMP, "timestamp"),
	(meaning::HOST, "hostname"),
	(meaning::SERVICE, "service"),
	(meaning::SOURCE, "ddsource"),
	(meaning::TAGS, DDTAGS),
	];

Datadog Logs Reserved Attributes:
https://docs.datadoghq.com/logs/log_configuration/attributes_naming_convention/?s=severity#reserved-attributes

[adam-bartlett-sp]

@pront thanks for that.

I'm trying to work out (other than setting log level ERROR) how to replace the host value from journald with something more meaningful - the default host name in this case isn't meaningful to the end user. I can't seem to find a clearly documented method - the traditional datadog agent has the agent-hostname option, but I can't seem to track one down here (short of the usual "user didn't read the manual deep enough" problem).

[pront]

@pront thanks for that.

I'm trying to work out (other than setting log level ERROR) how to replace the host value from journald with something more meaningful

Setting .host in your Remap (like you are already doing) should achieve this. Can you share a sample log from your Datadog Logs explorer (or live tail)?

I am thinking that this warning might be confusing to users but it was added to raise awareness that the original value was moved to another path. You can ignore it.