Why is structured json not recognised?

[tamis-laan]

I'm using docker compose logs as followed:

docker compose logs --no-color -f caddy | lnav

Which works and the structured json is formatted properly.

But when I run it for my API:

docker compose logs --no-color -f api | lnav

It's not properly formatted.

Why is that? What makes lnav recognice json or not???

[tstack]

What version of lnav are you using?

[tstack]

I'm using docker compose logs as followed:

docker compose logs --no-color -f caddy | lnav

Which works and the structured json is formatted properly.

There is a builtin log format definition for caddy here:

https://github.com/tstack/lnav/blob/master/src/formats/caddy_log.json

That format matched the caddy log messages that were piped in.

But when I run it for my API:

docker compose logs --no-color -f api | lnav

It's not properly formatted.

You'll need to add a log format definition that can recognize your log messages. The documentation for log formats is here. There's a few formats for JSON-lines files that you can use as a reference. You could also provide some sample lines here (or send to support@lnav.org) and I can help you get started.

To get an idea of why the log format matched, you can press TAB a couple times to focus on the "Files" panel on the bottom. Then, select the file and press CTRL+] to focus on the details view. It should look something like this:

The "match quality" refers to how many of the properties in the log message match the "line-format" in the log format definition.

[tamis-laan]

@tstack in the log formats I can see the json-lines format. So shouldn't that work on my api? It's literally a json object per line. Or am I missing something?

{"created": 1742475248.7940516, "levelname": "INFO", "name": "uvicorn.access", "message": "127.0.0.1:43148 - \"GET /health HTTP/1.1\" 200", "module": "httptools_impl", "funcName": "send", "lineno": 476}
{"created": 1742475278.8656027, "levelname": "INFO", "name": "uvicorn.access", "message": "127.0.0.1:45390 - \"GET /health HTTP/1.1\" 200", "module": "httptools_impl", "funcName": "send", "lineno": 476}
{"created": 1742475308.9265337, "levelname": "INFO", "name": "uvicorn.access", "message": "127.0.0.1:58650 - \"GET /health HTTP/1.1\" 200", "module": "httptools_impl", "funcName": "send", "lineno": 476}

[tstack]

Using the sample log lines you provided, I wrote up this log format -- tamis_api_log.json

Download it and install it using the following command:

$ lnav -i tamis_api_log.json

Here's what those samples look like for me:

You should edit it the format so the display is to your liking.

[tamis-laan]

@tstack Ah I see how it works. It can't be any kind of json-lines it has to follow a schema. Then instead of creating my own schema I think it would be better to use an existing schema. I think ecs logging would work best as it's the most comprehensive json type logging.

This is what my loggs look like:

{"@timestamp":"2025-03-20T13:50:42.044Z","log.level":"info","message":"127.0.0.1:59826 - \"GET /health HTTP/1.1\" 200","ecs.version":"1.6.0","log":{"logger":"uvicorn.access","origin":{"file":{"line":476,"name":"httptools_impl.py"},"function":"send"},"original":"127.0.0.1:59826 - \"GET /health HTTP/1.1\" 200"},"process":{"name":"MainProcess","pid":1,"thread":{"id":139798436514600,"name":"MainThread"}}}

I created it using the official python ecs logging library from elastic search.
https://www.elastic.co/guide/en/ecs-logging/python/current/installation.html

However lnav still doesn't parse it, could this be a ecs schema issue?

I think this is because the ecs_log requires log/level instead of log.level, not sure if ECS standard changed or something.