Add documentation about logging (#4426)

This commit adds a small description of how our loggers are
configured and instantiated.

Author: belimawr

docs/elastic-agent-logging.md

@@ -0,0 +1,71 @@
+# Elastic-Agent logging
+The Elastic-Agent process defines two log outputs:
+ - The "internal" core that is used by monitoring components and
+   collected in the diagnostics. It's configuration is **hardcoded**.
+   This output always logs to files in
+   `data/elastic-agent-<hash>/logs` and uses the default configuration
+   defined in `elastic-agent-libs/logp`.
+ - The "default" logger that is the user-configurable logger, it logs
+   to the Elastic-Agent's root folder, it can also be configured to
+   log to `stderr`. When running in a container environment, it
+   defaults to logging to `stderr`.
+
+## Logger initialization
+The logger initialization is **not** one of the first things done by
+the Elastic-Agent. Looking at the normal Elastic-Agent run, here is
+the stack trace from the logging initialization.
+```
+ 0  0x00005e8768fea7ac in github.com/elastic/elastic-agent/pkg/core/logger.new
+    at /devel/elastic-agent/pkg/core/logger/logger.go:83
+ 1  0x00005e8768fea54f in github.com/elastic/elastic-agent/pkg/core/logger.NewFromConfig
+    at /devel/elastic-agent/pkg/core/logger/logger.go:65
+ 2  0x00005e876b511dba in github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent
+    at /devel/elastic-agent/internal/pkg/agent/cmd/run.go:151
+ 3  0x00005e876b5119e5 in github.com/elastic/elastic-agent/internal/pkg/agent/cmd.run
+    at /devel/elastic-agent/internal/pkg/agent/cmd/run.go:138
+ 4  0x00005e876b51127e in github.com/elastic/elastic-agent/internal/pkg/agent/cmd.newRunCommandWithArgs.func1
+    at /devel/elastic-agent/internal/pkg/agent/cmd/run.go:78
+```
+This means some log entries might not be collected by diagnostics or
+shipped to the monitoring output. Everything in the `run` function
+happens before the logger initialization.
+
+https://github.com/elastic/elastic-agent/blob/574aa5db629231d56062ab40d27ccceb02cbbe4d/internal/pkg/agent/cmd/run.go#L104-L138
+
+## Internal logging
+The internal logging output is crucial for the Elastic-Agent
+self-monitoring and diagnostics. It is instantiated by
+[MakeInternalFileOutput](https://github.com/elastic/elastic-agent/blob/574aa5db629231d56062ab40d27ccceb02cbbe4d/pkg/core/logger/logger.go#L153-L182)
+function that is called when a new logger is created. Its default
+configuration is:
+ - 10Mb per log file
+ - Maximum of 7 log files
+ - Rotated on startup
+ - ECS/JSON encoded
+ - UTC timestamps
+
+## Default logging
+The default logger is the easiest to discover because it's user
+configurable, logs to the Agent's root directory and can output to
+`stderr`. It's default configuration comes from
+https://github.com/elastic/elastic-agent/blob/574aa5db629231d56062ab40d27ccceb02cbbe4d/pkg/core/logger/logger.go#L132-L148
+and defaults to:
+ - 20Mb per log file
+ - Maximum of 7 log files
+ - Rotated on startup
+ - ECS/JSON are not explicitly set
+
+## Collecting logs for diagnostics
+The Elastic-Agent will only collect
+`data/elastic-agent-<hash>/logs`. The functions responsible for
+collecting logs during diagnostics are:
+ - [`zipLogs`](https://github.com/elastic/elastic-agent/blob/574aa5db629231d56062ab40d27ccceb02cbbe4d/internal/pkg/diagnostics/diagnostics.go#L383-L415)
+ - [`zipLogsWithPath`](https://github.com/elastic/elastic-agent/blob/574aa5db629231d56062ab40d27ccceb02cbbe4d/internal/pkg/diagnostics/diagnostics.go#L418-L476)
+
+`zipLogsWithPath` always appends the `/logs` to whatever path it
+receives.
+
+## Total footprint
+Given the two log outputs and their default log rotation policies, the
+Elastic-Agent needs about 210Mb (20Mb x 7 + 10Mb + 7 = 210Mb) of disk
+for logging.