Proposal to integrate L3 tooling with LOC-tooling, for improved code-level troubleshooting.

[gapisback]

Hi, @gregthelaw -- This l3 tooling is very good and easy to use.

Thanks for opening up your repo and this Discussions page!

I would like to propose and explore how this could be integrated with another tooling I've been working on, which I call LOC: Line-of-Code tracking. The git-repo is here, and this Workflow document will explain how that technique can be absorbed into an existing code-base.

The basic idea for integrating the two tools is as follows:

Getting the code-location into L3's log-entry structure:

1. LOC provides a Python script which is used to process all the .c/ .c++ files in your repo of any project that you want to annotate debugging with LOC information.

LOC Python script generates 3 files which then need-to-be built along with the sources of the project: loc.h, loc_tokens.h, loc_filenames.c. All this stuff will need to come along with the L3 package.

2. L3's l3.h will <#include> loc.h, so the access to LOC-macros is hidden away from the user. The end-user only has to know and care about l3.h.

3. Your existing l3_log_simple() and l3_log_fast() interfaces, which are currently functions, will [probably] change to become caller-macros as follows. E.g.

#define l3_log_simple() -> l3_log__simple(__LOC__, ... )
#define l3_log_fast()   -> l3_log__fast(__LOC__, ... )

4. The dunder-functions [l3_log__simple() and l3_log__fast() ] will receive a __LOC__ synthesized by the caller-macros.

NOTE: __LOC__ is a #define whose value will be generated at compile-time by the LOC-machinery.

This is how we generate and pass-down the code-location 4-byte ID to the lower level L3 functions.

5. The core L3 tracking struct struct l3_entry{} will store the loc_t loc, 4-byte quantity, right after the pid_t tid field. There is already a spare 4-bytes in this entry structure, so there will be no footprint bloat of your log info.

6. Stashing off the loc_ in the tracking structure by the slow method is easy. For the fast logging, you will need to enhance the assembly code to save this off in the appropriate place, like it's being done for pid.

Unpacking the code-location from the log-entry by l3_dump.py:

Unpacking the loc field needs access to the generated loc_filenames.c, which we expect will be linked into the user's binary. There are couple of approaches here and need some design work:

1. LOC also generates a stand-alone tool, loc, as part of its build process. A version of the loc tool is specific to the project's code-base which is being processed by LOC. For instance, in, say, PostgreSQL repo, you may have multiple versions of this program like postgres_loc, pg_backup_loc, pg_replication_loc and so on. That means each component of the overall project has its own LOC-deciphering stand-alone loc tool as indicated by the s/w component's name.

This loc tool will unpack a valid 4-byte ID-value to its constituent file name / line # pair. One option is to have l3_dump.py repeatedly call the loc program to unpack each of the loc_t ID-values found in the traces.

2. Another option is for l3_dump.py to somehow get access to the Filenames[] array that is compiled-into the binary, and lookup the file-name given its index.

The 1st approach is easier to do, but creates a dependency on having the loc stand-alone tool available to the end-user as part of the release / build area.

---

I think there is a good synergy between the two tools and would like to work with you to take this idea forward.

[gapisback]

HI, @gregthelaw - I have implemented most of this integration proposal and it's in review, with PR #8 assigned to you.

The only thing left is to address [2], how-to unpack the encoded LOC-ID value to its filename / line#. To get that working, I need to re-do some Python machinery on LOC side, which I will push separately to that repo.

Then, I will come back to this repo to finish-up the unpacking effort. That's well-understood, so this discussion item need not remain open for me to chase that down.

Once you clear the review assigned to you, this discussion item can be closed.

Hope this helps with tracking the status of this proposal.

[gapisback]

The work in this proposal has been productized and integrated to L3 /main, under diff PRs. See these commits:

019f673 Aditya P. Gurajada 9 days ago Fri, 29-Mar-2024, 05:25:56 PM (Authored: Thu, 21-Mar-2024, 05:22:21 PM)
  [LOC] Decode LOC-ID in L3 data files to report file / line#

23abce0 Aditya P. Gurajada 3 weeks ago Thu, 21-Mar-2024, 04:55:17 PM (Authored: Sun, 10-Mar-2024, 04:47:01 PM)
 [LOC] Integrate with LOC package for improved L3 diagnostic

8283e76 Aditya P. Gurajada 3 weeks ago Wed, 13-Mar-2024, 03:37:55 PM (Authored: Sun, 10-Mar-2024, 04:47:01 PM)
 [Build] Rework Makefile rules to ease addition of LOC package