Not sending the application binary debug meta image info

[lfdominguez]

Environment

Version: 0.36
Features: anyhow debug-images

Steps to Reproduce

1. Sent (using upload-dif) the application debug info to sentry

2. Send a report (force a fail on the application)

Expected Result

On the Stack Trace of the Issue show the debug information, like line code, etc.

Actual Result

Already opened an issue on getsentry/sentry#88294 referring to this problem, but there detected that sentry-rust is not sending all the required debug meta images:

It's only sending the native library info, but is missing the binary of the program itself, so without that info sentry can't relate the debug info with each frame.

[lcian]

Thanks for reporting this, we'll take a look soon.

[lcian]

@lfdominguez It would be helpful if you could provide more information on your platform (OS, kernel version, architecture) to understand when this happens.
It's highly likely that this is a bug in https://github.com/gimli-rs/findshlibs that we use to detect library and binary info.
You could also try running the example in that repo and see if it detects the binary itself or not.

[lfdominguez]

Thanks for response, I will check the findshlibs library and share the results.

OS:

• Tested on Ubuntu 22.04 using a Docker Image built with nix (that image only contain the executable and dependencies)
• NixOS 24.11

Arch: amd64
Kernel: 6.13.7

[lfdominguez]

Hi, tested the 0.10.2 version and get correct results:

Cargo.tom:

[package]
name = "test-shlib"
version = "0.1.0"
edition = "2021"

[dependencies]
findshlibs = "0.10.2"

main.rs

use findshlibs::{Segment, SharedLibrary, TargetSharedLibrary};

fn main() {
    TargetSharedLibrary::each(|shlib| {
        println!("{}", shlib.name().to_string_lossy());

        for seg in shlib.segments() {
            println!("    {}: segment {}",
                     seg.actual_virtual_memory_address(shlib),
                     seg.name().to_string());
        }
    });
}

Result:

[lfdominguez]

And on my software it's working great too:

[lfdominguez]

Now i will try using the same code found here on https://github.com/getsentry/sentry-rust/blob/6401cfc9dc2917bc61882c571ee8970c07939b9b/sentry-debug-images/src/images.rs

[lfdominguez]

and is working:

like you can see the program itself is detected and printed, I used exactly the same code

[lcian]

Thanks @lfdominguez for all the context.
Could try with a release build as well?
There's some cases where that crate could behave differently depending on the type of build.

[lcian]

From the last screenshot you sent it looks like the debug_images() function works correctly in this case, at least in what I assume is a debug build.

If that function works, then the images are also going to be sent to Sentry, as the SDK just uses that function and sends whatever is returned along.
You can always double check by setting a before_send in your ClientOptions like this:

            before_send: Some(Arc::new(|event| {
                dbg!(&event); // or println
                Some(event)
            })),

And inspecting the payload.

[lfdominguez]

This is the test in release:

This is my app in release with debug=true:

[profile.release]
opt-level = 's'  # Optimize for speed.
lto = true
debug = true
codegen-units = 1
panic = 'abort'

[lfdominguez]

From the last screenshot you sent it looks like the debug_images() function works correctly in this case, at least in what I assume is a debug build.

If that function works, then the images are also going to be sent to Sentry, as the SDK just uses that function and sends whatever is returned along. You can always double check by setting a before_send in your ClientOptions like this:

        before_send: Some(Arc::new(|event| {
            dbg!(&event); // or println
            Some(event)
        })),

And inspecting the payload.

I will try this, to check that is sending the right event, thanks

[lfdominguez]

Humm this is really strange, now it's working, sending the right image of my app. Let me check deployed on production server to see if environment change and get the wrong event.

[lfdominguez]

Yes on production deployment is missing the info of app itself:

[lfdominguez]

The docker image is an optimized one built with Nix:

Only has the really necessary for run the app

[lfdominguez]

I need to check later if the problem is Nix itself, because when you compile something with nix, it's use patchelf to link directly the dependencies, maybe there is the problem

[lcian]

It could be, the first thing I would check is just findshlibs in isolation as you did above so that we can be certain the problem is there.
We need to decide what to do regarding that library because it's unmaintained and it's also causing other problems #358 (comment)

[lfdominguez]

Results of checking, compiled directly with Nix, even building the docker image and both artifacts findshlibs show the software itself. Now going to test the app in production and check if is inside Docker or what....

[lfdominguez]

@lcian found finally a reproducible case, I created the production docker image of my app, printing the findshlibs libs and print OK my app binary:

But activated the sentry events and on logs, sentry is not sending my app binary:

At the same execution findshlibs show my app, but sentry-rust exclude that entry, there is some additional check by sentry-rust excluding my app findshlibs entry?

[lcian]

Thanks for the thorough testing 🙏
I'll check!

[lcian]

We're not excluding any of the images found by findshlibs.
However, we are initing the debug meta like this in its own event processor:

        static DEBUG_META: Lazy<DebugMeta> = Lazy::new(|| DebugMeta {
            images: crate::debug_images(),
            ..Default::default()
        });

The only possibility is that when this Lazy is inited (during the event processor, so during the first time you send an event), for some reason the binary cannot be detected anymore by findshlibs.
So I think the solution should be to force the Lazy to init when the integration itself is registered instead of having it inizialize lazily.

I'm assuming you're currently checking with a panic, right?
If you could please do a last check, please try with sentry::capture_message("something", Level::ERROR) and let's see if in the corresponding payload the binary debug meta is there.

[lfdominguez]

Hi, nop i was testing in this case with a tracing::error! but i will try with sentry::capture_message... and share the results

[lfdominguez]

Results:
Changed my main to this one

fn main() -> anyhow::Result<()> {
    let orig_hook = panic::take_hook();
    panic::set_hook(Box::new(move |panic_info| {
        // invoke the default handler and exit the process
        orig_hook(panic_info);
        process::exit(1);
    }));

    let sentry_dsn = Dsn::from_str(&globals::APP.config.sentry.sentry_dsn.clone());

    if sentry_dsn.is_err() {
        tracing::error!("Malformed Sentry DSN URI");
    }

    let sentry = sentry::init(sentry::ClientOptions {
        release: sentry::release_name!(),
        auto_session_tracking: true,
        traces_sample_rate: 0.1,
        attach_stacktrace: true,
        session_mode: sentry::SessionMode::Request,
        debug: false,

        // For debug event sent to sentry
        before_send: Some(Arc::new(|event| {
            dbg!(&event); // or println
            Some(event)
        })),

        dsn: sentry_dsn.ok(),
        ..Default::default()
    });

    TargetSharedLibrary::each(|shlib| {
        println!("{}", shlib.name().to_string_lossy());

        for seg in shlib.segments() {
            println!("    {}: segment {}",
                     seg.actual_virtual_memory_address(shlib),
                     seg.name().to_string());
        }
    });

    sentry::capture_message("Testing libs", Level::Error);

    return Ok(());
}

And same behavior, findshlibs show the app:

But the event missing that one:

[lcian]

I'm not entirely sure what's happening but I think this might fix it #773.
Please give it a try @lfdominguez 🙏

[lfdominguez]

Testing

[lfdominguez]

Bad news, not working, using this:

sentry = { git = "https://github.com/getsentry/sentry-rust.git", rev = "refs/pull/773/head", features = ["anyhow", "debug-images"] }
sentry-actix = "0.37"
sentry-tracing = "0.37"

I will do something, I will put a debug build inside the docker image and add a gdb, then I will run the gdb in remote mode and control from my IDE to debug directly to check where is losing this entry.

[lfdominguez]

Finally found the issue, findhslib can't get the debug id from executable

And in debug_images:

skip if not found a debug_id, that's why is skipping that, now I need to search who is stripping that debug info from executable, because I think the compiler is not setting gnu build id, need to check...

[lfdominguez]

and is working:

like you can see the program itself is detected and printed, I used exactly the same code

Need to check here how this worked here, maybe is because i was using a remote compiler and has some different defaults??? hummmm

[lfdominguez]

And this RUSTFLAGS="-Clink-args=-Wl,--build-id" fix the issue and force the linker on Rust to set a build-id:

[lfdominguez]

@lcian I will close this issue (thanks for all the investigation and support from your side) because at the end is not a sentry-rust issue, it's a cargo/rustc/linker default behavior issue about not saving the build-id on the executable.
For anyone that require this functionality you can:

• Use RUSTFLAGS="-Clink-args=-Wl,--build-id" env var before the cargo run/build
• Add a new file config.toml inside .cargo on the project root or in $CARGO_HOME (now I'm using both bethods) to set to all cargo executions with the content:

[build]
rustflags = [ "-Clink-args=-Wl,--build-id" ]

[lcian]

Hey @lfdominguez, thank you so much for all the help and cooperation from your side.
It's good to know the root cause, we can also add it to the docs :)