Skip to content

Should the crater-bot result comment contain the amount of crates on the retry-regressed-list.txt? #791

@Skgland

Description

@Skgland

Since the addition of the prepare-fail category a lot more spurious results that could hide actual regressions can be retired, but the result comment does not inform about spurious results which I think might lead to users of crater bot to think a result is good even though hundreds thousand crates have an indeterminate result and might deserve a retry. Adding the amount of crates on the retry-regressed-list.txt (maybe with an explanation of how to use it) would probably help to more often run a retry when a lot of spurious failures are encountered.

Current Comment Example rust-lang/rust#145608 (comment):

🎉 Experiment pr-145608 is completed!
📊 6 regressed and 8 fixed (685389 total)
📰 Open the summary report.

⚠️ If you notice any spurious failure please add them to the denylist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

Suggestion

🎉 Experiment pr-145608 is completed!
📊 6 regressed and 8 fixed (685389 total)
📊 155079 Spurious results, consider a retry with crater <mode> crates=https://crater-reports.s3.amazonaws.com/pr-145608/retry-regressed-list.txt if these are many.
📰 Open the summary report.

⚠️ If you notice any spurious failure please add them to the denylist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

#763 would make the retry command easier.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions