-
Notifications
You must be signed in to change notification settings - Fork 95
Description
Since the addition of the prepare-fail category a lot more spurious results that could hide actual regressions can be retired, but the result comment does not inform about spurious results which I think might lead to users of crater bot to think a result is good even though hundreds thousand crates have an indeterminate result and might deserve a retry. Adding the amount of crates on the retry-regressed-list.txt (maybe with an explanation of how to use it) would probably help to more often run a retry when a lot of spurious failures are encountered.
Current Comment Example rust-lang/rust#145608 (comment):
🎉 Experiment
pr-145608
is completed!
📊 6 regressed and 8 fixed (685389 total)
📰 Open the summary report.
⚠️ If you notice any spurious failure please add them to the denylist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more
Suggestion
🎉 Experiment
pr-145608
is completed!
📊 6 regressed and 8 fixed (685389 total)
📊 155079 Spurious results, consider a retry withcrater <mode> crates=https://crater-reports.s3.amazonaws.com/pr-145608/retry-regressed-list.txt
if these are many.
📰 Open the summary report.
⚠️ If you notice any spurious failure please add them to the denylist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more
#763 would make the retry command easier.