feat: Dedicated --show-warnings command for test runner #35464
Conversation
| @@ -134,4 +140,82 @@ export class Runner { | |||
| ]); | |||
| return { status }; | |||
| } | |||
|
|
|||
| async printWarnings(lastRun: LastRunReporter) { | |||
There was a problem hiding this comment.
I'm not really sure this is the right place for this to live. This is where the other test parsing paths live, so I put it here for now, but I'm open to suggestions on where to move it.
Test results for "tests 1"4 failed 3 flaky39017 passed, 805 skipped Merge workflow run. |
packages/playwright/src/program.ts
Outdated
| @@ -415,6 +425,7 @@ const testOptions: [string, string][] = [ | |||
| ['--reporter <reporter>', `Reporter to use, comma-separated, can be ${builtInReporters.map(name => `"${name}"`).join(', ')} (default: "${defaultReporter}")`], | |||
| ['--retries <retries>', `Maximum retry count for flaky tests, zero for no retries (default: no retries)`], | |||
| ['--shard <shard>', `Shard tests and execute only the selected shard, specify in the form "current/all", 1-based, for example "3/5"`], | |||
| ['--show-warnings', `Show all warning messages collected in the last test run. Does not execute tests.`], | |||
There was a problem hiding this comment.
- I wonder whether we want a mode where warnings are printed inline, and also fail the test run, for the strict mode on CI.
- How many warnings do we get for
npm run ctest?
There was a problem hiding this comment.
I'm not seen any unawaited promise warnings anywhere in our repo. Just reran ctest to verify and no warnings were captured.
| const testRun = new TestRun(this._config, reporter); | ||
| const status = await runTasks(testRun, [ | ||
| ...createPluginSetupTasks(this._config), | ||
| createLoadTask('in-process', { failOnLoadErrors: true, filterOnly: false }) |
There was a problem hiding this comment.
Hmm... If we store warnings in .last-run.json, why do we need to load any tests? I'd think we have all the information needed for the warning in the json, so we can just print it? If so, I'd put all this logic into a separate file warnings.ts and let it manage both the "warnings data in last-run.json" and the terminal output.
That said, I think we should make warnings conveniently available to other reporters. For example, do we want to show warning per test result in the html reporter, or do we want a similar summary? If we want a summary, perhaps generate it and attach? Needs brainstorming.
There was a problem hiding this comment.
Storing all of the necessary test information is valid, but I didn't want to deviate significantly from the current patterns. If we did store more information than just test ID, would we store code frames? Or would we just store the test name and still fetch the code frame at runtime?
You don't consider the annotation reporter API to be sufficient? A reporter can read all warning annotations onEnd.
packages/playwright/src/program.ts
Outdated
| const lastRun = new LastRunReporter(config); | ||
| const runner = new Runner(config, lastRun); | ||
|
|
||
| if (opts.showWarnings) { |
There was a problem hiding this comment.
If we go with this approach:
- handle
--show-warningsbefore UI mode; - perhaps do not filter projects, files, etc;
- throw when
--show-warningsis used together with--list,--uiorPWTEST_WATCH.
| @@ -41,7 +46,8 @@ export class FloatingPromiseScope { | |||
| } | |||
| }); | |||
|
|
|||
| this._floatingCalls.add(promise); | |||
| const location = filteredLocation(captureRawStack()); | |||
There was a problem hiding this comment.
- Let's pass the location from outside. I am pretty sure we do already capture location for all the callers.
- I see that
filteredLocationfunction has sneaked in, unused! Let's remove it!
Introduces a dedicated
--show-warningscommand to the test runner. This concludes the initial pass of #35443 for v1.52.Every test run only shows a single line with information about warnings:
There is now a
--show-warningsoption forplaywright testthat displays all warnings in the last test run. It stores this information in the same.last-run.jsonas the suite failures. Warning listings are deduped and codeframes are shown, when available: