[red-knot] add ecosystem check

[carljm]

Description

We regularly observe the effects of changes to red-knot in the diagnostics emitted when running our (performance) benchmark against tomllib, and sometimes catch issues by seeing new false positives. But a) tomllib is one small codebase, not necessarily representative, and b) the ergonomics of catching these issues in the performance benchmark is poor.

We should select some larger real-world codebases and implement tooling to run red-knot on them and snapshot the diagnostics emitted, such that we can see (and easily update) this snapshot when we make changes to red-knot.

We should run this in CI to ensure our snapshot stays up to date.

(This is very similar to the ruff ecosystem check, and we can likely reuse some of that infrastructure?)

Bonus: if we implement #15696, we can also track changes to the prevalence of Todo types over time.

[sharkdp]

Just noting down some thoughts while working on this:

• It would help to have a concise (or structured) diagnostics diagnostics format. mypy_primer uses a simple line-based diff (without context) and it turns out that diffs between two rich-diagnostics red-knot outputs are … not great
  

  • This is planned in [red-knot] Different diagnostic output formats #14194

• We do still panic on a lot of the projects included in mypy_primer's list

  • This will likely be fixed by [red-knot] use fixpoint iteration for all cycles #14029

• Support for importing from -stubs packages is another thing that would make the ecosystem check more helpful.

  • This is planned in [red-knot] PEP 561 compliant module resolver #11653 (comment)

[sharkdp]

While selecting some projects from mypy_primer, I noted down some issues that appear very frequently across multiple codebases. If we want to work on reducing false positives diagnostics, those would likely be the candidates with the biggest impact:

• Panics due to salsa cycles => [red-knot] use fixpoint iteration for all cycles #14029
• Importing from -stubs packages => [red-knot] support -stubs packages in module resolver #16612
• lint:unresolved-import and subsequent lint:unresolved-attribute due to *-imports from e.g. collections.abc or asyncio => [red-knot] support * imports #14169
• Consider __all__ for re-export convention => [red-knot] Consider __all__ for re-export convention #16159
• Assignability of intersection types (results in e.g. lint:invalid-argument-type) => [red-knot] (Gradual) intersection types are not handled in assignability #14899
• Incorrect lint:unresolved-reference in the presence of unreachable code => [red-knot] How should we handle unreachable code? #15797
• lint:unresolved-attribute due to custom __getattr__, e.g. argparse.Namespace => [red-knot] support classes defining __getattr__ #16614
• lint:unused-ignore-comment => Some of these could be a signal for false negatives? Others could just be an indication that we might need a feature to silence those unused generic # type: ignore comments
• lint:unresolved-attribute because we don't understand super => [red-knot] support super #16615
• Missing understanding of @property => [red-knot] support properties #16616