--strict-equality for None

[tyralla]

Fixes #18386 (at least partly)

(edited)

In a first test run, in which I included checks against None in --strict-equality, the Mypy primer gave hundreds of new comparison-overlap reports. Many of them seem really helpful (including those for the Mypy source code itself), because it is often hard to tell if non-overlapping None checks are just remnants of incomplete refactorings or can handle cases with corrupted data or similar issues. As it was only a little effort, I decided to add the option --strict-equality-for-none to Mypy, which is disabled even in --strict mode. Other libraries could adjust to this new behaviour if and when they want. If many of them do so, we could eventually enable --strict-equality-for-none in --strict mode or even merge it with --strict-equality later.

The remaining new true positives revealed by the Mypy primer are the result of no longer excluding types with custom __eq__ methods for identity checks (which, in my opinion, makes sense even in case --strict-equality-for-none would be rejected).

[hauntsaninja]

Thanks, this makes sense to me!

[TeamSpen210]

Would it be a good idea to add a warning/error if --strict-equality-for-none is set, but not --strict-equality? Since that's not going to do anything.

[tyralla]

Would it be a good idea to add a warning/error if --strict-equality-for-none is set, but not --strict-equality? Since that's not going to do anything.

I thought about this too, but did not take the time to check how Mypy reacts in similar cases. So thanks for the soft push.

First experiment: selecting only --allow-redefinition-new results in error: --local-partial-types must be enabled if using --allow-redefinition-new

I will implement a similar check later today.

[ilevkivskyi]

Would it be a good idea to add a warning/error if --strict-equality-for-none is set, but not --strict-equality? Since that's not going to do anything.

Not a strong opinion, but IIUC these are per-module flags, so I guess some people may want to specify --strict-equality-for-none once, so that it takes effect for all modules where --strict-equality is enabled (without extra noise for modules where it is disabled).

[ilevkivskyi]

This is not the only reason, and probably not the main reason. IMO the main reason is that in large code-bases there is a chance that None can get through the cracks easily, so people often add some assert x is not None in tests. Flagging those test asserts would cause a lot of noise.

[tyralla]

Okay, then I will just remove the "for historical reasons (support for type comments)" part from this sentence.

By the way: I temporarily played with skipping None checks in assert statements, and I believe Pyright avoids all non-overlap checks in assertions. However, one might sometimes be happy if assertions are checked in this way, and adding (explained) type: ignores in such cases (or disabling --strict-quality-for-none specific modules) does not seem too much effort. That's why I dropped the idea again.

[tyralla]

I have now removed it here and a similar clause in command_line.rst.

[tyralla]

Would it be a good idea to add a warning/error if --strict-equality-for-none is set, but not --strict-equality? Since that's not going to do anything.

Not a strong opinion, but IIUC these are per-module flags, so I guess some people may want to specify --strict-equality-for-none once, so that it takes effect for all modules where --strict-equality is enabled (without extra noise for modules where it is disabled).

Oh yes, good point. Considering this, I would also tend to avoid throwing an error. At least, it is clearly documented behaviour.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

aiohttp (https://github.com/aio-libs/aiohttp)
+ aiohttp/client_reqrep.py:147:48: error: Non-overlapping identity check (left operand type: "URL", right operand type: "type[_SENTINEL]")  [comparison-overlap]
+ aiohttp/client_reqrep.py:147:48: note: See https://mypy.rtfd.io/en/stable/_refs.html#code-comparison-overlap for more info

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/arrays/datetimes.py:2958: error: Non-overlapping identity check (left operand type: "Timestamp", right operand type: "NaTType")  [comparison-overlap]
+ pandas/core/arrays/datetimes.py:2966: error: Non-overlapping identity check (left operand type: "Timestamp", right operand type: "NaTType")  [comparison-overlap]
+ pandas/core/arrays/arrow/array.py:703: error: Non-overlapping identity check (left operand type: "int | integer[Any] | slice[Any, Any, Any] | list[int]", right operand type: "EllipsisType")  [comparison-overlap]
+ pandas/core/arrays/sparse/array.py:972: error: Non-overlapping identity check (left operand type: "int | integer[Any] | slice[Any, Any, Any] | list[int] | ndarray[tuple[Any, ...], dtype[Any]] | tuple[int | ellipsis, ...]", right operand type: "ellipsis")  [comparison-overlap]
+ pandas/io/json/_json.py:1209: error: Non-overlapping identity check (left operand type: "ExtensionDtype | str | dtype[Any] | type[object] | Mapping[Hashable, ExtensionDtype | str | dtype[Any] | type[str] | type[complex] | type[bool] | type[object]]", right operand type: "Literal[True]")  [comparison-overlap]

[ilevkivskyi]

OK, in general the bar for a new flag is quite high, but this is something that would look worse with a e.g. dedicated error code.