diff --git a/docs/source/command_line.rst b/docs/source/command_line.rst index b455e287017e..9d84531984b9 100644 --- a/docs/source/command_line.rst +++ b/docs/source/command_line.rst @@ -1172,6 +1172,13 @@ Miscellaneous stubs at the end of the run, but only if any missing modules were detected. + It is not recommended to use this option in `CI/CD + `_, as it will make your + dependencies less reproducible. Instead, you should require and + install type dependencies like you would any other (test) dependency, + such as by using a dependency section in your `pyproject.toml + `_. + .. note:: This is new in mypy 0.900. Previous mypy versions included a @@ -1192,6 +1199,10 @@ Miscellaneous stub packages were found, they are installed and then another run is performed. + It is not recommended to use ``--install-types --non-interactive`` + in `CI/CD `_; see the other + flag for more details. + .. option:: --junit-xml JUNIT_XML Causes mypy to generate a JUnit XML test result document with diff --git a/docs/source/error_code_list.rst b/docs/source/error_code_list.rst index 49cb8a0c06c1..b4c8442f855b 100644 --- a/docs/source/error_code_list.rst +++ b/docs/source/error_code_list.rst @@ -640,7 +640,8 @@ Check for an issue with imports [import] ---------------------------------------- Mypy generates an error if it can't resolve an `import` statement. -This is a parent error code of `import-not-found` and `import-untyped` +This is a parent error code of `import-not-found`, `import-untyped`, +and `import-untyped-stubs-available` See :ref:`ignore-missing-imports` for how to work around these errors. @@ -664,7 +665,7 @@ See :ref:`ignore-missing-imports` for how to work around these errors. .. _code-import-untyped: Check that import target can be found [import-untyped] --------------------------------------------------------- +------------------------------------------------------ Mypy generates an error if it can find the source code for an imported module, but that module does not provide type annotations (via :ref:`PEP 561 `). @@ -673,7 +674,7 @@ Example: .. code-block:: python - # Error: Library stubs not installed for "bs4" [import-untyped] + # Error: Library stubs not installed for "bs4" [import-untyped-stubs-available] import bs4 # Error: Skipping analyzing "no_py_typed": module is installed, but missing library stubs or py.typed marker [import-untyped] import no_py_typed @@ -681,6 +682,27 @@ Example: In some cases, these errors can be fixed by installing an appropriate stub package. See :ref:`ignore-missing-imports` for more details. +.. _code-import-untyped-stubs-available: + +Check that import target with known stubs can be found [import-untyped-stubs-available] +--------------------------------------------------------------------------------------- + +Like :ref:`code-import-untyped`, but used when mypy knows there is an appropriate +type stub package corresponding to the library, which you could install. + +Example: + +.. code-block:: python + + # Error: Library stubs not installed for "bs4" [import-untyped-stubs-available] + import bs4 + # Error: Skipping analyzing "no_py_typed": module is installed, but missing library stubs or py.typed marker [import-untyped] + import no_py_typed + +These errors can be fixed by installing the appropriate +stub package. See :ref:`ignore-missing-imports` for more details. + + .. _code-no-redef: Check that each name is defined once [no-redef] diff --git a/docs/source/running_mypy.rst b/docs/source/running_mypy.rst index 9f7461d24f72..2cc2983f309e 100644 --- a/docs/source/running_mypy.rst +++ b/docs/source/running_mypy.rst @@ -382,26 +382,29 @@ the library, you will get a message like this: main.py:1: note: (or run "mypy --install-types" to install all missing stub packages) You can resolve the issue by running the suggested pip commands. + If you're running mypy in CI, you can ensure the presence of any stub packages you need the same as you would any other test dependency, e.g. by adding them to -the appropriate ``requirements.txt`` file. +the appropriate ``requirements.txt`` file or dependency section of ``pyproject.toml``. -Alternatively, add the :option:`--install-types ` -to your mypy command to install all known missing stubs: +The :option:`--install-types ` flag +makes mypy list and (after a prompt) install all known missing stubs: .. code-block:: text mypy --install-types This is slower than explicitly installing stubs, since it effectively -runs mypy twice -- the first time to find the missing stubs, and +runs mypy twice — the first time to find the missing stubs, and the second time to type check your code properly after mypy has installed the stubs. It also can make controlling stub versions harder, -resulting in less reproducible type checking. +resulting in less reproducible type checking — it might even install +incompatible versions of your project's non-type dependencies, if the +type stubs require them! By default, :option:`--install-types ` shows a confirmation prompt. Use :option:`--non-interactive ` to install all suggested -stub packages without asking for confirmation *and* type check your code: +stub packages without asking for confirmation *and* then type check your code. If you've already installed the relevant third-party libraries in an environment other than the one mypy is running in, you can use :option:`--python-executable diff --git a/mypy/build.py b/mypy/build.py index 355ba861385e..874c06f87147 100644 --- a/mypy/build.py +++ b/mypy/build.py @@ -2780,11 +2780,10 @@ def module_not_found( msg, notes = reason.error_message_templates(daemon) if reason == ModuleNotFoundReason.NOT_FOUND: code = codes.IMPORT_NOT_FOUND - elif ( - reason == ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS - or reason == ModuleNotFoundReason.APPROVED_STUBS_NOT_INSTALLED - ): + elif reason == ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS: code = codes.IMPORT_UNTYPED + elif reason == ModuleNotFoundReason.APPROVED_STUBS_NOT_INSTALLED: + code = codes.IMPORT_UNTYPED_STUBS_AVAILABLE else: code = codes.IMPORT errors.report(line, 0, msg.format(module=target), code=code) diff --git a/mypy/errorcodes.py b/mypy/errorcodes.py index 8f650aa30605..0d8c4eb72b94 100644 --- a/mypy/errorcodes.py +++ b/mypy/errorcodes.py @@ -34,6 +34,9 @@ def __init__( sub_code_map[sub_code_of.code].add(code) error_codes[code] = self + def is_import_related_code(self) -> bool: + return IMPORT in (self.code, self.sub_code_of) + def __str__(self) -> str: return f"" @@ -113,6 +116,12 @@ def __hash__(self) -> int: IMPORT_UNTYPED: Final = ErrorCode( "import-untyped", "Require that imported module has stubs", "General", sub_code_of=IMPORT ) +IMPORT_UNTYPED_STUBS_AVAILABLE: Final = ErrorCode( + "import-untyped-stubs-available", + "Require that imported module (with known stubs) has stubs", + "General", + sub_code_of=IMPORT, +) NO_REDEF: Final = ErrorCode("no-redef", "Check that each name is defined once", "General") FUNC_RETURNS_VALUE: Final = ErrorCode( "func-returns-value", "Check that called function returns a value in value context", "General" diff --git a/mypy/errors.py b/mypy/errors.py index c9510ae5f1eb..4b69ddee6d1a 100644 --- a/mypy/errors.py +++ b/mypy/errors.py @@ -10,7 +10,7 @@ from mypy import errorcodes as codes from mypy.error_formatter import ErrorFormatter -from mypy.errorcodes import IMPORT, IMPORT_NOT_FOUND, IMPORT_UNTYPED, ErrorCode, mypy_error_codes +from mypy.errorcodes import ErrorCode, mypy_error_codes from mypy.options import Options from mypy.scope import Scope from mypy.util import DEFAULT_SOURCE_OFFSET, is_typeshed_file @@ -476,7 +476,7 @@ def _add_error_info(self, file: str, info: ErrorInfo) -> None: self.error_info_map[file].append(info) if info.blocker: self.has_blockers.add(file) - if info.code in (IMPORT, IMPORT_UNTYPED, IMPORT_NOT_FOUND): + if info.code is not None and info.code.is_import_related_code(): self.seen_import_error = True def _filter_error(self, file: str, info: ErrorInfo) -> bool: @@ -522,7 +522,7 @@ def add_error_info(self, info: ErrorInfo) -> None: self.only_once_messages.add(info.message) if ( self.seen_import_error - and info.code not in (IMPORT, IMPORT_UNTYPED, IMPORT_NOT_FOUND) + and (info.code is None or (not info.code.is_import_related_code())) and self.has_many_errors() ): # Missing stubs can easily cause thousands of errors about diff --git a/mypy/report.py b/mypy/report.py index 39cd80ed38bf..f90ca3333732 100644 --- a/mypy/report.py +++ b/mypy/report.py @@ -26,7 +26,7 @@ from mypy.version import __version__ try: - from lxml import etree # type: ignore[import-untyped] + from lxml import etree LXML_INSTALLED = True except ImportError: diff --git a/mypy/test/testcheck.py b/mypy/test/testcheck.py index e6415ddff906..27d9311a1255 100644 --- a/mypy/test/testcheck.py +++ b/mypy/test/testcheck.py @@ -5,6 +5,9 @@ import os import re import sys +from importlib.util import find_spec as import_exists + +import pytest from mypy import build from mypy.build import Graph @@ -24,14 +27,6 @@ ) from mypy.test.update_data import update_testcase_output -try: - import lxml # type: ignore[import-untyped] -except ImportError: - lxml = None - - -import pytest - # List of files that contain test case descriptions. # Includes all check-* files with the .test extension in the test-data/unit directory typecheck_files = find_test_files(pattern="check-*.test") @@ -55,7 +50,7 @@ class TypeCheckSuite(DataSuite): files = typecheck_files def run_case(self, testcase: DataDrivenTestCase) -> None: - if lxml is None and os.path.basename(testcase.file) == "check-reports.test": + if not import_exists("lxml") and os.path.basename(testcase.file) == "check-reports.test": pytest.skip("Cannot import lxml. Is it installed?") incremental = ( "incremental" in testcase.name.lower() diff --git a/mypy/test/testcmdline.py b/mypy/test/testcmdline.py index 9bc02d319964..eea6ed4519fc 100644 --- a/mypy/test/testcmdline.py +++ b/mypy/test/testcmdline.py @@ -10,6 +10,9 @@ import re import subprocess import sys +from importlib.util import find_spec as import_exists + +import pytest from mypy.test.config import PREFIX, test_temp_dir from mypy.test.data import DataDrivenTestCase, DataSuite @@ -19,13 +22,6 @@ normalize_error_messages, ) -try: - import lxml # type: ignore[import-untyped] -except ImportError: - lxml = None - -import pytest - # Path to Python 3 interpreter python3_path = sys.executable @@ -38,7 +34,7 @@ class PythonCmdlineSuite(DataSuite): native_sep = True def run_case(self, testcase: DataDrivenTestCase) -> None: - if lxml is None and os.path.basename(testcase.file) == "reports.test": + if not import_exists("lxml") and os.path.basename(testcase.file) == "reports.test": pytest.skip("Cannot import lxml. Is it installed?") for step in [1] + sorted(testcase.output2): test_python_cmdline(testcase, step) diff --git a/mypy/test/testreports.py b/mypy/test/testreports.py index f638756ad819..7ea8cb1cc7fc 100644 --- a/mypy/test/testreports.py +++ b/mypy/test/testreports.py @@ -3,27 +3,23 @@ from __future__ import annotations import textwrap +from importlib.util import find_spec as import_exists + +import pytest from mypy.report import CoberturaPackage, get_line_rate from mypy.test.helpers import Suite, assert_equal -try: - import lxml # type: ignore[import-untyped] -except ImportError: - lxml = None - -import pytest - class CoberturaReportSuite(Suite): - @pytest.mark.skipif(lxml is None, reason="Cannot import lxml. Is it installed?") + @pytest.mark.skipif(not import_exists("lxml"), reason="Cannot import lxml. Is it installed?") def test_get_line_rate(self) -> None: assert_equal("1.0", get_line_rate(0, 0)) assert_equal("0.3333", get_line_rate(1, 3)) - @pytest.mark.skipif(lxml is None, reason="Cannot import lxml. Is it installed?") + @pytest.mark.skipif(not import_exists("lxml"), reason="Cannot import lxml. Is it installed?") def test_as_xml(self) -> None: - import lxml.etree as etree # type: ignore[import-untyped] + import lxml.etree as etree cobertura_package = CoberturaPackage("foobar") cobertura_package.covered_lines = 21 diff --git a/pyproject.toml b/pyproject.toml index 8a1177f60009..2dc5be78d71f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -62,6 +62,10 @@ python2 = [] reports = ["lxml"] install-types = ["pip"] faster-cache = ["orjson"] +dev = [ + "pip-tools", # Not strictly needed for building per se, but pip-compile from this package is needed if you want to update the requirement files. (Word on the street is you can also use uv pip compile instead.) + "pip<24.3" # Needed to update the requirement files correctly with pip-tools ay ay ay https://github.com/jazzband/pip-tools/issues/2131 +] [project.urls] Homepage = "https://www.mypy-lang.org/" diff --git a/test-data/unit/check-errorcodes.test b/test-data/unit/check-errorcodes.test index 21112b7d85a2..af302a1755a3 100644 --- a/test-data/unit/check-errorcodes.test +++ b/test-data/unit/check-errorcodes.test @@ -535,7 +535,10 @@ if int() is str(): # E: Non-overlapping identity check (left operand type: "int [builtins fixtures/primitives.pyi] [case testErrorCodeMissingModule] -from defusedxml import xyz # E: Library stubs not installed for "defusedxml" [import-untyped] \ +# Note: it was too difficult for me to figure out how to test [import-untyped] here, +# (ideally, it would!) +# but testNamespacePkgWStubs does test that, anyway. +from defusedxml import xyz # E: Library stubs not installed for "defusedxml" [import-untyped-stubs-available] \ # N: Hint: "python3 -m pip install types-defusedxml" \ # N: (or run "mypy --install-types" to install all missing stub packages) from nonexistent import foobar # E: Cannot find implementation or library stub for module named "nonexistent" [import-not-found] diff --git a/test-requirements.in b/test-requirements.in index 666dd9fc082c..0c2007828c8e 100644 --- a/test-requirements.in +++ b/test-requirements.in @@ -6,6 +6,7 @@ attrs>=18.0 filelock>=3.3.0 lxml>=5.3.0; python_version<'3.14' +lxml-stubs psutil>=4.0 pytest>=8.1.0 pytest-xdist>=1.34.0 diff --git a/test-requirements.txt b/test-requirements.txt index 51281f0e4c11..6dae2b676c83 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -8,6 +8,8 @@ attrs==25.1.0 # via -r test-requirements.in cfgv==3.4.0 # via pre-commit +colorama==0.4.6 + # via pytest coverage==7.6.10 # via pytest-cov distlib==0.3.9 @@ -24,6 +26,8 @@ iniconfig==2.0.0 # via pytest lxml==5.3.0 ; python_version < "3.14" # via -r test-requirements.in +lxml-stubs==0.5.1 + # via -r test-requirements.in mypy-extensions==1.0.0 # via -r mypy-requirements.txt nodeenv==1.9.1