Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid triggering the DeprecationWarning on os.fork #429

Merged
merged 30 commits into from
Mar 7, 2025

Conversation

ogrisel
Copy link
Contributor

@ogrisel ogrisel commented Feb 27, 2025

Fixes #424, #420.

I added some non-regression tests for each problem and checked they failed on main with Python 3.13.

@ogrisel
Copy link
Contributor Author

ogrisel commented Feb 28, 2025

Since we dropped Python 3.7 support, there is probably a lot of compat code that can be simplified in the resource tracker and the posix popen modules (and maybe the win32 module as well).

@ogrisel
Copy link
Contributor Author

ogrisel commented Feb 28, 2025

Some of the last commits above can be reverted to use #435 instead.

@ogrisel ogrisel marked this pull request as ready for review February 28, 2025 18:46
@ogrisel
Copy link
Contributor Author

ogrisel commented Feb 28, 2025

This should be ready for review @jeremiedbb @tomMoral @lesteve @ryandvmartin @sam-s.

@ogrisel
Copy link
Contributor Author

ogrisel commented Mar 3, 2025

The test_parallel_call_cached_function_defined_in_jupyter failure that happened on 3cd336e (part of the joblib test suite) is unrelated. I will open a PR in joblib to report more info in case of failure.

EDIT: done in joblib/joblib#1683.

else:
passfds = sorted(passfds)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lack of coverage is a false negative. This branch is actually executed when running the resource manager on Windows (to track temporary folders) but we properly do not configure coverage tracking in the resource manager process our our CI.

is_rng = name == "n/dev/urandom"

# Not sure if this really expected to be open or not...
is_dev_null = name == "n/dev/null"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not 100% sure why this is needed with the new code, and it wasn't before, but keeping an open fd on /dev/null sounds harmless.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, when editing the assertion below, I checked that there was only one open fd to /dev/null and not thousands.

@ogrisel
Copy link
Contributor Author

ogrisel commented Mar 6, 2025

@tomMoral this is ready for review/merge.

Copy link
Contributor

@tomMoral tomMoral left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks super nice!! Thx a lot @ogrisel

Just a few extra comment to add but otherwise LGTM


FILE_CONTENT = """\
import coverage; coverage.process_startup()
"""

filename = op.join(get_python_lib(), "coverage_subprocess.pth")
filename = op.join(get_path("platlib"), "coverage_subprocess.pth")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this change?
I did not know platlib, maybe worth adding a comment

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I stole it from @jeremiedbb's PR to add support for Python 3.13. distutils is no longer shipped in recent versions of Python. It can be reinstalled by adding setuptools as a test dependency, but it's better to stick to things available in the standard library.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the comment.

Co-authored-by: Thomas Moreau <thomas.moreau.2010@gmail.com>
@ogrisel
Copy link
Contributor Author

ogrisel commented Mar 7, 2025

@tomMoral I addressed all your comments. Thanks for the review!

@tomMoral tomMoral merged commit e6838b2 into joblib:master Mar 7, 2025
12 checks passed
@tomMoral
Copy link
Contributor

tomMoral commented Mar 7, 2025

Perfect, thanks!

@ogrisel ogrisel deleted the fix-warning-on-fork branch March 10, 2025 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants