FIX deadlock in test_shutdown_with_sys_exit_at_pickle

[ogrisel]

Trying to reproduce #437 in the current master branch.

EDIT: reproduced at d833e8f and fixed by c4217e4.

Fixes #437.

[ogrisel]

I cannot reproduce the race condition on main.

[ogrisel]

So 17ebf79 managed to reproduce the problem (by deadlocking at a different place but the warning is still there) with a high probability.

The probability of triggering the deadlock goes down if we remove the loop in the test. We need to find a balance between catching race conditions with a high probability and test time.

[ogrisel]

5 iterations in both test_shutdown_with_pickle_error and test_shutdown_with_sys_exit_at_pickle is definitely enough to trigger it with high probability, either under Linux or Windows. macOS seems to be unaffected for some reason (and I could never reproduce locally on macOS either).

So let's try to see if the proposed change of #437 (comment) can help resolve this problem.

[ogrisel]

Victory! c4217e4 fixed the problem! 🥳

[ogrisel]

Note: the tests are still running in less than 8 minutes despite the changes in this PR.

[tomMoral]

Oh, weird that we never got this one before!
Thanks for the reproducer and the fix, it makes a lot of sense.
LGTM