ci: workflows: pin python dependencies #87609

nashif · 2025-03-25T08:31:13Z

Pin python dependencies to hashes and cleanup/unify python setup steps in
various workflows.

We now have one dependency file containing all requirements for github
actions that is managed centrally with hashes. No direct pip installs
are needed in workflow files and everything shall go via the
requirements file.

Pinning to specific version and hashes helps with preventing supply
chain attacks.

Signed-off-by: Anas Nashif anas.nashif@intel.com

kernel/init.c

.github/workflows/backport_issue_check.yml

scripts/requirements-actions.in

nashif · 2025-03-25T09:53:09Z

hmm, not sure about this one:

 Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.9/x64/bin/gitlint", line 5, in <module>
    from gitlint.cli import cli
  File "/opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/gitlint/cli.py", line 3, in <module>
    from gitlint.config import LintConfig, LintConfigError
  File "/opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/gitlint/config.py", line 2, in <module>
    import ConfigParser
ModuleNotFoundError: No module named 'ConfigParser'
Compliance error, check for error messages in the "Run Compliance Tests" st

I have not seen this during testing :(

pdgendt · 2025-03-25T11:05:24Z

hmm, not sure about this one:

 Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.9/x64/bin/gitlint", line 5, in <module>
    from gitlint.cli import cli
  File "/opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/gitlint/cli.py", line 3, in <module>
    from gitlint.config import LintConfig, LintConfigError
  File "/opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/gitlint/config.py", line 2, in <module>
    import ConfigParser
ModuleNotFoundError: No module named 'ConfigParser'
Compliance error, check for error messages in the "Run Compliance Tests" st

I have not seen this during testing :(

Can we update the gitlint package? Also for the requirements.in file, we could specify gitlint-core instead.

scripts/requirements-actions.txt

kartben · 2025-03-27T01:40:57Z

I think you will need to update dependabot configuration as well dependabot/dependabot-core#10040 (comment)

hmm, is this reallyneeded? we do have a requirements.txt file in this case. Anyways, one step at a time, I do not think we need to change depandabot workflow for this to go in, no?

Well I would argue we do. This effectively freezes all dependencies (duh!) so should any one of them be affected by a CVE 1/ we won't know about it and 2/, and more importantly, we will never get the fix that may come in a later version and will just be hard-stuck with the vulnerable version, where the current approach would at least have had the benefit to always give us the latest version of the package (for better or worse, too).

So what I'm saying is that basically we will need dependabot eventually if we are to pin dependencies ; and while it can be added in a PR coming shortly after this one, we do need to at least test it works and be confident it will be happy with managing uv compiled requirements before introducing this change in the first place.

In any case, CI seems to be failing so maybe that'll be the opportunity to add this to the next update :)

nashif · 2025-03-27T09:42:52Z

In any case, CI seems to be failing so maybe that'll be the opportunity to add this to the next update :)

maybe, but I prefer to do that later because tbh I have no idea what needs to be done.

Re this CI failure, I am not sure why the twister thing is failing with this version of python. @LukaszMrugala any idea?

cfriedt · 2025-03-27T23:47:47Z

Should help with caching / false positives as well.

pdgendt · 2025-03-28T07:57:52Z

Why was the "Checkout source code" step skipped in the failed "Publish Unit Tests Results"? 🤔

nashif · 2025-03-28T16:27:52Z

Why was the "Checkout source code" step skipped in the failed "Publish Unit Tests Results"? 🤔

this is fixed now

pdgendt · 2025-03-28T17:53:14Z

The commit to temporarily disable the failing test also touches the .github/workflows/twister.yaml file to checkout the pull request head. Is this intentional?

nashif · 2025-03-28T18:49:29Z

The commit to temporarily disable the failing test also touches the .github/workflows/twister.yaml file to checkout the pull request head. Is this intentional?

no, will fix, thanks.

nashif · 2025-03-28T19:06:31Z

The commit to temporarily disable the failing test also touches the .github/workflows/twister.yaml file to checkout the pull request head. Is this intentional?

no, will fix, thanks.

ok, fixed

kartben · 2025-03-28T20:34:04Z

.github/workflows/bsim-tests.yaml

@@ -178,7 +178,6 @@ jobs:

      - name: Merge Test Results
        run: |
-          pip install junitparser junit2html


Looks like you just dropped them as I can't see any step where you install the requirements?

those are already in the docker image, no need to install them

kartben · 2025-03-28T20:36:25Z

.github/workflows/clang.yaml

+          cache: pip
+          cache-dependency-path: scripts/requirements-actions.txt
+
+      - name: install-packages


If you have a chance to do a s/install-packages/Install Python packages/g so that steps have consistent naming that would be great (non blocking obviously)

kartben · 2025-03-28T20:41:03Z

.github/workflows/scripts_tests.yml

-      - name: Rebase
-        continue-on-error: true
-        env:
-          BASE_REF: ${{ github.base_ref }}
-          PR_HEAD: ${{ github.event.pull_request.head.sha }}
-        run: |
-          git config --global user.email "actions@zephyrproject.org"
-          git config --global user.name "Github Actions"
-          rm -fr ".git/rebase-apply"
-          rm -fr ".git/rebase-merge"
-          git rebase origin/${BASE_REF}
-          git clean -f -d
-          git log --graph --oneline HEAD...${PR_HEAD}
-


Unrelated change?

Pin python dependencies to hashes and cleanup/unify python setup steps in various workflows. We now have one dependency file containing all requirements for github actions that is managed centrally with hashes. No direct pip installs are needed in workflow files and everything shall go via the requirements file. Pinning to specific version and hashes helps with preventing supply chain attacks. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

Pin dependencies on the workflow and move it from using docker to the zephyr setup action. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

This test fails on older python versions (3.10) and only on CI. Disabling it while we investigate. The test itself verifies inline logs options, so the functionality test is not impacted. Tracked in zephyrproject-rtos#87769 Signed-off-by: Anas Nashif <anas.nashif@intel.com>

zephyrbot added platform: nRF BSIM Nordic Semiconductors, nRF BabbleSim area: Kernel area: Continuous Integration labels Mar 25, 2025

zephyrbot requested review from aescolar, andyross, ceolin, cfriedt, dcpleung, fabiobaltieri, kartben, npitre, peter-mitsis, stephanosio and TaiJuWu March 25, 2025 08:31

zephyrbot assigned aescolar Mar 25, 2025

aescolar reviewed Mar 25, 2025

View reviewed changes

kernel/init.c Outdated Show resolved Hide resolved

nashif force-pushed the topic/ci/pin_deps branch 2 times, most recently from e13a855 to 68c2a9b Compare March 25, 2025 08:41

aescolar reviewed Mar 25, 2025

View reviewed changes

.github/workflows/backport_issue_check.yml Outdated Show resolved Hide resolved

nashif force-pushed the topic/ci/pin_deps branch 2 times, most recently from 18b79c3 to 374367f Compare March 25, 2025 08:59

pdgendt reviewed Mar 25, 2025

View reviewed changes

scripts/requirements-actions.in Outdated Show resolved Hide resolved

nashif removed the area: Kernel label Mar 25, 2025

nashif removed request for andyross and peter-mitsis March 25, 2025 09:53

pdgendt reviewed Mar 25, 2025

View reviewed changes

scripts/requirements-actions.txt Outdated Show resolved Hide resolved

nashif force-pushed the topic/ci/pin_deps branch 2 times, most recently from f7b1d2e to 4ebcffb Compare March 26, 2025 02:24

pdgendt previously approved these changes Mar 26, 2025

View reviewed changes

nashif dismissed pdgendt’s stale review via ba2ad47 March 27, 2025 15:54

nashif force-pushed the topic/ci/pin_deps branch from a21ca0d to ba2ad47 Compare March 27, 2025 15:54

github-actions bot added the area: Twister Twister label Mar 27, 2025

github-actions bot requested review from gchwier, golowanow, hakehuang, LukaszMrugala and PerMac March 27, 2025 15:55

nashif force-pushed the topic/ci/pin_deps branch from ba2ad47 to 30216ea Compare March 27, 2025 21:01

nashif force-pushed the topic/ci/pin_deps branch from 30216ea to 81f8987 Compare March 28, 2025 10:19

nashif force-pushed the topic/ci/pin_deps branch from 81f8987 to fb12ef5 Compare March 28, 2025 19:05

nashif assigned nashif and stephanosio and unassigned aescolar Mar 28, 2025

kartben reviewed Mar 28, 2025

View reviewed changes

nashif added 3 commits March 28, 2025 18:25

ci: overhaul and pin dependencies on twister_tests_blackbox

3b3a374

Pin dependencies on the workflow and move it from using docker to the zephyr setup action. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

nashif force-pushed the topic/ci/pin_deps branch from 9b43181 to fc7e570 Compare March 28, 2025 22:26

kartben approved these changes Mar 28, 2025

View reviewed changes

pdgendt approved these changes Mar 29, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: workflows: pin python dependencies #87609

ci: workflows: pin python dependencies #87609

nashif commented Mar 25, 2025

nashif commented Mar 25, 2025

pdgendt commented Mar 25, 2025

kartben commented Mar 27, 2025

nashif commented Mar 27, 2025

cfriedt commented Mar 27, 2025

pdgendt commented Mar 28, 2025

nashif commented Mar 28, 2025

pdgendt commented Mar 28, 2025

nashif commented Mar 28, 2025

nashif commented Mar 28, 2025 •

edited

Loading

kartben Mar 28, 2025

nashif Mar 28, 2025

kartben Mar 28, 2025

kartben Mar 28, 2025

ci: workflows: pin python dependencies #87609

Are you sure you want to change the base?

ci: workflows: pin python dependencies #87609

Conversation

nashif commented Mar 25, 2025

nashif commented Mar 25, 2025

pdgendt commented Mar 25, 2025

kartben commented Mar 27, 2025

nashif commented Mar 27, 2025

cfriedt commented Mar 27, 2025

pdgendt commented Mar 28, 2025

nashif commented Mar 28, 2025

pdgendt commented Mar 28, 2025

nashif commented Mar 28, 2025

nashif commented Mar 28, 2025 • edited Loading

kartben Mar 28, 2025

Choose a reason for hiding this comment

nashif Mar 28, 2025

Choose a reason for hiding this comment

kartben Mar 28, 2025

Choose a reason for hiding this comment

kartben Mar 28, 2025

Choose a reason for hiding this comment

nashif commented Mar 28, 2025 •

edited

Loading