Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: doc: pin Python dependencies to specific versions/hashes #87311

Merged
merged 7 commits into from
Mar 21, 2025

Conversation

kartben
Copy link
Collaborator

@kartben kartben commented Mar 18, 2025

Use secure install for Python documentation dependencies.

@kartben kartben requested a review from pdgendt March 18, 2025 18:12
@kartben kartben force-pushed the pin_documentation_py_reqs branch from 2fe3bd0 to 04b921b Compare March 18, 2025 20:11
@kartben kartben marked this pull request as ready for review March 18, 2025 20:12
@kartben kartben requested a review from nashif March 18, 2025 20:12
pdgendt
pdgendt previously approved these changes Mar 18, 2025
Copy link
Collaborator

@pdgendt pdgendt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes please!

@kartben kartben force-pushed the pin_documentation_py_reqs branch from 04b921b to 2bd6444 Compare March 20, 2025 08:31
@pdgendt
Copy link
Collaborator

pdgendt commented Mar 20, 2025

The job doc-build-pdf also has a install-pip step:

    - name: install-pip
      run: |
        pip install -r doc/requirements.txt
        pip install west==${WEST_VERSION}
        pip install cmake==${CMAKE_VERSION}

Which has a different construct for the package versions, what's the rationale here, and why can't we simply add these in doc/requirements.in?

@@ -1,26 +1,722 @@
# DOC: used to generate docs
#
# This file is autogenerated by pip-compile with Python 3.13
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what version are we using in CI? Should this match the same version used in CI?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the question is unrelated to dependabot , wondering if this would be a problem to generate hashes using one version and to be using a different version when installing the packages, i.e. there might be incompatibilities.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we're good. The version is there to help dependabot generate requirements.txt files that are "consistent" (i.e. no unnecessary diffs). The actual hashes have nothing to do with it AFAICT and any pip 8+ version will be happy with the format of the file (https://pip.pypa.io/en/stable/topics/secure-installs/#hash-checking-mode)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-gen'd the requirements.txt with python 3.12 (Python version used in our docker image)

@kartben
Copy link
Collaborator Author

kartben commented Mar 20, 2025

The job doc-build-pdf also has a install-pip step:

    - name: install-pip
      run: |
        pip install -r doc/requirements.txt
        pip install west==${WEST_VERSION}
        pip install cmake==${CMAKE_VERSION}

Which has a different construct for the package versions, what's the rationale here,

Yeah I need to look into this as it seems to actually be failing anyway
https://github.com/kartben/zephyr/actions/runs/13961212510/job/39082783857

and why can't we simply add these in doc/requirements.in?

Yeah this needs some cleanup. Adding cmake to the requirements file would likely be problematic for folks install dependencies locally and already having CMake? I will install it as a Linux package

kartben added 6 commits March 21, 2025 12:32
Add a check that we're installing a legit Doxygen binary

Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Use secure install for ̦Python documentation dependencies.

Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Removed coverxygen installation from the CI workflow and added it to the
requirements.txt/.in files and make it easier for anyone to potentially
use it locally.

Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Added a new configuration for pip dependencies in the doc/
Groups all updates into one PR.

Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Moved away from a texlive docker container to a vanilla ubuntu runner
so that we can better track the actual dependencies a user needs to
build the PDF rather than relying on the gigabytes of random stuff
that the texlive docker container pulls in.

Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
We get West from the action-zephyr-setup action

Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
@kartben
Copy link
Collaborator Author

kartben commented Mar 21, 2025

Ended up doing lots of houskeeping on the PDF job so as to get rid of the texlive docker container we were using so that we actually install the required packages and therefore cross-check that our documentation has up-to-date instructions w.r.t what people might need to install to build the PDF locally

@kartben kartben requested review from nashif and pdgendt March 21, 2025 11:38
pdgendt
pdgendt previously approved these changes Mar 21, 2025
nashif
nashif previously approved these changes Mar 21, 2025
require hashes in doc ci workflow

Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
@kartben kartben dismissed stale reviews from nashif and pdgendt via 7983d8a March 21, 2025 18:13
@kartben
Copy link
Collaborator Author

kartben commented Mar 21, 2025

Just added one more extra flag that makes extra sure hashes are mandatory (and makes scorecard happy as well)

@kartben kartben requested review from nashif and pdgendt March 21, 2025 18:14
@nashif nashif merged commit 911d803 into zephyrproject-rtos:main Mar 21, 2025
15 checks passed
@kartben kartben deleted the pin_documentation_py_reqs branch March 27, 2025 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants