Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

twister: Fix quarantine filtering simulations #87421

Merged
merged 2 commits into from
Mar 24, 2025

Conversation

gchwier
Copy link
Collaborator

@gchwier gchwier commented Mar 20, 2025

Quarantine filter was not correctly identifying the simulator name. Now it uses the simulator name from the Platform object, ensuring that quarantined tests are properly excluded or verified.

Tested with:

Create quarantine.yaml file:

cat quarantine.yaml
- simulations:
  - armfvp
  comment: "armfvp"
- simulations:
  - qemu
  comment: "qemu"

Check if quarantine works for the default simulations (qemu):
./scripts/twister -T samples/hello_world -p mps2/an521/cpu0 -vv -ll debug -c --quarantine-list quarantine.yaml
logged:
DEBUG - mps2/an521/cpu0 samples/hello_world/sample.basic.helloworld FILTERED: Quarantine: qemu

Check if quarantine works with non-default simulator (armfvp):
./scripts/twister -T samples/hello_world -p mps2/an521/cpu0 -vv -ll debug -c --simulation armfvp --quarantine-list quarantine.yaml
logged:
DEBUG - mps2/an521/cpu0 samples/hello_world/sample.basic.helloworld FILTERED: Quarantine: armfvp

Run with other simulations:
./scripts/twister -T samples/hello_world -p qemu_x86 -p native_sim -vv -ll debug -c --quarantine-list quarantine.yaml
logged

DEBUG   - qemu_x86/atom             samples/hello_world/sample.basic.helloworld        FILTERED: Quarantine: qemu
...
INFO    - 1/1 native_sim/native         samples/hello_world/sample.basic.helloworld        PASSED (native 2.623s <host>)

Issue introduced by: #79174

Quarantine filter was not correctly identifying the simulator name.
Now it uses the simulator name from the Platform object,
ensuring that quarantined tests are properly excluded or verified

Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Copy link
Member

@golowanow golowanow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you for fixing this!
could you also add a missing example of quarantine.yaml with simulations here
https://github.com/zephyrproject-rtos/zephyr/blob/main/doc/develop/test/twister.rst?plain=1#L1529

Update the quarantine section in the twister.rst documentation
to include information about excluding specific platforms,
architectures, and simulators

Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
@gchwier
Copy link
Collaborator Author

gchwier commented Mar 20, 2025

thank you for fixing this! could you also add a missing example of quarantine.yaml with simulations here https://github.com/zephyrproject-rtos/zephyr/blob/main/doc/develop/test/twister.rst?plain=1#L1529

thanks for review, updated docs

@kartben kartben merged commit 3acfeaf into zephyrproject-rtos:main Mar 24, 2025
36 checks passed
@gchwier gchwier deleted the grch_quarantine_simulations branch March 25, 2025 11:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants