Skip to content

Commit ada2eaa

Browse files
committedFeb 7, 2024
fix a couple of todos
1 parent d3b97c4 commit ada2eaa

File tree

1 file changed

+25
-15
lines changed

1 file changed

+25
-15
lines changed
 

‎docs/testing/python.md

+25-15
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,5 @@
11
# Python framework tests
22

3-
This file is a placeholder for python framework test information.
4-
5-
NOTE: be sure to include information about how you need to commission with the
6-
python controller, not chip-tool and how to do that in the scripts
7-
8-
Note - go through the docs in the readme and move them over here, and take out
9-
the deprecated stuff
10-
11-
Note - add notes about record error and fail test vs. asserts
12-
13-
---
14-
153
The python test framework is built on top of the ChipDeviceCtrl.py python
164
controller API and the mobly test framework. Python tests are interaction tests,
175
and can be used for certification testing, and / or integration testing in the
@@ -25,7 +13,7 @@ Python tests located in src/python_testing
2513
sample test showing test setup and test harness integration
2614
- [https://github.com/google/mobly/blob/master/docs/tutorial.md](https://github.com/google/mobly/blob/master/docs/tutorial.md)
2715
- [ChipDeviceCtrl.py](https://github.com/project-chip/connectedhomeip/blob/master/src/controller/python/chip/ChipDeviceCtrl.py) -
28-
Controller implementation with API documentation
16+
Controller implementation ([API documentation](./ChipDeviceCtrlAPI.md))
2917
- [scripts/tests/run_python_test.py](https://github.com/project-chip/connectedhomeip/blob/master/scripts/tests/run_python_test.py)
3018
to easily set up app and script for testing - used in CI
3119

@@ -408,6 +396,25 @@ reference.
408396
- if the test is exceptionally long running, define a property function
409397
`default_timeout` to adjust the timeout. The default is 90 seconds
410398

399+
Deferred failures: For some tests, it makes sense to perform the entire test
400+
before failing and collect all the errors so the developers can address all the
401+
failures without needing to re-run the test multiple times. For example, tests
402+
that look at every attribute on the cluster and perform independent operations
403+
on them etc.
404+
405+
For such tests, use the ProblemNotice format and the convenience functions:
406+
407+
- self.record_error
408+
- self.record_warning
409+
410+
These functions keep track of the problems, and will print them at the end of
411+
the test. The test will not be failed until the assert is called.
412+
413+
A good example of this type of test can be found in the device basic composition
414+
tests, where all the test steps are independent and performed on a single read.
415+
See
416+
[Device Basic Composition tests](https://github.com/project-chip/connectedhomeip/blob/master/src/python_testing/TC_DeviceBasicComposition.py)
417+
411418
## Command line arguments
412419

413420
- Use help to get a full list
@@ -476,11 +483,14 @@ second_ctrl = fa.new_fabric_admin.NewController(nodeId=node_id)
476483
You can run the python script as-is for local testing against an already-running
477484
DUT
478485

479-
./scripts/tests/run_python_test.py is a convenient script to fire up an example
480-
DUT on the host, with factory reset support
486+
`./scripts/tests/run_python_test.py` is a convenient script to fire up an
487+
example DUT on the host, with factory reset support
481488

482489
`./scripts/tests/run_python_test.py --factoryreset --app <your_app> --app-args "whatever" --script <your_script> --script-args "whatever"`
483490

491+
Note that devices must be commissioned by the python test harness to run tests.
492+
chip-tool and the python test harness DO NOT share a fabric.
493+
484494
# Running tests in CI
485495

486496
- add to .github/workflows/tests.yaml repl_tests_linux

0 commit comments

Comments
 (0)