Skip to content

Commit 20ecbe2

Browse files
committed
Moving to individual files, adding plantuml source
1 parent 6316f56 commit 20ecbe2

18 files changed

+500
-644
lines changed
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.

docs/temp/out.md

-640
This file was deleted.
109 KB
Loading

docs/testing/img/plant_uml_source.txt

+55
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
@startuml
2+
class AttributeAccessInterface {
3+
mEndpointId : Optional<EndpointId>
4+
mClusterId : ClusterId
5+
Read(...)
6+
Write(...)
7+
}
8+
9+
class CommandHandlerInterface {
10+
InvokeCommand(...)
11+
}
12+
13+
class ClusterServer {
14+
Read(...)
15+
Write(...)
16+
InvokeCommand(...)
17+
RegisterEndpoint(...)
18+
mEndpoints: ClusterLogic[]
19+
}
20+
21+
class MatterContext {
22+
LogEvent()
23+
MarkAttributeDirty()
24+
mPersistentStorageDelegate
25+
mOtherFakeableThings
26+
}
27+
28+
class ClusterLogic {
29+
Init(...)
30+
GetXAttribute(...)
31+
SetXAttribute(...)
32+
HandleXCommand(...)
33+
mStateVariables
34+
}
35+
36+
class ClusterDriver {
37+
OnClusterStateChange(...)
38+
RegisterListener(...)
39+
mListener
40+
}
41+
42+
AttributeAccessInterface <|-- ClusterServer
43+
CommandHandlerInterface <|-- ClusterServer
44+
ClusterServer "1" *-- "Many" ClusterLogic
45+
ClusterLogic "1" *-- "1" ClusterDriver
46+
ClusterLogic "1" *-- "1" MatterContext
47+
48+
hide ClusterServer members
49+
hide AttributeAccessInterface members
50+
hide CommandHandlerInterface members
51+
hide ClusterLogic members
52+
hide ClusterDriver members
53+
hide MatterContext members
54+
55+
@enduml
70.4 KB
Loading

docs/testing/img/unit_tests.png

28.3 KB
Loading

docs/testing/index.md

+20-3
Original file line numberDiff line numberDiff line change
@@ -11,16 +11,33 @@ in the SDK.
1111
*
1212
```
1313

14-
## Unit testing
14+
## Integration and Certification tests
1515

16-
- [Unit tests](./unit_testing.md)
16+
Integration tests test the entire software stack using the same mechanisms as a
17+
standard controller.
1718

18-
## Integration and Certification tests
19+
![](./img/integration_tests.png)
20+
21+
The certification tests are all integration tests, since they run against the
22+
product as a black box.
1923

2024
- [Integration and Certification tests](./integration_tests.md)
2125
- [YAML](./yaml.md)
2226
- [Python testing framework](./python.md)
2327
- [Enabling tests in the CI](./ci_testing.md)
28+
- [Integration test utilities](./integration_test_utilities.md)
29+
30+
## Unit testing
31+
32+
Unit tests run on small pieces (“units”) of business logic. They do not use an
33+
external controller and instead test at the public interface of the class or
34+
function. For cluster, this requires an API that separates the cluster logic
35+
from the global ember and message delivery layers.
36+
37+
![](./img/unit_tests.png)
38+
39+
- [Unit tests](./unit_testing.md)
40+
- [Designing clusters for unit testing and portability](./unit_testing_clusters.md)
2441

2542
## PICS and PIXIT
2643

+134
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# Integration Test utilities
2+
3+
There are several test utilities that can be used to simulate or force behavior
4+
on devices for the purposes of testing.
5+
6+
When using any of these utilties it is important to inject the errors at the
7+
point where they are running through the MOST code that they can.
8+
9+
If the cluster uses the [ClusterLogic](./unit_testing_clusters.md) pattern, this
10+
means injecting errors as close as possible to the driver layer, rather than
11+
catching errors in the server.
12+
13+
## TestEventTriggers
14+
15+
TestEventTriggers are used to test interactions on the DUT that are difficult to
16+
perform during certification testing (ex. triggering a smoke alarm)
17+
18+
**These should be used sparingly!**
19+
20+
TestEventTriggers are started though a command in the General Diagnostics
21+
cluster. The command takes a “test key” and a “trigger code” to request that a
22+
device to perform a specific action. Currently most devices use a default key,
23+
but it can be overridden by a specific device if required.
24+
25+
**TestEventTriggers need to be turned off outside of certification tests**
26+
27+
To use these triggers:
28+
29+
- Derive from
30+
[TestEventTriggerHandler](https://github.com/project-chip/connectedhomeip/blob/master/src/app/TestEventTriggerDelegate.h)
31+
- Implement HandleEventTrigger function
32+
- Register with TestEventTriggerDelegate::AddHandler
33+
34+
Please see
35+
[EnergyEvseTestEventTriggerHandler](https://github.com/project-chip/connectedhomeip/blob/master/src/app/clusters/energy-evse-server/EnergyEvseTestEventTriggerHandler.h)
36+
for a good example.
37+
38+
## NamedPipes
39+
40+
NamedPipes are used to trigger actions on Linux applications. These can be used
41+
in the CI, and are normally used to simulate manual actions for CI integration.
42+
Any required manual action in a test (ex. push a button) should have a
43+
corresponding NamedPipe action to allow the test to run in the CI.
44+
45+
In python tests, the app-pid required to access the named pipe can be passed in
46+
as a flag (--app-pid).
47+
48+
NamedPipes are implemented in
49+
[NamedPipeCommands.h](https://github.com/project-chip/connectedhomeip/blob/master/examples/platform/linux/NamedPipeCommands.h)
50+
51+
To use NamedPipes
52+
53+
- Derive from NamedPipeCommandDelegate
54+
- Implement the OnEventCommandReceived(const char \* json) function
55+
- Instantiate and start a NamedPipeCommands object to receive commands and
56+
pass in the NamedPipeCommandDelegate and a file path basename
57+
- (while running) Write to the file (basename_pid) to trigger the actions
58+
59+
For a good example, see Air Quality:
60+
61+
- [Delegate](https://github.com/project-chip/connectedhomeip/blob/master/examples/air-quality-sensor-app/linux/AirQualitySensorAppAttrUpdateDelegate.cpp)
62+
- [main](https://github.com/project-chip/connectedhomeip/blob/master/examples/air-quality-sensor-app/linux/main.cpp)
63+
- [README](https://github.com/project-chip/connectedhomeip/blob/master/examples/air-quality-sensor-app/linux/README.md)
64+
65+
[RVC Clean Mode](https://github.com/project-chip/connectedhomeip/blob/master/src/python_testing/TC_RVCCLEANM_2_1.py)
66+
gives an example of how to use named pipes in testing.
67+
68+
## Fault Injection
69+
70+
Fault injection is used to inject conditional code paths at runtime, e.g.
71+
errors. This is very useful for things like client testing, to check error
72+
handling for paths that are difficult to hit under normal operating conditions.
73+
74+
The fault injection framework we are currently using is nlFaultInjection.The
75+
framework uses a macro for injecting the code paths, and the macro is set to a
76+
no-op if the option is turned off at compile time. The build option to turn on
77+
fault inject is `chip_with_nlfaultinjection`.
78+
79+
Fault injection has been plumbed out through a manufacturer-specific
80+
[fault injection cluster](#fault-injection-cluster) that is available in the
81+
SDK. This allows fault injection to be turned on and off using standard cluster
82+
mechanisms during testing. For certification, operating these using a secondary,
83+
non-DUT controller is recommended. For a good example of this, please see
84+
[TC-IDM-1.3](https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/interactiondatamodel.adoc#tc-idm-1-3-batched-commands-invoke-request-action-from-dut-to-th-dut_client).
85+
86+
The nlFaultInjection allows the application to define multiple managers. In the
87+
SDK, we have managers for System, inet and CHIP. CHIP should be used for
88+
anything above the system layer (basically all new cluster development). The
89+
CHIP fault manager is available at
90+
[lib/support/CHIPFaultInjection.h](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/CHIPFaultInjection.h).
91+
92+
To add new fault injection code paths:
93+
94+
- Add new IDs (aFaultID) to the enum in
95+
[CHIPFaultInjection](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/CHIPFaultInjection.h)
96+
- add CHIP_FAULT_INJECT(aFaultID, aStatements) at the point where the fault
97+
injection should occur
98+
99+
### Fault Injection example
100+
101+
```c++
102+
CHIP_ERROR CASEServer::OnMessageReceived(Messaging::ExchangeContext * ec,
103+
const PayloadHeader & payloadHeader,
104+
System::PacketBufferHandle && payload)
105+
{
106+
MATTER_TRACE_SCOPE("OnMessageReceived", "CASEServer");
107+
108+
bool busy = GetSession().GetState() != CASESession::State::kInitialized;
109+
CHIP_FAULT_INJECT(FaultInjection::kFault_CASEServerBusy, busy = true);
110+
if (busy)
111+
{
112+
113+
```
114+
115+
### Fault Injection cluster
116+
117+
The Fault injection cluster is a manufacturer-specific cluster, available in the
118+
SDK (0xFFF1FC06).
119+
120+
- [server](https://github.com/project-chip/connectedhomeip/blob/master/src/app/clusters/fault-injection-server/fault-injection-server.cpp)
121+
- [xml](https://github.com/project-chip/connectedhomeip/blob/master/src/app/zap-templates/zcl/data-model/chip/fault-injection-cluster.xml)
122+
123+
Example apps can be compiled with this cluster for client-side certification and
124+
integration tests.
125+
126+
To use this cluster to turn on a fault, use the FailAtFault command:
127+
128+
- Type: FaultType - use FaultType::kChipFault (0x03)
129+
- Id: int32u - match the ID you set up for your fault
130+
- NumCallsToSkip: int32u - number of times to run normally
131+
- NumCallsToFail: int32u - number of times to hit the fault injection
132+
condition after NumCallsToSkip
133+
- TakeMutex: bool - controls access to the fault injection manager for
134+
multi-threaded systems. False is fine.

docs/testing/unit_testing.md

+120-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,122 @@
11
# Unit testing
22

3-
This doc is a placeholder for an guide around unit tests.
3+
## Why?
4+
5+
- MUCH faster than integration tests.
6+
- Runs as part of build process.
7+
- Allows testing specific error conditions that are difficult to trigger under
8+
normal operating conditions.
9+
- e.g. out of memory errors etc.
10+
- Allows testing different device compositions without defining multiple
11+
example applications.
12+
- e.g. feature combinations not in example apps.
13+
14+
## Unit testing in the SDK - nlUnitTest
15+
16+
The following example gives a small demonstration of how to use nlUnitTest to
17+
write a unit test
18+
19+
```
20+
#include <lib/support/UnitTestContext.h>
21+
#include <lib/support/UnitTestRegistration.h>
22+
#include <nlunit-test.h>
23+
24+
class YourTestContext : public Test::AppContext {
25+
...
26+
};
27+
28+
29+
static void TestName(nlTestSuite * apSuite, void * apContext) {
30+
// If you register the test suite with a context, cast
31+
// apContext as appropriate
32+
YourTestContext * ctx = static_cast<YourTestContext *>(aContext);
33+
34+
// Do some test things here, then check the results using NL_TEST_ASSERT
35+
NL_TEST_ASSERT(apSuite, <boolean condition>)
36+
}
37+
38+
static const nlTest sTests[] =
39+
{
40+
NL_TEST_DEF("TestName", TestName), // Can have multiple of these
41+
NL_TEST_SENTINEL() // If you forget this, you’re going to have a bad time
42+
};
43+
44+
nlTestSuite sSuite =
45+
{
46+
"TheNameOfYourTestSuite", // Test name
47+
&sTests[0], // The list of tests to run
48+
TestContext::Initialize, // Runs before all the tests (can be nullptr)
49+
TestContext::Finalize // Runs after all the tests (can be nullptr)
50+
};
51+
52+
int YourTestSuiteName()
53+
{
54+
return chip::ExecuteTestsWithContext<YourTestContext>(&sSuite); // or “without”
55+
}
56+
57+
CHIP_REGISTER_TEST_SUITE(YourTestSuiteName)
58+
```
59+
60+
Each test gets an nlTestSuite object (apSuite) that is passed into the test
61+
assertions, and a void\* context (apContext) that is yours to do with as you
62+
require.
63+
64+
The apContext should be derived from
65+
[Test::AppContext](https://github.com/project-chip/connectedhomeip/blob/master/src/app/tests/AppTestContext.h)
66+
67+
See
68+
[TestSpan.cpp](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/tests/TestSpan.cpp)
69+
for a great example of a good unit test.
70+
71+
## nlUnitTest - Compiling and running
72+
73+
- Add to src/some_directory/tests/BUILD.gn
74+
- chip_test_suite_using_nltest("tests")
75+
- See for example
76+
[src/lib/support/tests/BUILD.gn](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/tests/BUILD.gn)
77+
- [./gn_build.sh](https://github.com/project-chip/connectedhomeip/blob/master/gn_build.sh)
78+
will build and run all tests
79+
- CI runs this, so any unit tests that get added will automatically be
80+
added to the CI
81+
- Test binaries are compiled into:
82+
- out/debug/<host_compiler>/tests
83+
- e.g. out/debug/linux_x64_clang/tests
84+
- Tests are run when ./gn_build.sh runs, but you can run them individually in
85+
a debugger from their location.
86+
87+
## Debugging unit tests
88+
89+
- After running ./gn_build.sh, test binaries are compiled into
90+
- out/debug/<host_compiler>/tests
91+
- e.g. out/debug/linux_x64_clang/tests
92+
- Individual binaries, can be run through regular tools:
93+
- gdb
94+
- valgrind
95+
- Your favorite tool that you tell everyone about.
96+
97+
## Utilities
98+
99+
We have a small number of unit testing utilities that should be used in unit
100+
tests.
101+
102+
Consider adding more utilities for general use if you require them for your
103+
tests.
104+
105+
### Mock clock
106+
107+
The mock clock is located in
108+
[src/system/SystemClock.h](https://github.com/project-chip/connectedhomeip/blob/master/src/system/SystemClock.h)
109+
as `System::Clock::Internal::MockClock`.
110+
111+
To use the mock clock, use the `chip::System::SystemClock()` function as normal.
112+
In the test, instantiate a MockClock and use the `SetSystemClockForTesting` to
113+
inject the clock. The Set and Advance functions in the MockClock can then be
114+
used to set exact times for testing. This allows testing specific edge
115+
conditions in tests, helps reduce test flakiness due to race conditions, and
116+
reduces the time required for testing as tests no long require real-time waits.
117+
118+
### TestPersistentStorageDelegate
119+
120+
The TestPersistentStorageDelegate is an in-memory version of storage that easily
121+
allows removal of keys, presence checks, etc. It is available at
122+
[src/lib/support/TestPersistentStorageDelegate.h](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/TestPersistentStorageDelegate.h)

0 commit comments

Comments
 (0)