From 7857d4e830ff974c11073373eaf395be4b7e4978 Mon Sep 17 00:00:00 2001 From: Rob Bultman Date: Thu, 11 Apr 2024 14:34:04 -0400 Subject: [PATCH 1/8] Move add new device howto --- .../cluster_and_device_type_dev.md | 8 +- .../how_to_add_new_dts_and_clusters.md | 248 ++++++++++++++++++ docs/cluster_and_device_type_dev/index.md | 1 + 3 files changed, 252 insertions(+), 5 deletions(-) create mode 100644 docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md diff --git a/docs/cluster_and_device_type_dev/cluster_and_device_type_dev.md b/docs/cluster_and_device_type_dev/cluster_and_device_type_dev.md index f4c1c5eb890e71..e016ea78b7b7bf 100644 --- a/docs/cluster_and_device_type_dev/cluster_and_device_type_dev.md +++ b/docs/cluster_and_device_type_dev/cluster_and_device_type_dev.md @@ -36,11 +36,9 @@ types in the SDK. - XML defines conformance - [src/app/zap-templates/zcl/data-model/chip/matter-devices.xml](https://github.com/project-chip/connectedhomeip/blob/master/src/app/zap-templates/zcl/data-model/chip/matter-devices.xml) -The following wiki page has a detailed description of how and where to add -cluster and device type definitions so they are picked up properly by ZAP/ember -and the SDK. - -[https://groups.csa-iot.org/wg/matter-tsg/wiki/howto-add-a-new-device-type](https://groups.csa-iot.org/wg/matter-tsg/wiki/howto-add-a-new-device-type) +See [How To Add New Device Types & Clusters](how_to_add_new_dts_and_clusters.md) +for a detailed description of how and where to add cluster and device type +definitions so they are picked up properly by ZAP/ember and the SDK. Note that the output should also be verified against the spec using the [.matter parser tools](https://project-chip.github.io/connectedhomeip-doc/guides/matter_idl_tooling.html). diff --git a/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md b/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md new file mode 100644 index 00000000000000..e0421ed9b4149d --- /dev/null +++ b/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md @@ -0,0 +1,248 @@ +# How to Add New Device Types & Clusters + +This document outlines the process needed to add a new Matter device type and +related clusters. Obviously, the steps below assume that the related Matter +specifications were properly reviewed and approved. + +## Add the cluster definitions to the SDK + +The following steps should be followed to add new cluster definitions to the +SDK. + +**Add your new cluster definition to an appropriately-name file in this +directory:** +[src/app/zap-templates/zcl/data-model/chip](https://github.com/project-chip/connectedhomeip/tree/master/src/app/zap-templates/zcl/data-model/chip) + +**Add references to each new cluster definition to these files:** + +1. [.github/workflows/tests.yaml](https://github.com/project-chip/connectedhomeip/tree/master/.github/workflows/tests.yaml) +2. [scripts/rules.matterlint](https://github.com/project-chip/connectedhomeip/tree/master/scripts/rules.matterlint) +3. [src/app/zap-templates/zcl/data-model/all.xml](https://github.com/project-chip/connectedhomeip/tree/master/src/app/zap-templates/zcl/data-model/all.xml) +4. [src/app/zap-templates/zcl/zcl-with-test-extensions.json](https://github.com/project-chip/connectedhomeip/tree/master/src/app/zap-templates/zcl/zcl-with-test-extensions.json) +5. [src/app/zap-templates/zcl/zcl.json](https://github.com/project-chip/connectedhomeip/tree/master/src/app/zap-templates/zcl/zcl.json) +6. If it is a derived cluster, add a reference to the base cluster definition. + (e.g. in mode-base-cluster.xml you may need to add cluster codes - otherwise + you may get strange exceptions which aren't clear when running regen_all.py) + + > ``` + > + > + > + > + > ``` + +7. [src/controller/python/chip/clusters/\_\_init\_\_.py](https://github.com/project-chip/connectedhomeip/tree/master/src/controller/python/chip/clusters/__init__.py) + +**Enable your new cluster in the Python and Android clients** in +[src/controller/data_model/controller-clusters.zap](https://github.com/project-chip/connectedhomeip/blob/master/src/controller/data_model/controller-clusters.zap) + +You will need the ZAP tool to edit the ZAP file. + +- Unless you already have the tool installed, you can use one of the + [nightly builds](https://github.com/project-chip/zap/releases) +- [ZAP tool info](https://developers.home.google.com/matter/tools/zap) +- [ZAP tool repo](https://github.com/project-chip/zap) + +Use the ZAP GUI tool to edit the file above: + +1. From the command line, navigate to the directory containing + [controller-clusters.zap](https://github.com/project-chip/connectedhomeip/blob/master/src/controller/data_model) +2. Run zap: `../../../scripts/tools/zap/run_zaptool.sh controller-clusters.zap`. + Alternatively, run the zap tool and navigate to the zap file that you wish to + open, or run as + `./scripts/tools/zap/run_zaptool.sh src/controller/data_model/controller-clusters.zap`. +3. In the gui, select `Endpoint-1` from the left pane. +4. Open the cluster group, for example, `Appliances`. +5. Find the cluster to be enabled, e.g. `Dishwasher Control`. +6. In the Enable column, select "Client" from the drop-down box. +7. Click `File->Save` to save the configuration. +8. Close the GUI. + +**Add entries for your new cluster to +[BUILD.gn](c/src/controller/data_model/BUILD.gn)** in the outputs section of the +java-jni-generate bits. The entries should look like +"jni/YourClusterNameClient-InvokeSubscribeImpl.cpp" and +"jni/YourClusterNameClient-ReadImpl.cpp". + +**Add an entry to the ClientDirectories** section of +[src/app/zap_cluster_list.json](https://github.com/project-chip/connectedhomeip/blob/master/src/app/zap_cluster_list.json). + +**Update `chip-tool`** + +1. Regenerate all zap generated code using `./scripts/tools/zap_regen_all.py` +2. Rebuild chip-tool and it will have new cluster support: + `./scripts/examples/gn_build_example.sh examples/chip-tool SOME-PATH/` + +## Add the device type definition to the SDK + +1. Add the XML definition of the device to + [matter-devices.xml](https://github.com/project-chip/connectedhomeip/blob/master/src/app/zap-templates/zcl/data-model/chip/matter-devices.xml) +2. Implement the incoming command behaviors common to all applications. The + parsing of the payload from TLV to a C++ struct is done by code + auto-generated from the XML (see + [zap-generated](https://github.com/project-chip/connectedhomeip/blob/master/zzz_generated/app-common/app-common/zap-generated)) + The rest of the functionality must be manually implemented. Note: for the + auto-generated code run one of the following: + 1. for everything: `./scripts/tools/zap_regen_all.py` + 2. just for the app-common part: + `./scripts/tools/zap/generate.py -t src/app/common/templates/templates.json -o zzz_generated/app-common/app-common/zap-generated src/controller/data_model/controller-clusters.zap` +3. Implement the read/write/storage operations for the attributes of any type, + list, or struct which are not the global attributes present in all clusters. + For example, there's no need to implement CommandList, AttributeList etc. For + the attributes which are not list of struct type, the handling code is + implemented generically so most likely no work is needed for them. +4. Implement any attribute spec requirements that are common to all + applications. For example: code to enforce specific ranges, code for handling + the interactions between attributes etc. + +## Implement Code and Tests + +Implement the clusters, the example cluster server application and add the +related SDK tests. + +1. Implement new clusters here: + [src/app/clusters](https://github.com/project-chip/connectedhomeip/tree/master/src/app/clusters) +2. Implement tests here: + [src/app/tests/suites](https://github.com/project-chip/connectedhomeip/tree/master/src/app/tests/suites) +3. Implement the example cluster server application: + 1. The YAML tests will run against this server + 2. Depending on the clusters, there are two options: + 1. Enable them in the all-clusters-app and use that as the example + cluster server app. If choosing this option, consider adding an + example application that has just the relevant clusters enabled; this + part is a nice to have. + 2. If the clusters have complicated global application requirements + consider using a separate example app. see the door lock, bridge, TV, + OTA clusters. + 3. NOTES: If adding a new cluster derived from `mode-base` into + [examples/all-clusters-app/](https://github.com/project-chip/connectedhomeip/tree/master/examples/all-clusters-app/) + (Assume your new cluster is called `XZYMode`): + 1. Create your new `xyz-mode-cluster.xml` in + [src/app/zap-templates/zcl/data-model/chip](https://github.com/project-chip/connectedhomeip/tree/master/src/app/zap-templates/zcl/data-model/chip) + (as above). Note that this is a much cut down cluster definition + compared to normal clusters, since it derives from + [mode-base-cluster.xml](https://github.com/project-chip/connectedhomeip/tree/master/src/app/zap-templates/zcl/data-model/chip/mode-base-cluster.xml). + See + [dishwasher-mode-cluster.xml](https://github.com/project-chip/connectedhomeip/tree/master/src/app/zap-templates/zcl/data-model/chip/dishwasher-mode-cluster.xml) + as an example. Note you should review if you need the `StartUpMode` + and `OnMode` attributes based on the spec. + 2. Check that you have added your cluster code into + [mode-base-cluster.xml](https://github.com/project-chip/connectedhomeip/tree/master/src/app/zap-templates/zcl/data-model/chip/mode-base-cluster.xml) + 1. ` ` - replace + `XXXX` with your cluster ID + 2. ` ` - + replace `XXXX` with your cluster ID + 3. ` ` - + replace `XXXX` with your cluster ID + 3. Add your new Mode definitions in `xyz-mode.h`. In this header you + define the modes / labels and tags. See + [dishwasher-mode.h](https://github.com/project-chip/connectedhomeip/tree/master/examples/all-clusters-app/all-clusters-common/include/dishwasher-mode.h) + as an example. + 4. Add your new stub to instantiate the mode. See + [dishwasher-mode.cpp](https://github.com/project-chip/connectedhomeip/tree/master/examples/all-clusters-app/all-clusters-common/src/dishwasher-mode.cpp) + as an example. + 5. In + [examples/all-clusters-app/linux/main-common.cpp](https://github.com/project-chip/connectedhomeip/tree/master/examples/all-clusters-app/linux/main-common.cpp): + 1. Add `#include "xyz-mode.h"` + 2. In `ApplicationShutdown()`, add a call to your + `Clusters::XYZMode::Shutdown();`. + 6. In + [examples/all-clusters-app/linux/BUILD.gn](https://github.com/project-chip/connectedhomeip/tree/master/examples/all-clusters-app/linux/BUILD.gn), + add the `xyz-mode.cpp` file to the `sources` section. + 7. In + [src/app/common/templates/config-data.yaml](https://github.com/project-chip/connectedhomeip/blob/master/src/app/common/templates/config-data.yaml): + 1. Add a `::ModeTag` to the `EnumsNotUsedAsTypeInXML` + section. + 2. Add an `XYZ Mode` entry to the + `CommandHandlerInterfaceOnlyClusters` section. + 8. In + [src/app/util/util.cpp](https://github.com/project-chip/connectedhomeip/blob/master/src/app/util/util.cpp), + in the `// Cluster Init Functions...` section, add a void + `MatterXYZModePluginServerInitCallback() {}` definition. + 9. In + [src/app/zap-templates/zcl/zcl-with-test-extensions.json](https://github.com/project-chip/connectedhomeip/blob/master/src/app/zap-templates/zcl/zcl-with-test-extensions.json): + 1. Add the `xyz-mode-cluster.xml` to the `xmlFile` list + 2. In the `attributeAccessInterfaceAttributes` entry, add your new + entry + `"XYZ Mode": [ "SupportedModes", "CurrentMode", "FeatureMap" ]` - + this will mean ZAP won't autogen code to handle these attributes + 10. In + [src/app/zap_cluster_list.json](https://github.com/project-chip/connectedhomeip/blob/master/src/app/zap_cluster_list.json): + 1. Add your `XYZ_MODE_CLUSTER: []` to `ClientDirectories: { }` + object + 2. Add your `"XYZ_MODE_CLUSTER": ["mode-base-server"]` to + `"ServerDirectories": { }` + 4. The code under + [src/app/tests/suites/certification](https://github.com/project-chip/connectedhomeip/blob/master/src/app/tests/suites/certification) + for YAML or + [src/python_testing](https://github.com/project-chip/connectedhomeip/tree/master/src/python_testing) + for Python should ideally implement the test plan (section 4 below). + 5. A test under + [src/app/tests/suites](https://github.com/project-chip/connectedhomeip/blob/master/src/app/tests/suites) + (not certification) can test anything, in addition to, or preceding the + official YAML representing the official test plan. +4. Add the test plan, using the templates below: + + 1. [cluster_test_plan_template.adoc](https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/template/cluster_test_plan_template.adoc) + 2. [test_plan_template.adoc](https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/template/test_plan_template.adoc) + + Also, join this private Slack channel: csg-tt-test-plans + +5. Note: the CHIP-Tool reference client is generated from XML +6. If applicable, add tests: + 1. For relatively simple tests, add YAML tests here: + [src/app/tests/suites/certification](https://github.com/project-chip/connectedhomeip/blob/master/src/app/tests/suites/certification). + Remember to update this file: + [src/app/tests/suites/certification/PICS.yaml](https://github.com/project-chip/connectedhomeip/blob/master/src/app/tests/suites/certification/PICS.yaml) + 2. For more complex tests, add Python tests here: + [src/python_testing](https://github.com/project-chip/connectedhomeip/tree/master/src/python_testing) + 3. To add tests to CI, if applicable: + 1. Add the Python tests here: + [.github/workflows/tests.yaml](https://github.com/project-chip/connectedhomeip/tree/master/.github/workflows/tests.yaml). + Remember to provide all arguments required for each Python script, + such as the endpoint PIXIT. + 2. Add the YAML tests by editing this file: + [src/app/tests/suites/ciTests.json](https://github.com/project-chip/connectedhomeip/tree/master/src/app/tests/suites/ciTests.json) + 1. Create a section ("MyDevice") which lists all YAML tests for your + device + 2. Add the section's name to the list of items named "collection" + 3. Do a ZAP code regen: `./scripts/tools/zap_regen_all.py`. +7. Add the device type spec to the test plan tools: + + 1. [tools/device_type_requirements](https://github.com/CHIP-Specifications/chip-test-plans/tree/master/tools/device_type_requirements) + 2. The file above is used by + [src/app/tests/suites/certification/Test_TC_DESC_2_1.yaml](https://github.com/project-chip/connectedhomeip/blob/master/src/app/tests/suites/certification/Test_TC_DESC_2_1.yaml) + + Note: the plan is to make the DM tools generate the device type requirements + data based on the spec, so the above will become obsolete. + +8. Add the device type to Chef: + [examples/chef/devices](https://github.com/project-chip/connectedhomeip/tree/master/examples/chef/devices) + +## Q & A + +**Q1**: What kind of devices can be used for the test events? Can one of them be +the example cluster server app running on a RasPI? Do the independent +realizations need to run on vendor hardware or can they also run on generic +hardware, such as ESP32 or RasPI? + +**A1**: one realization can be the test harness + the all clusters example app + +RasPI. the two independent realizations need to run on target hardware, which +may be mock-ups, prototypes etc + +**Q2**: How can the Chef tool be used for any of the deliverables above? + +**A2**: TBD + +**Q3**: What is the process for using the Zap tool in order to auto-generate +code and to commit the results to the git repo? + +**A3**: Search for zap_regen above. Add all the changed files to the repo after. + +**Q4**: Where can the older cluster definitions be found? + +**A4**: src/app/zap-templates/zcl/data-model/silabs/general.xml + +**Q5**: Where can I find ZAP documentation? + +**A5**: https://github.com/project-chip/zap/blob/master/README.md diff --git a/docs/cluster_and_device_type_dev/index.md b/docs/cluster_and_device_type_dev/index.md index 6d5cf71edfef1c..9159cdb3e511d5 100644 --- a/docs/cluster_and_device_type_dev/index.md +++ b/docs/cluster_and_device_type_dev/index.md @@ -13,3 +13,4 @@ types in the SDK. ``` - [Cluster and device type development](./cluster_and_device_type_dev.md) +- [How To Add New Device Types & Clusters](how_to_add_new_dts_and_clusters.md) From 4bd95e845ef4d8d61a10815a6ca6886cff8fc6b5 Mon Sep 17 00:00:00 2001 From: Rob Bultman Date: Thu, 11 Apr 2024 14:45:09 -0400 Subject: [PATCH 2/8] Add words --- .github/.wordlist.txt | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/.wordlist.txt b/.github/.wordlist.txt index 543786c1473460..b08c62ec3928fe 100644 --- a/.github/.wordlist.txt +++ b/.github/.wordlist.txt @@ -103,6 +103,7 @@ autocompletion AutoConf autoconnect autocrlf +autogen autogenerated automake autotools @@ -369,6 +370,7 @@ DelayedActionTime delayedApplyActionTimeSec delayedQueryActionTimeSec delayQuery +deliverables demangle deployable depottools @@ -727,6 +729,7 @@ JLink JLinkExe JLinkRTTClient JN +jni jpg jre js @@ -1438,6 +1441,7 @@ TrustedRootCertificates tsan TSG tsv +tt TTL tty ttyACM From 432a1a6567d799c4b7509375ab3e7b26ac3f2cb5 Mon Sep 17 00:00:00 2001 From: Rob Bultman Date: Fri, 12 Apr 2024 13:03:45 -0400 Subject: [PATCH 3/8] Update docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md Co-authored-by: Andrei Litvin --- .../how_to_add_new_dts_and_clusters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md b/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md index e0421ed9b4149d..5d8df0df3bc63b 100644 --- a/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md +++ b/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md @@ -165,7 +165,7 @@ related SDK tests. 2. In the `attributeAccessInterfaceAttributes` entry, add your new entry `"XYZ Mode": [ "SupportedModes", "CurrentMode", "FeatureMap" ]` - - this will mean ZAP won't autogen code to handle these attributes + this will mean ZAP won't generate code to handle these attributes 10. In [src/app/zap_cluster_list.json](https://github.com/project-chip/connectedhomeip/blob/master/src/app/zap_cluster_list.json): 1. Add your `XYZ_MODE_CLUSTER: []` to `ClientDirectories: { }` From 4be001294a692f6d8c9faf1cf290d5cb77abfd27 Mon Sep 17 00:00:00 2001 From: Rob Bultman Date: Fri, 12 Apr 2024 13:05:01 -0400 Subject: [PATCH 4/8] Remove word --- .github/.wordlist.txt | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/.wordlist.txt b/.github/.wordlist.txt index b08c62ec3928fe..90a4d622ad603c 100644 --- a/.github/.wordlist.txt +++ b/.github/.wordlist.txt @@ -103,7 +103,6 @@ autocompletion AutoConf autoconnect autocrlf -autogen autogenerated automake autotools From 2d70d82570409443c923040f120415378bdd2bea Mon Sep 17 00:00:00 2001 From: Rob Bultman Date: Fri, 12 Apr 2024 13:10:48 -0400 Subject: [PATCH 5/8] Update docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md Co-authored-by: Andrei Litvin --- .../how_to_add_new_dts_and_clusters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md b/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md index 5d8df0df3bc63b..7e600f5e43812c 100644 --- a/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md +++ b/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md @@ -186,7 +186,7 @@ related SDK tests. 1. [cluster_test_plan_template.adoc](https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/template/cluster_test_plan_template.adoc) 2. [test_plan_template.adoc](https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/template/test_plan_template.adoc) - Also, join this private Slack channel: csg-tt-test-plans + Also, join this private Slack channel: `csg-tt-test-plans` 5. Note: the CHIP-Tool reference client is generated from XML 6. If applicable, add tests: From d55d3fad5a6fa734ca751b711f1d42407e4480bd Mon Sep 17 00:00:00 2001 From: Rob Bultman Date: Fri, 12 Apr 2024 13:13:59 -0400 Subject: [PATCH 6/8] Update docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md --- .../how_to_add_new_dts_and_clusters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md b/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md index 7e600f5e43812c..c01a843b3f7be4 100644 --- a/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md +++ b/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md @@ -186,7 +186,7 @@ related SDK tests. 1. [cluster_test_plan_template.adoc](https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/template/cluster_test_plan_template.adoc) 2. [test_plan_template.adoc](https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/template/test_plan_template.adoc) - Also, join this private Slack channel: `csg-tt-test-plans` + Also, ask to join the private `csg-tt-test-plans` Slack channel. 5. Note: the CHIP-Tool reference client is generated from XML 6. If applicable, add tests: From b010724b6f40a8e0d8845e38aeb1d944a44db5a9 Mon Sep 17 00:00:00 2001 From: Rob Bultman Date: Fri, 12 Apr 2024 13:14:23 -0400 Subject: [PATCH 7/8] Update docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md --- .../how_to_add_new_dts_and_clusters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md b/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md index c01a843b3f7be4..8d73a257c28996 100644 --- a/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md +++ b/docs/cluster_and_device_type_dev/how_to_add_new_dts_and_clusters.md @@ -186,7 +186,7 @@ related SDK tests. 1. [cluster_test_plan_template.adoc](https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/template/cluster_test_plan_template.adoc) 2. [test_plan_template.adoc](https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/template/test_plan_template.adoc) - Also, ask to join the private `csg-tt-test-plans` Slack channel. + Also, ask to be added to the private `csg-tt-test-plans` Slack channel. 5. Note: the CHIP-Tool reference client is generated from XML 6. If applicable, add tests: From 72a5b2bd5e2a339eb6773347dc392c3d62ee027d Mon Sep 17 00:00:00 2001 From: Rob Bultman Date: Tue, 16 Apr 2024 10:29:46 -0400 Subject: [PATCH 8/8] Remove word from wordlist --- .github/.wordlist.txt | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/.wordlist.txt b/.github/.wordlist.txt index 90a4d622ad603c..f9ca1e4f827aeb 100644 --- a/.github/.wordlist.txt +++ b/.github/.wordlist.txt @@ -1440,7 +1440,6 @@ TrustedRootCertificates tsan TSG tsv -tt TTL tty ttyACM