TC-ACE-2.3: Add (test for command access and presence) #36808

cecille · 2024-12-11T21:10:18Z

Please see #36716 for why the OTA requestor command is skipped in CI. The goal of this PR is not to fix the OTA requestor issue, but to catch it in cert.

https://github.com/CHIP-Specifications/chip-test-plans/pull/4798 for test plan

Testing

XML changes - TC_AccessChecker.py from this PR is the test that caught the problem with the command access XML files that are fixed here
TC_AccessChecker.py - being run in CI against all clusters
additions to spec_parsing have unit-style tests in TestSpecParsingSupport.py

Note: Currently failing because of the OTA requestor.

semanticdiff-com · 2024-12-11T21:10:21Z

Review changes with

Changed Files

File	Status
src/python_testing/matter_testing_infrastructure/chip/testing/spec_parsing.py	75% smaller
src/python_testing/TC_AccessChecker.py	8% smaller
src/python_testing/TestSpecParsingSupport.py	3% smaller

github-actions · 2024-12-12T14:44:51Z

PR #36808: Size comparison from 41a9dea to c84bffa

Full report (3 builds for cc32xx, stm32)

platform	target	config	section	`41a9dea`	`c84bffa`
cc32xx	air-purifier	CC3235SF_LAUNCHXL	FLASH	631050	631050
			RAM	205824	205824
	lock	CC3235SF_LAUNCHXL	FLASH	669646	669646
			RAM	205968	205968
stm32	light	STM32WB5MM-DK	FLASH	484720	484720
			RAM	144880	144880

github-actions · 2024-12-13T18:58:59Z

PR #36808: Size comparison from 41a9dea to 39ca7ea

Full report (69 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, linux, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)

platform	target	config	section	`41a9dea`	`39ca7ea`	change	% change
bl602	lighting-app	bl602+mfd+littlefs+rpc	FLASH	1353340	1353340	0	0.0
			RAM	104112	104112	0	0.0
bl702	lighting-app	bl702+eth	FLASH	651826	651826	0	0.0
			RAM	25353	25353	0	0.0
		bl702+wifi	FLASH	829154	829154	0	0.0
			RAM	14093	14093	0	0.0
		bl706+mfd+rpc+littlefs	FLASH	1057626	1057626	0	0.0
			RAM	23933	23933	0	0.0
bl702l	lighting-app	bl702l+mfd+littlefs	FLASH	979000	979000	0	0.0
			RAM	16596	16596	0	0.0
cc13x4_26x4	lighting-app	LP_EM_CC1354P10_6	FLASH	839760	839760	0	0.0
			RAM	123664	123664	0	0.0
	lock-ftd	LP_EM_CC1354P10_6	FLASH	825308	825308	0	0.0
			RAM	125552	125552	0	0.0
	pump-app	LP_EM_CC1354P10_6	FLASH	772096	772096	0	0.0
			RAM	114020	114020	0	0.0
	pump-controller-app	LP_EM_CC1354P10_6	FLASH	756284	756300	16	0.0
			RAM	114228	114228	0	0.0
cc32xx	air-purifier	CC3235SF_LAUNCHXL	FLASH	631050	539605	-91445	-14.5
			RAM	205824	205760	-64	-0.0
	lock	CC3235SF_LAUNCHXL	FLASH	669646	573885	-95761	-14.3
			RAM	205968	205904	-64	-0.0
cyw30739	light	CYW30739B2-P5-EVK-01	unknown	2040	2040	0	0.0
			FLASH	681505	681513	8	0.0
			RAM	78724	78724	0	0.0
		CYW30739B2-P5-EVK-02	unknown	2040	2040	0	0.0
			FLASH	701349	701357	8	0.0
			RAM	81364	81364	0	0.0
		CYW30739B2-P5-EVK-03	unknown	2040	2040	0	0.0
			FLASH	701349	701357	8	0.0
			RAM	81364	81364	0	0.0
		CYW930739M2EVB-02	unknown	2040	2040	0	0.0
			FLASH	658293	658301	8	0.0
			RAM	73792	73792	0	0.0
	light-switch	CYW30739B2-P5-EVK-01	unknown	2040	2040	0	0.0
			FLASH	618065	618073	8	0.0
			RAM	71708	71708	0	0.0
		CYW30739B2-P5-EVK-02	unknown	2040	2040	0	0.0
			FLASH	637693	637701	8	0.0
			RAM	74252	74252	0	0.0
		CYW30739B2-P5-EVK-03	unknown	2040	2040	0	0.0
			FLASH	637693	637701	8	0.0
			RAM	74252	74252	0	0.0
	lock	CYW30739B2-P5-EVK-01	unknown	2040	2040	0	0.0
			FLASH	637465	637473	8	0.0
			RAM	74724	74724	0	0.0
		CYW30739B2-P5-EVK-02	unknown	2040	2040	0	0.0
			FLASH	657173	657181	8	0.0
			RAM	77268	77268	0	0.0
		CYW30739B2-P5-EVK-03	unknown	2040	2040	0	0.0
			FLASH	657173	657181	8	0.0
			RAM	77268	77268	0	0.0
	thermostat	CYW30739B2-P5-EVK-01	unknown	2040	2040	0	0.0
			FLASH	613925	613933	8	0.0
			RAM	68812	68812	0	0.0
		CYW30739B2-P5-EVK-02	unknown	2040	2040	0	0.0
			FLASH	633777	633785	8	0.0
			RAM	71444	71444	0	0.0
		CYW30739B2-P5-EVK-03	unknown	2040	2040	0	0.0
			FLASH	633777	633785	8	0.0
			RAM	71444	71444	0	0.0
efr32	lock-app	BRD4187C	FLASH	932340	932340	0	0.0
			RAM	160192	160192	0	0.0
		BRD4338a	FLASH	746144	746248	104	0.0
			RAM	233320	233320	0	0.0
	window-app	BRD4187C	FLASH	`1024784`	1024912	128	0.0
			RAM	128296	128296	0	0.0
esp32	all-clusters-app	c3devkit	DRAM	95360	95360	0	0.0
			FLASH	1543480	1543078	-402	-0.0
			IRAM	82542	82542	0	0.0
		m5stack	DRAM	116312	116312	0	0.0
			FLASH	1550134	1549678	-456	-0.0
			IRAM	117039	117039	0	0.0
linux	air-purifier-app	debug	unknown	4720	4720	0	0.0
			FLASH	2715063	2715629	566	0.0
			RAM	129800	129800	0	0.0
	all-clusters-app	debug	unknown	5560	5560	0	0.0
			FLASH	6007064	6007630	566	0.0
			RAM	523544	523544	0	0.0
	all-clusters-minimal-app	debug	unknown	5456	5456	0	0.0
			FLASH	5344804	5345370	566	0.0
			RAM	242600	242600	0	0.0
	bridge-app	debug	unknown	5440	5440	0	0.0
			FLASH	4684372	4684938	566	0.0
			RAM	218416	218416	0	0.0
	chip-tool	debug	unknown	5992	5992	0	0.0
			FLASH	12847832	12849310	1478	0.0
			RAM	582474	582506	32	0.0
	chip-tool-ipv6only	arm64	unknown	21352	21352	0	0.0
			FLASH	10982560	10983936	1376	0.0
			RAM	633392	633424	32	0.0
	fabric-admin	debug	unknown	5816	5816	0	0.0
			FLASH	11255157	11255859	702	0.0
			RAM	582850	582850	0	0.0
	fabric-bridge-app	debug	unknown	4696	4696	0	0.0
			FLASH	4509948	`4510514`	566	0.0
			RAM	205600	205600	0	0.0
	fabric-sync	debug	unknown	4936	4936	0	0.0
			FLASH	5608837	5610629	1792	0.0
			RAM	472584	472584	0	0.0
	lighting-app	debug+rpc+ui	unknown	6104	6104	0	0.0
			FLASH	5621073	`5621633`	560	0.0
			RAM	228792	228792	0	0.0
	lock-app	debug	unknown	5376	5376	0	0.0
			FLASH	`4733612`	`4734178`	566	0.0
			RAM	204776	204776	0	0.0
	ota-provider-app	debug	unknown	4752	4752	0	0.0
			FLASH	4359350	4359916	566	0.0
			RAM	198448	198448	0	0.0
	ota-requestor-app	debug	unknown	4688	4688	0	0.0
			FLASH	4498342	4498908	566	0.0
			RAM	203032	203032	0	0.0
	shell	debug	unknown	4248	4248	0	0.0
			FLASH	3030077	3030653	576	0.0
			RAM	160424	160424	0	0.0
	thermostat-no-ble	arm64	unknown	9536	9536	0	0.0
			FLASH	4103472	4104176	704	0.0
			RAM	243040	243040	0	0.0
	tv-app	debug	unknown	5704	5704	0	0.0
			FLASH	5958773	5959461	688	0.0
			RAM	596016	596016	0	0.0
	tv-casting-app	debug	unknown	5288	5288	0	0.0
			FLASH	11054589	11055165	576	0.0
			RAM	692184	692184	0	0.0
nrfconnect	all-clusters-app	nrf52840dk_nrf52840	FLASH	917616	917624	8	0.0
			RAM	143292	143292	0	0.0
		nrf7002dk_nrf5340_cpuapp	FLASH	890104	890104	0	0.0
			RAM	141487	141487	0	0.0
	all-clusters-minimal-app	nrf52840dk_nrf52840	FLASH	851760	851772	12	0.0
			RAM	142200	142200	0	0.0
nxp	contact	k32w0+release	FLASH	585440	585440	0	0.0
			RAM	71080	71080	0	0.0
		mcxw71+release	FLASH	600048	600048	0	0.0
			RAM	63176	63176	0	0.0
	light	k32w0+release	FLASH	612396	612412	16	0.0
			RAM	70472	70472	0	0.0
		k32w1+release	FLASH	686576	686592	16	0.0
			RAM	48808	48808	0	0.0
	lock	mcxw71+release	FLASH	762928	762928	0	0.0
			RAM	70844	70844	0	0.0
psoc6	all-clusters	cy8ckit_062s2_43012	FLASH	1646364	`1646492`	128	0.0
			RAM	212104	212104	0	0.0
	all-clusters-minimal	cy8ckit_062s2_43012	FLASH	`1554108`	1554236	128	0.0
			RAM	208904	208904	0	0.0
	light	cy8ckit_062s2_43012	FLASH	1469436	1469564	128	0.0
			RAM	200880	200880	0	0.0
	lock	cy8ckit_062s2_43012	FLASH	1467164	1467292	128	0.0
			RAM	225240	225240	0	0.0
qpg	lighting-app	qpg6105+debug	FLASH	664008	664024	16	0.0
			RAM	105424	105424	0	0.0
	lock-app	qpg6105+debug	FLASH	621796	621812	16	0.0
			RAM	99868	99868	0	0.0
stm32	light	STM32WB5MM-DK	FLASH	484720	484728	8	0.0
			RAM	144880	144880	0	0.0
telink	bridge-app	tlsr9258a	FLASH	682920	682916	-4	-0.0
			RAM	91208	91208	0	0.0
	contact-sensor-app	tlsr9528a_retention	FLASH	623350	623346	-4	-0.0
			RAM	31440	31440	0	0.0
	light-app-ota-compress-lzma-shell-factory-data	tl3218x	FLASH	772180	772176	-4	-0.0
			RAM	49300	49300	0	0.0
	light-switch-app-ota-compress-lzma-shell-factory-data	tlsr9528a	FLASH	710774	710770	-4	-0.0
			RAM	73504	73504	0	0.0
	lighting-app-ota-factory-data	tlsr9118bdk40d	FLASH	627794	627790	-4	-0.0
			RAM	142140	142140	0	0.0
	lighting-app-ota-rpc-factory-data-4mb	tlsr9518adk80d	FLASH	813808	813804	-4	-0.0
			RAM	99684	99684	0	0.0
tizen	all-clusters-app	arm	unknown	4988	4988	0	0.0
			FLASH	`1732528`	1732944	416	0.0
			RAM	90744	90744	0	0.0
	chip-tool-ubsan	arm	unknown	10804	10804	0	0.0
			FLASH	17969958	17973406	3448	0.0
			RAM	7840924	7842608	1684	0.0

github-actions · 2025-02-20T16:07:14Z

PR #36808: Size comparison from 4fa81f6 to c1ef758

Full report (73 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, linux, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)

platform	target	config	section	`4fa81f6`	`c1ef758`
bl602	lighting-app	bl602+mfd+littlefs+rpc	FLASH	1095124	1095124
			RAM	94802	94802
bl702	lighting-app	bl702+eth	FLASH	651570	651570
			RAM	33509	33509
		bl702+wifi	FLASH	827734	827734
			RAM	22233	22233
		bl706+mfd+rpc+littlefs	FLASH	1060896	1060896
			RAM	32157	32157
bl702l	contact-sensor-app	bl702l+mfd+littlefs	FLASH	892082	892082
			RAM	26896	26896
	lighting-app	bl702l+mfd+littlefs	FLASH	974978	974978
			RAM	24644	24644
cc13x4_26x4	lighting-app	LP_EM_CC1354P10_6	FLASH	815252	815252
			RAM	120256	120256
	lock-ftd	LP_EM_CC1354P10_6	FLASH	824164	824164
			RAM	125352	125352
	pump-app	LP_EM_CC1354P10_6	FLASH	771032	771032
			RAM	113724	113724
	pump-controller-app	LP_EM_CC1354P10_6	FLASH	755308	755308
			RAM	113932	113932
cc32xx	air-purifier	CC3235SF_LAUNCHXL	FLASH	538802	538802
			RAM	205112	205112
	lock	CC3235SF_LAUNCHXL	FLASH	572950	572950
			RAM	205360	205360
cyw30739	light	CYW30739B2-P5-EVK-01	unknown	2040	2040
			FLASH	656381	656381
			RAM	75324	75324
		CYW30739B2-P5-EVK-02	unknown	2040	2040
			FLASH	676241	676241
			RAM	77964	77964
		CYW30739B2-P5-EVK-03	unknown	2040	2040
			FLASH	676241	676241
			RAM	77964	77964
		CYW930739M2EVB-02	unknown	2040	2040
			FLASH	633165	633165
			RAM	70392	70392
	light-switch	CYW30739B2-P5-EVK-01	unknown	2040	2040
			FLASH	616245	616245
			RAM	71532	71532
		CYW30739B2-P5-EVK-02	unknown	2040	2040
			FLASH	635881	635881
			RAM	74076	74076
		CYW30739B2-P5-EVK-03	unknown	2040	2040
			FLASH	635881	635881
			RAM	74076	74076
	lock	CYW30739B2-P5-EVK-01	unknown	2040	2040
			FLASH	635741	635741
			RAM	74540	74540
		CYW30739B2-P5-EVK-02	unknown	2040	2040
			FLASH	655457	655457
			RAM	77084	77084
		CYW30739B2-P5-EVK-03	unknown	2040	2040
			FLASH	655457	655457
			RAM	77084	77084
	thermostat	CYW30739B2-P5-EVK-01	unknown	2040	2040
			FLASH	612089	612089
			RAM	68628	68628
		CYW30739B2-P5-EVK-02	unknown	2040	2040
			FLASH	631949	631949
			RAM	71268	71268
		CYW30739B2-P5-EVK-03	unknown	2040	2040
			FLASH	631949	631949
			RAM	71268	71268
efr32	lock-app	BRD4187C	FLASH	937712	937712
			RAM	159916	159916
		BRD4338a	FLASH	731760	731760
			RAM	234772	234772
	window-app	BRD4187C	FLASH	1030368	1030368
			RAM	128020	128020
esp32	all-clusters-app	c3devkit	DRAM	97312	97312
			FLASH	1581916	1581916
			IRAM	83820	83820
		m5stack	DRAM	116100	116100
			FLASH	1549914	1549914
			IRAM	117039	117039
linux	air-purifier-app	debug	unknown	4720	4720
			FLASH	`2649535`	`2649535`
			RAM	111088	111088
	all-clusters-app	debug	unknown	5528	5528
			FLASH	5921072	5921072
			RAM	513712	513712
	all-clusters-minimal-app	debug	unknown	5424	5424
			FLASH	5266042	5266042
			RAM	221112	221112
	bridge-app	debug	unknown	5440	5440
			FLASH	4624718	4624718
			RAM	199816	199816
	camera-app	debug	unknown	5424	5424
			FLASH	4671498	4671498
			RAM	194592	194592
	chip-tool	debug	unknown	6096	6096
			FLASH	13300341	13300341
			RAM	603392	603392
	chip-tool-ipv6only	arm64	unknown	21976	21976
			FLASH	11321680	11321680
			RAM	655184	655184
	fabric-admin	debug	unknown	5784	5784
			FLASH	11564903	11564903
			RAM	603176	603176
	fabric-bridge-app	debug	unknown	4688	4688
			FLASH	`4450040`	`4450040`
			RAM	187016	187016
	fabric-sync	debug	unknown	4944	4944
			FLASH	5568885	5568885
			RAM	470400	470400
	lighting-app	debug+rpc+ui	unknown	6152	6152
			FLASH	5514129	5514129
			RAM	203952	203952
	lock-app	debug	unknown	5392	5392
			FLASH	4688154	4688154
			RAM	191144	191144
	ota-provider-app	debug	unknown	4728	4728
			FLASH	4310334	4310334
			RAM	179832	179832
	ota-requestor-app	debug	unknown	4680	4680
			FLASH	4440492	4440492
			RAM	184320	184320
	shell	debug	unknown	4216	4216
			FLASH	2948844	2948844
			RAM	143672	143672
	thermostat-no-ble	arm64	unknown	9448	9448
			FLASH	`4041864`	`4041864`
			RAM	228096	228096
	tv-app	debug	unknown	5720	5720
			FLASH	5907797	5907797
			RAM	593832	593832
	tv-casting-app	debug	unknown	5296	5296
			FLASH	11471181	11471181
			RAM	718656	718656
nrfconnect	all-clusters-app	nrf52840dk_nrf52840	FLASH	907352	907352
			RAM	142315	142315
		nrf7002dk_nrf5340_cpuapp	FLASH	901716	901716
			RAM	124663	124663
	all-clusters-minimal-app	nrf52840dk_nrf52840	FLASH	845956	845956
			RAM	141243	141243
nxp	contact	k32w0+release	FLASH	585168	585168
			RAM	70876	70876
		mcxw71+release	FLASH	600640	600640
			RAM	63096	63096
	light	k32w0+release	FLASH	611156	611156
			RAM	70164	70164
		k32w1+release	FLASH	685632	685632
			RAM	48584	48584
	lock	mcxw71+release	FLASH	749464	749464
			RAM	67500	67500
psoc6	all-clusters	cy8ckit_062s2_43012	FLASH	1646156	1646156
			RAM	211472	211472
	all-clusters-minimal	cy8ckit_062s2_43012	FLASH	1552908	1552908
			RAM	208288	208288
	light	cy8ckit_062s2_43012	FLASH	1438596	1438596
			RAM	197040	197040
	lock	cy8ckit_062s2_43012	FLASH	1467484	1467484
			RAM	224704	224704
qpg	lighting-app	qpg6105+debug	FLASH	662140	662140
			RAM	105116	105116
	lock-app	qpg6105+debug	FLASH	620248	620248
			RAM	99664	99664
stm32	light	STM32WB5MM-DK	FLASH	459784	459784
			RAM	141472	141472
telink	bridge-app	tl7218x	FLASH	664988	664988
			RAM	90728	90728
	contact-sensor-app	tlsr9528a_retention	FLASH	622032	622032
			RAM	31488	31488
	light-app-ota-shell-factory-data	tl3218x	FLASH	745172	745172
			RAM	40396	40396
		tl7218x	FLASH	753734	753734
			RAM	97540	97540
	light-switch-app-ota-compress-lzma-factory-data	tl7218x_retention	FLASH	680804	680804
			RAM	52192	52192
	light-switch-app-ota-compress-lzma-shell-factory-data	tlsr9528a	FLASH	709366	709366
			RAM	73400	73400
	lighting-app-ota-factory-data	tlsr9118bdk40d	FLASH	600546	600546
			RAM	138812	138812
	lighting-app-ota-rpc-factory-data-4mb	tlsr9518adk80d	FLASH	788688	788688
			RAM	96388	96388
tizen	all-clusters-app	arm	unknown	5076	5076
			FLASH	1750632	1750632
			RAM	93260	93260
	chip-tool-ubsan	arm	unknown	11492	11492
			FLASH	18980782	18980782
			RAM	8305600	8305600

cecille added 10 commits December 3, 2024 22:53

TC-IDM-10.7: Implement command checking test

3e78f34

Note: Currently failing because of the OTA requestor.

skip sending some commands, including workaround for OTA in CI

eb73c7c

Add invoke privilege parsing for commands

21e2552

Start of ACE tests

f36cc1b

check

2ed6434

add test

65ce525

Merge remote-tracking branch 'upstream/master' into IDM_10_7

ef5d474

back out changes in device composition

6e5ef85

couple of fixes, timed commands, new id check functions

58defdd

Add OTA exception

3f879bd

github-actions bot added tests matter-1.4-te2-script-change Script changes before end of Matter 1.4 TE2 labels Dec 11, 2024

Restyled by isort

c84bffa

Linter

39ca7ea

cecille added 3 commits December 16, 2024 15:03

separate test runs so longs don't overflow

1dee9cc

General diagnostics: Fix access on PayloadTestRequest

0b11911

Scenes management: Fix access on CopyScene

374fa2d

github-actions bot added examples scripts controller app labels Dec 16, 2024

This was referenced Dec 16, 2024

General diagnostics: Fix access on PayloadTestRequest #36864

Merged

Scenes management: Fix access on CopyScene #36865

Merged

cecille removed the matter-1.4-te2-script-change Script changes before end of Matter 1.4 TE2 label Dec 17, 2024

cecille marked this pull request as ready for review January 19, 2025 17:32

cecille requested review from a team as code owners January 19, 2025 17:32

pullapprove bot added the review - pending label Jan 19, 2025

Merge remote-tracking branch 'upstream/master' into ACE_2_3

c1ef758

Fix bad merge

9bbb9ae

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TC-ACE-2.3: Add (test for command access and presence) #36808

TC-ACE-2.3: Add (test for command access and presence) #36808

cecille commented Dec 11, 2024 •

edited

Loading

semanticdiff-com bot commented Dec 11, 2024 •

edited

Loading

github-actions bot commented Dec 12, 2024

github-actions bot commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Feb 20, 2025 •

edited

Loading

TC-ACE-2.3: Add (test for command access and presence) #36808

Are you sure you want to change the base?

TC-ACE-2.3: Add (test for command access and presence) #36808

Conversation

cecille commented Dec 11, 2024 • edited Loading

Testing

semanticdiff-com bot commented Dec 11, 2024 • edited Loading

github-actions bot commented Dec 12, 2024

github-actions bot commented Dec 13, 2024 • edited Loading

github-actions bot commented Feb 20, 2025 • edited Loading

cecille commented Dec 11, 2024 •

edited

Loading

semanticdiff-com bot commented Dec 11, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Feb 20, 2025 •

edited

Loading