Skip to content

Commit 33dda32

Browse files
j-ororkerestyled-commitscecille
authoredDec 16, 2024··
[Fix] Possible fix for matter-test-scripts issue #227: Remove PICS from OPSTATE tests (#34290)
* Possible fix for matter-test-scripts issue #227: - Removed PICS checks and replaced with attribute and command checks from endpoint during tests. * Restyled by autopep8 * Updated TC_OpstateCommon.py: - Removed some variables that were no longer needed * Updated TC_RVCOPSTATE_2_1 test module: - Removed automatable PICS checks and replaced with attributes available to be gathered from endpoint. * Restyled by autopep8 * Restyled by isort * Updated TC_RVCOPSTATE_2_1 test module: - Adding back in missing time import and test runner comments into test module * Updated TC_RVCOPSTATE_2_1 test module: - Replaced input() with wait_for_user_input() in test script - Added back in short sleep to script, not sure why it got removed. * Resolving Linting issue in TC_RVCOPSTATE_2_1: - Had to remove "test_step" variable that was attempting to be called in wait_for_user_input() as it was not being defined earlier in the test module. * Updating TC_RVCOPSTATE_2_1 test module: - Minor change to remove unneeded f-string from wait_for_user_input(). * Updated TC_RVCOPSTATE_2_1 test module: - Re-imported the test_step variable for test steps 6 and 7 manual testing that were accidentally removed - Re-imported the variable being called in the self.wait_for_user_input() * Updated TC_RVCOPSTATE_2_1 test module: - Replaced missing test_step variable and calls for it in self.wait_for_user_input() * Restyled by autopep8 * Updated TC_OpstateCommon and TC_RVCOPSTATE_2_1: - Removed oprtnlstate_attr_id variable and if statements as it is a mandatory attribute. - Created new common functions in OpstateCommon test module to create dictionary containing attributes and commands. - Renamed some variables that had upper case letters to contain only lower case letters. - Removed creating variable for events and returned PICS checks for those in TC_OpstateCommon test module as not currently able to automate * Restyled by autopep8 * Updated TC_OpstateCommon module: - Removed unneeded local variable "phase_list_attr_id" as lint mentioned it is not being used in test 2_3. * Updated TC_OpstateCommon and TC_RVCOPSTATE_2_1: - Removed variable functions and replaced with calling named attributes in if checks directly. * Restyled by autopep8 * Updated matter_testing_support, OpstateCommon, and RVCOPSTATE_2_1 modules: - Added attributes_guard to matter_testing_support helper module to check if attributes are in attributes list using has_attributes function - Changed attributes checks to using attributes_guard function in OpstateCommon and RVCOPSTATE_2_1 test modules * Restyled by autopep8 * Restyled by isort * Updated TC_OpstateCommon.py: - Resolved linting errors * Updating method for attributes_guard functionality * Updated TC_RVCOPSTATE_2_1 test module: - Debugging to find issue why test is failing in CI pipeline. * Updating TC_RVCOPSTATE_2_1 test module: - Continuing effort to resolve issue with CI pipeline * Updated TC_OpstateCommon, TC_RVCOPSTATE_2_1, and matter_testing support: - Resolved issues with attributes_guard function in matter_testing support module * Restyled by autopep8 * Updated TC_RVCOPSTATE_2_1 test module: - changed verbosity in CI arguments to make it quieter. * Updating matter_testing support module: - Updated attributes_guard function to make it async * Updating TC_OpstateCommon and TC_RVCOPSTATE_2_1 test modules: - Updated method for attributes_guard functionality. - Added additional check to make sure that endpoint is not 0 or not provided in command line * Restyled by autopep8 * Updated matter_testing helper module: - Resolved linting error * Updating TC_RVCOPSTATE_2_1 test module: - Resolving linting error * Update src/python_testing/matter_testing_infrastructure/chip/testing/matter_testing.py Adding coding change from Cecille! Co-authored-by: C Freeman <cecille@google.com> * Updating OPSTATECommon and RVCOPSTATE_2_1 modules: - Updated to using attribute_guard function in place of attributes_guard. * Updated matter_testing, added new TC_TestAttrAvail modules: - Updated matter_testing support module to include new command_guard() and feature_guard() - Created standalone test to show that guard functionality works for CASE, PASE, and no factory reset commissioning - Added TC_TestAttrAvail to slow tests as it takes ~30 seconds to run the tests * Restyled by autopep8 * Restyled by isort * Updated TC_OpstateCommon python module: - Updated to using new command_guard() for OPSTATE tests * Updated matter_testing, TC_OpstateCommon, and TC_TestAttrAvail modules: - Resolving linting errors * Restyled by isort * Update src/python_testing/matter_testing_infrastructure/chip/testing/matter_testing.py Adding commi from Cecille to the code here to help better clarify and explain reasoning for this coding change. Co-authored-by: C Freeman <cecille@google.com> --------- Co-authored-by: Restyled.io <commits@restyled.io> Co-authored-by: C Freeman <cecille@google.com>
1 parent 43f66f0 commit 33dda32

File tree

5 files changed

+433
-185
lines changed

5 files changed

+433
-185
lines changed
 

‎src/python_testing/TC_OpstateCommon.py

+157-178
Large diffs are not rendered by default.

‎src/python_testing/TC_RVCOPSTATE_2_1.py

+10-7
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,8 @@ def TC_RVCOPSTATE_2_1(self) -> list[str]:
8585

8686
@async_test_body
8787
async def test_TC_RVCOPSTATE_2_1(self):
88+
if self.matter_test_config.endpoint is None or self.matter_test_config.endpoint == 0:
89+
asserts.fail("--endpoint must be set and not set to 0 for this test to run correctly.")
8890
self.endpoint = self.get_endpoint()
8991
asserts.assert_false(self.endpoint is None, "--endpoint <endpoint> must be included on the command line in.")
9092
self.is_ci = self.check_pics("PICS_SDK_CI_ONLY")
@@ -94,15 +96,16 @@ async def test_TC_RVCOPSTATE_2_1(self):
9496
asserts.fail("The --app-pid flag must be set when PICS_SDK_CI_ONLY is set")
9597
self.app_pipe = self.app_pipe + str(app_pid)
9698

97-
attributes = Clusters.RvcOperationalState.Attributes
99+
cluster = Clusters.RvcOperationalState
100+
attributes = cluster.Attributes
98101

99102
self.print_step(1, "Commissioning, already done")
100103

101104
# Ensure that the device is in the correct state
102105
if self.is_ci:
103106
self.write_to_app_pipe({"Name": "Reset"})
104107

105-
if self.check_pics("RVCOPSTATE.S.A0000"):
108+
if await self.attribute_guard(endpoint=self.endpoint, attribute=attributes.PhaseList):
106109
self.print_step(2, "Read PhaseList attribute")
107110
phase_list = await self.read_mod_attribute_expect_success(endpoint=self.endpoint, attribute=attributes.PhaseList)
108111

@@ -115,7 +118,7 @@ async def test_TC_RVCOPSTATE_2_1(self):
115118

116119
asserts.assert_less_equal(phase_list_len, 32, "PhaseList length(%d) must be less than 32!" % phase_list_len)
117120

118-
if self.check_pics("RVCOPSTATE.S.A0001"):
121+
if await self.attribute_guard(endpoint=self.endpoint, attribute=attributes.CurrentPhase):
119122
self.print_step(3, "Read CurrentPhase attribute")
120123
current_phase = await self.read_mod_attribute_expect_success(endpoint=self.endpoint, attribute=attributes.CurrentPhase)
121124
logging.info("CurrentPhase: %s" % (current_phase))
@@ -126,7 +129,7 @@ async def test_TC_RVCOPSTATE_2_1(self):
126129
asserts.assert_true(0 <= current_phase < phase_list_len,
127130
"CurrentPhase(%s) must be between 0 and %d" % (current_phase, (phase_list_len - 1)))
128131

129-
if self.check_pics("RVCOPSTATE.S.A0002"):
132+
if await self.attribute_guard(endpoint=self.endpoint, attribute=attributes.CountdownTime):
130133
self.print_step(4, "Read CountdownTime attribute")
131134
countdown_time = await self.read_mod_attribute_expect_success(endpoint=self.endpoint,
132135
attribute=attributes.CountdownTime)
@@ -136,7 +139,7 @@ async def test_TC_RVCOPSTATE_2_1(self):
136139
asserts.assert_true(countdown_time >= 0 and countdown_time <= 259200,
137140
"CountdownTime(%s) must be between 0 and 259200" % countdown_time)
138141

139-
if self.check_pics("RVCOPSTATE.S.A0003"):
142+
if await self.attribute_guard(endpoint=self.endpoint, attribute=attributes.OperationalStateList):
140143
self.print_step(5, "Read OperationalStateList attribute")
141144
operational_state_list = await self.read_mod_attribute_expect_success(endpoint=self.endpoint,
142145
attribute=attributes.OperationalStateList)
@@ -159,7 +162,7 @@ async def test_TC_RVCOPSTATE_2_1(self):
159162

160163
asserts.assert_true(error_state_present, "The OperationalStateList does not have an ID entry of Error(0x03)")
161164

162-
if self.check_pics("RVCOPSTATE.S.A0004"):
165+
if await self.attribute_guard(endpoint=self.endpoint, attribute=attributes.OperationalState):
163166
self.print_step(6, "Read OperationalState attribute")
164167
operational_state = await self.read_mod_attribute_expect_success(endpoint=self.endpoint,
165168
attribute=attributes.OperationalState)
@@ -226,7 +229,7 @@ async def test_TC_RVCOPSTATE_2_1(self):
226229
self.wait_for_user_input(prompt_msg=f"{test_step}, and press Enter when done.\n")
227230
await self.read_and_validate_opstate(step="6n", expected_state=Clusters.RvcOperationalState.Enums.OperationalStateEnum.kDocked)
228231

229-
if self.check_pics("RVCOPSTATE.S.A0005"):
232+
if await self.attribute_guard(endpoint=self.endpoint, attribute=attributes.OperationalError):
230233
self.print_step(7, "Read OperationalError attribute")
231234
operational_error = await self.read_mod_attribute_expect_success(endpoint=self.endpoint,
232235
attribute=attributes.OperationalError)
+164
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
#
2+
# Copyright (c) 2023 Project CHIP Authors
3+
# All rights reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
# See https://github.com/project-chip/connectedhomeip/blob/master/docs/testing/python.md#defining-the-ci-test-arguments
19+
# for details about the block below.
20+
#
21+
# === BEGIN CI TEST ARGUMENTS ===
22+
# test-runner-runs:
23+
# run1:
24+
# app: ${ALL_CLUSTERS_APP}
25+
# app-args: --discriminator 1234 --KVS kvs1 --trace-to json:${TRACE_APP}.json
26+
# script-args: >
27+
# --storage-path admin_storage.json
28+
# --manual-code 10054912339
29+
# --PICS src/app/tests/suites/certification/ci-pics-values
30+
# --trace-to json:${TRACE_TEST_JSON}.json
31+
# --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
32+
# --endpoint 1
33+
# factory-reset: true
34+
# quiet: true
35+
# run2:
36+
# app: ${ALL_CLUSTERS_APP}
37+
# app-args: --discriminator 1234 --passcode 20202021 --KVS kvs1
38+
# script-args: >
39+
# --storage-path admin_storage.json
40+
# --discriminator 1234
41+
# --passcode 20202021
42+
# --endpoint 1
43+
# --commissioning-method on-network
44+
# factory-reset: true
45+
# quiet: true
46+
# run3:
47+
# app: ${ALL_CLUSTERS_APP}
48+
# app-args: --discriminator 1234 --KVS kvs1
49+
# script-args: >
50+
# --storage-path admin_storage.json
51+
# --endpoint 1
52+
# --discriminator 1234
53+
# --passcode 20202021
54+
# factory-reset: false
55+
# quiet: true
56+
# === END CI TEST ARGUMENTS ===
57+
58+
# Run 1: Tests PASE connection using manual code
59+
# Run 2: Tests CASE connection using manual discriminator and passcode
60+
# Run 3: Tests without factory reset
61+
62+
import asyncio
63+
64+
import chip.clusters as Clusters
65+
from chip.testing.matter_testing import MatterBaseTest, TestStep, async_test_body, default_matter_test_main
66+
from mobly import asserts
67+
68+
69+
class TC_TestAttrAvail(MatterBaseTest):
70+
# Using get_code and a modified version of setup_class_helper functions from chip.testing.basic_composition module
71+
def get_code(self, dev_ctrl):
72+
created_codes = []
73+
for idx, discriminator in enumerate(self.matter_test_config.discriminators):
74+
created_codes.append(dev_ctrl.CreateManualCode(discriminator, self.matter_test_config.setup_passcodes[idx]))
75+
76+
setup_codes = self.matter_test_config.qr_code_content + self.matter_test_config.manual_code + created_codes
77+
if not setup_codes:
78+
return None
79+
asserts.assert_equal(len(setup_codes), 1,
80+
"Require exactly one of either --qr-code, --manual-code or (--discriminator and --passcode).")
81+
return setup_codes[0]
82+
83+
async def setup_class_helper(self, allow_pase: bool = True):
84+
dev_ctrl = self.default_controller
85+
self.problems = []
86+
87+
node_id = self.dut_node_id
88+
89+
task_list = []
90+
if allow_pase and self.get_code(dev_ctrl):
91+
setup_code = self.get_code(dev_ctrl)
92+
pase_future = dev_ctrl.EstablishPASESession(setup_code, self.dut_node_id)
93+
task_list.append(asyncio.create_task(pase_future))
94+
95+
case_future = dev_ctrl.GetConnectedDevice(nodeid=node_id, allowPASE=False)
96+
task_list.append(asyncio.create_task(case_future))
97+
98+
for task in task_list:
99+
asyncio.ensure_future(task)
100+
101+
done, pending = await asyncio.wait(task_list, return_when=asyncio.FIRST_COMPLETED)
102+
103+
for task in pending:
104+
try:
105+
task.cancel()
106+
await task
107+
except asyncio.CancelledError:
108+
pass
109+
110+
wildcard_read = (await dev_ctrl.Read(node_id, [()]))
111+
112+
# ======= State kept for use by all tests =======
113+
# All endpoints in "full object" indexing format
114+
self.endpoints = wildcard_read.attributes
115+
116+
def steps_TC_TestAttrAvail(self) -> list[TestStep]:
117+
return [
118+
TestStep(1, "Commissioning, already done", is_commissioning=True),
119+
TestStep(2, "Checking OperationalState attribute is available on endpoint"),
120+
TestStep(3, "Checking Operational Resume command is available on endpoint"),
121+
TestStep(4, "Checking Timezone feature is available on endpoint"),
122+
]
123+
124+
def TC_TestAttrAvail(self) -> list[str]:
125+
return ["RVCOPSTATE.S"]
126+
127+
@async_test_body
128+
async def setup_class(self):
129+
super().setup_class()
130+
await self.setup_class_helper()
131+
132+
# ======= START OF ACTUAL TESTS =======
133+
@async_test_body
134+
async def test_TC_TestAttrAvail(self):
135+
self.step(1)
136+
137+
if self.matter_test_config.endpoint is None or self.matter_test_config.endpoint == 0:
138+
asserts.fail("--endpoint must be set and not set to 0 for this test to run correctly.")
139+
self.endpoint = self.get_endpoint()
140+
asserts.assert_false(self.endpoint is None, "--endpoint <endpoint> must be included on the command line in.")
141+
142+
cluster = Clusters.RvcOperationalState
143+
attributes = cluster.Attributes
144+
commands = cluster.Commands
145+
self.th1 = self.default_controller
146+
147+
self.step(2)
148+
attr_should_be_there = await self.attribute_guard(endpoint=self.endpoint, attribute=attributes.OperationalState)
149+
asserts.assert_true(attr_should_be_there, True)
150+
self.print_step("Operational State Attr", attr_should_be_there)
151+
152+
self.step(3)
153+
cmd_should_be_there = await self.command_guard(endpoint=self.endpoint, command=commands.Resume)
154+
asserts.assert_true(cmd_should_be_there, True)
155+
self.print_step("Operational Resume Command available ", cmd_should_be_there)
156+
157+
self.step(4)
158+
feat_should_be_there = await self.feature_guard(endpoint=self.endpoint, cluster=Clusters.BooleanStateConfiguration, feature_int=Clusters.BooleanStateConfiguration.Bitmaps.Feature.kAudible)
159+
asserts.assert_true(feat_should_be_there, True)
160+
self.print_step("Boolean State Config Audio Feature available ", feat_should_be_there)
161+
162+
163+
if __name__ == "__main__":
164+
default_matter_test_main()

‎src/python_testing/matter_testing_infrastructure/chip/testing/matter_testing.py

+101
Original file line numberDiff line numberDiff line change
@@ -1124,6 +1124,12 @@ def setup_class(self):
11241124
self.current_step_index = 0
11251125
self.step_start_time = datetime.now(timezone.utc)
11261126
self.step_skipped = False
1127+
self.global_wildcard = asyncio.wait_for(self.default_controller.Read(self.dut_node_id, [(Clusters.Descriptor), Attribute.AttributePath(None, None, GlobalAttributeIds.ATTRIBUTE_LIST_ID), Attribute.AttributePath(
1128+
None, None, GlobalAttributeIds.FEATURE_MAP_ID), Attribute.AttributePath(None, None, GlobalAttributeIds.ACCEPTED_COMMAND_LIST_ID)]), timeout=60)
1129+
# self.stored_global_wildcard stores value of self.global_wildcard after first async call.
1130+
# Because setup_class can be called before commissioning, this variable is lazy-initialized
1131+
# where the read is deferred until the first guard function call that requires global attributes.
1132+
self.stored_global_wildcard = None
11271133

11281134
def setup_test(self):
11291135
self.current_step_index = 0
@@ -1455,6 +1461,66 @@ def pics_guard(self, pics_condition: bool):
14551461
self.mark_current_step_skipped()
14561462
return pics_condition
14571463

1464+
async def attribute_guard(self, endpoint: int, attribute: ClusterObjects.ClusterAttributeDescriptor):
1465+
"""Similar to pics_guard above, except checks a condition and if False marks the test step as skipped and
1466+
returns False using attributes against attributes_list, otherwise returns True.
1467+
For example can be used to check if a test step should be run:
1468+
1469+
self.step("1")
1470+
if self.attribute_guard(condition1_needs_to_be_true_to_execute):
1471+
# do the test for step 1
1472+
1473+
self.step("2")
1474+
if self.attribute_guard(condition2_needs_to_be_false_to_skip_step):
1475+
# skip step 2 if condition not met
1476+
"""
1477+
if self.stored_global_wildcard is None:
1478+
self.stored_global_wildcard = await self.global_wildcard
1479+
attr_condition = _has_attribute(wildcard=self.stored_global_wildcard, endpoint=endpoint, attribute=attribute)
1480+
if not attr_condition:
1481+
self.mark_current_step_skipped()
1482+
return attr_condition
1483+
1484+
async def command_guard(self, endpoint: int, command: ClusterObjects.ClusterCommand):
1485+
"""Similar to attribute_guard above, except checks a condition and if False marks the test step as skipped and
1486+
returns False using command id against AcceptedCmdsList, otherwise returns True.
1487+
For example can be used to check if a test step should be run:
1488+
1489+
self.step("1")
1490+
if self.command_guard(condition1_needs_to_be_true_to_execute):
1491+
# do the test for step 1
1492+
1493+
self.step("2")
1494+
if self.command_guard(condition2_needs_to_be_false_to_skip_step):
1495+
# skip step 2 if condition not met
1496+
"""
1497+
if self.stored_global_wildcard is None:
1498+
self.stored_global_wildcard = await self.global_wildcard
1499+
cmd_condition = _has_command(wildcard=self.stored_global_wildcard, endpoint=endpoint, command=command)
1500+
if not cmd_condition:
1501+
self.mark_current_step_skipped()
1502+
return cmd_condition
1503+
1504+
async def feature_guard(self, endpoint: int, cluster: ClusterObjects.ClusterObjectDescriptor, feature_int: IntFlag):
1505+
"""Similar to command_guard and attribute_guard above, except checks a condition and if False marks the test step as skipped and
1506+
returns False using feature id against feature_map, otherwise returns True.
1507+
For example can be used to check if a test step should be run:
1508+
1509+
self.step("1")
1510+
if self.feature_guard(condition1_needs_to_be_true_to_execute):
1511+
# do the test for step 1
1512+
1513+
self.step("2")
1514+
if self.feature_guard(condition2_needs_to_be_false_to_skip_step):
1515+
# skip step 2 if condition not met
1516+
"""
1517+
if self.stored_global_wildcard is None:
1518+
self.stored_global_wildcard = await self.global_wildcard
1519+
feat_condition = _has_feature(wildcard=self.stored_global_wildcard, endpoint=endpoint, cluster=cluster, feature=feature_int)
1520+
if not feat_condition:
1521+
self.mark_current_step_skipped()
1522+
return feat_condition
1523+
14581524
def mark_current_step_skipped(self):
14591525
try:
14601526
steps = self.get_test_steps(self.current_test_info.name)
@@ -2105,6 +2171,41 @@ def has_attribute(attribute: ClusterObjects.ClusterAttributeDescriptor) -> Endpo
21052171
return partial(_has_attribute, attribute=attribute)
21062172

21072173

2174+
def _has_command(wildcard, endpoint, command: ClusterObjects.ClusterCommand) -> bool:
2175+
cluster = get_cluster_from_command(command)
2176+
try:
2177+
cmd_list = wildcard.attributes[endpoint][cluster][cluster.Attributes.AcceptedCommandList]
2178+
if not isinstance(cmd_list, list):
2179+
asserts.fail(
2180+
f"Failed to read mandatory AcceptedCommandList command value for cluster {cluster} on endpoint {endpoint}: {cmd_list}.")
2181+
return command.command_id in cmd_list
2182+
except KeyError:
2183+
return False
2184+
2185+
2186+
def has_command(command: ClusterObjects.ClusterCommand) -> EndpointCheckFunction:
2187+
""" EndpointCheckFunction that can be passed as a parameter to the run_if_endpoint_matches decorator.
2188+
2189+
Use this function with the run_if_endpoint_matches decorator to run this test on all endpoints with
2190+
the specified attribute. For example, given a device with the following conformance
2191+
2192+
EP0: cluster A, B, C
2193+
EP1: cluster D with command d, E
2194+
EP2, cluster D with command d
2195+
EP3, cluster D without command d
2196+
2197+
And the following test specification:
2198+
@run_if_endpoint_matches(has_command(Clusters.D.Commands.d))
2199+
test_mytest(self):
2200+
...
2201+
2202+
If you run this test with --endpoint 1 or --endpoint 2, the test will be run. If you run this test
2203+
with any other --endpoint the run_if_endpoint_matches decorator will call the on_skip function to
2204+
notify the test harness that the test is not applicable to this node and the test will not be run.
2205+
"""
2206+
return partial(_has_command, command=command)
2207+
2208+
21082209
def _has_feature(wildcard, endpoint: int, cluster: ClusterObjects.ClusterObjectDescriptor, feature: IntFlag) -> bool:
21092210
try:
21102211
feature_map = wildcard.attributes[endpoint][cluster][cluster.Attributes.FeatureMap]

‎src/python_testing/test_metadata.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,7 @@ slow_tests:
9797
- { name: TC_PS_2_3.py, duration: 30 seconds }
9898
- { name: TC_RR_1_1.py, duration: 25 seconds }
9999
- { name: TC_SWTCH.py, duration: 1 minute }
100+
- { name: TC_TestAttrAvail.py, duration: 30 seconds }
100101
- { name: TC_TIMESYNC_2_10.py, duration: 20 seconds }
101102
- { name: TC_TIMESYNC_2_11.py, duration: 30 seconds }
102103
- { name: TC_TIMESYNC_2_12.py, duration: 20 seconds }

0 commit comments

Comments
 (0)
Please sign in to comment.