TC-CGEN-2.2: Automate

[cecille]

• Test plan can have the PICS markings removed since all the attributes are mandatory
• PIXIT should be removed - change to some reasonable value rather than being settable by the tester
• Test is long-running because we do actually want to test the arm failsafe times out appropriately so this test might need some gating in the CI

Current test is entirely manual and doesn't appear to be being run in the 1.4 cert package. It is currently handled in the verification steps doc: https://groups.csa-iot.org/wg/matter-csg/document/folder/3979

test plan: https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/cluster/General_Commissioning.adoc

[andy31415]

Assigned to myself as a "worked on" placeholder pending getting seats for new developers in the project

[juandediosg]

PR created as draft. Validation and cleanup still in progress
project-chip/connectedhomeip#37262

[juandediosg]

PR is completed and currently in "Open" status, awaiting code review and approvals.
I’m also working on addressing the PR validations

[juandediosg]

Fixed: PR validations due "### Testing section not in PR"

[juandediosg]

Fixed: PR validations Restyle and lint code base

[juandediosg]

I have reviewed the code and the feedback from the code review. I will take action on the comments provided, but for the feedback related to the failsafe expiration changes and step 33, I will implement the following strategy:

1. Failsafe expiration:
I originally had the expiration set to 0 to allow the failsafe to re-arm quickly, but I will change it to 1 second for CI, using the is_pics_sdk_ci_only() function to ensure that in CI, the expiration is set to 1 second, and for certification tests, it will remain at a longer value (like 900 seconds, or whatever is defined for cert).

2. Step 32 (ArmFailSafe command):
In step 32, where we are testing the ArmFailSafe command, I understand the Test Plan specifies that the ExpiryLengthSeconds field should be set to maxFailsafe = 900 seconds. I will follow this for the certification tests to ensure the system correctly handles the maximum expiration time. For CI, I will use a smaller expiration time (like 5 seconds or 10 seconds) to keep the tests fast.

3. Step 33 (validating that the failsafe doesn’t expire too early):
For step 33, where we are checking that the failsafe doesn’t expire before the expected time, I’ll use 5 seconds in CI. This should be enough to validate that the timer is being updated correctly while still being fast enough for CI.

[juandediosg]

PR Final version of the Automated Python script has been pushed project-chip/connectedhomeip#37262.

Below are the highlighted updates, changes, and refactoring:

Note: The script has passed CI job checkpoints, including Lint code base, Restyled, etc.

• Refactored and cleaned up.
• Updated code and logger comments.
• For the CI flow, the logic was implemented with the is_pics_sdk_ci_only to correctly handle test depending on environment.
• Adjusted the sequence between maxFailSafe and PIXIT.CGEN.FailsafeExpiryLengthSeconds.
• In the CERT tests, failsafe_expiration_seconds = 20 was set for the waits, while for CI, the expire_failsafe_timer function was used with a small re-arm of the timer (1 second) to correctly verify the timer expiration.
• Reused previously generated certificates.
• Steps 3, 4, and 5 were wrapped into one function with boolean control, while steps 8 and 9 were grouped into another function with boolean control.
• For step 22, the code remained unchanged, and it was suggested not to modify the SetSkipCommissioningComplete(True) logic, as it resulted in an error at step 29.
• Implemented FindOrEstablishPASESession function from ChipDeviceCtrl.py to avoid duplication.
• Implemented Mobly asserts to improve error handling and overall test framework consistency.
• Updated logic on sequence of steps 29-36 to align with the new optimizations.

[juandediosg]

PR Test Plan updated: https://github.com/CHIP-Specifications/chip-test-plans/pull/4956

[juandediosg]

PR has been updated based on the code review. The following commits have been pushed:

• Fixed: Updated try/except with specific exceptions and fail the test in case of error.
• Fixed: Renamed function for clarity based on feedback.
• Fixed: Removed UTC time function, logging elapsed time in ms in steps 41 and 43.

In some comments, I opted not to implement certain changes, and I have detailed my reasoning in the specific comments provided.

Working on making adjustments configurable via a user parameter, so testers can modify it if necessary.

[juandediosg]

PR has been updated based on the code review. The following commits have been pushed:

• Fixed: Hooked failsafe_expiration_seconds to user parameter PIXIT.CGEN.FailsafeExpiryLengthSeconds for CI or Cert flow flexibility.
• Fixed: Lint Code base and Restyled.

[juandediosg]

PR has been updated based on the code review:

• Added constant variable FAILSAFE_EXPIRATION_SECONDS for cert tests to handle timeout more effectively

[juandediosg]

Python test automation PR has been updated:

• failsafe_expiration_seconds setting is adjustable for Cert and CI, with a default value set of 20 seconds.
• run_type variable has been fixed not eliminate redundancy.

Test Plan PR has been updated:

• Step 2a updated
• Step 7 updated