Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduced a python script (execute_python_tests.py) that automates the execution of python test scripts within the python_testing directory #34377

Merged
merged 36 commits into from
Aug 7, 2024
Merged

Introduced a python script (execute_python_tests.py) that automates the execution of python test scripts within the python_testing directory #34377

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
a7991b6
Adding the python script to glob the python_testing folder on executi…
vatsalghelani-csa Jul 16, 2024
dfca7f1
Removed the baed command temp file while inhertiing it from the tests…
vatsalghelani-csa Jul 16, 2024
ce7b796
Fixed the extra script execution part
vatsalghelani-csa Jul 16, 2024
ced8b67
Fixes to the script
vatsalghelani-csa Jul 16, 2024
30e16f5
Fixed the python script with correct command
vatsalghelani-csa Jul 16, 2024
8a7a768
Fixed the doyenv module not found error
vatsalghelani-csa Jul 16, 2024
a046129
Restyled by autopep8
restyled-commits Jul 17, 2024
db6c35b
Restyled by isort
restyled-commits Jul 17, 2024
a13f027
Fixed the script as per comments added to the code review
vatsalghelani-csa Jul 18, 2024
a04f1fd
Restyled by autopep8
restyled-commits Jul 18, 2024
2661bb0
Restyled by isort
restyled-commits Jul 18, 2024
5e73505
Fixed the comment review
vatsalghelani-csa Jul 18, 2024
7e7088d
Restyled by autopep8
restyled-commits Jul 18, 2024
7352518
Added the argument design patterns for TC_BOOLCFG* and TC_VALCC* scripts
vatsalghelani-csa Jul 22, 2024
0764957
Added --hex-arg as argument needed for TC_BOOLCFG tests
vatsalghelani-csa Jul 23, 2024
21339eb
Merge conflict resolution
vatsalghelani-csa Jul 23, 2024
d26f400
Revert "Added --hex-arg as argument needed for TC_BOOLCFG tests"
vatsalghelani-csa Jul 23, 2024
c2c4593
Undo submodule change
vatsalghelani-csa Jul 23, 2024
725d72d
Fixed the --hex-arg argument
vatsalghelani-csa Jul 23, 2024
0ba67a7
Fixed arguments according to test requirement
vatsalghelani-csa Jul 24, 2024
10a15ea
Excluded new script added since it is run via python3 directly
vatsalghelani-csa Jul 24, 2024
801e6dd
Fixed .py of the exclusion script
vatsalghelani-csa Jul 24, 2024
5619b8f
Fixed conflict issues
vatsalghelani-csa Jul 26, 2024
5365640
Added fixes to the script arguments and new scripts run
vatsalghelani-csa Jul 26, 2024
a322249
Fixing merge conflicts on tests.yaml
vatsalghelani-csa Jul 26, 2024
8ee1134
Removed support test to run
vatsalghelani-csa Jul 26, 2024
69564bf
Resolving conflicts
vatsalghelani-csa Jul 26, 2024
a4acdab
REPL Tests - Linux run
vatsalghelani-csa Jul 29, 2024
75e08c5
Excluded recently added DEMTEst support python from all script execution
vatsalghelani-csa Jul 29, 2024
cca8752
Fix merge conflicts
vatsalghelani-csa Jul 30, 2024
670a842
Excluded tests yet not ready for CI
vatsalghelani-csa Jul 30, 2024
c47fdb9
Excluded the support test for TC_EWATERHTR
vatsalghelani-csa Jul 30, 2024
bfcc78b
Excluded not ready tests
vatsalghelani-csa Aug 6, 2024
9b179b8
Merge conflcits and merge master
vatsalghelani-csa Aug 6, 2024
5d5e1eb
Fix file
vatsalghelani-csa Aug 6, 2024
d16704e
Fix file comma formatting
vatsalghelani-csa Aug 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 1 addition & 80 deletions .github/workflows/tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -496,86 +496,7 @@ jobs:
run: |
mkdir -p out/trace_data
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/controller/python/test/test_scripts/mobile-device-test.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_ACE_1_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_ACE_1_3.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_ACE_1_4.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_ACE_1_5.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_AccessChecker.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_CGEN_2_4.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_DA_1_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_DA_1_5.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_DA_1_7.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_DGGEN_2_4.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_DRLK_2_12.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_DRLK_2_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_DRLK_2_3.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_DeviceBasicComposition.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_DeviceConformance.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_EEM_2_1.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_EEM_2_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_EEM_2_3.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_EEM_2_4.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_EEM_2_5.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_EEVSE_2_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_EEVSE_2_4.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_EEVSE_2_5.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_EPM_2_1.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_EPM_2_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_FAN_3_1.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_FAN_3_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_FAN_3_3.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_FAN_3_4.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_FAN_3_5.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_ICDM_2_1.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_ICDM_3_1.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_ICDManagementCluster.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_IDM_1_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_IDM_1_4.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_IDM_4_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_PWRTL_2_1.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_RR_1_1.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_SC_3_6.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_TIMESYNC_2_1.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_TIMESYNC_2_10.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_TIMESYNC_2_11.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_TIMESYNC_2_12.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_TIMESYNC_2_13.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_TIMESYNC_2_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_TIMESYNC_2_4.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_TIMESYNC_2_5.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_TIMESYNC_2_6.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_TIMESYNC_2_7.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_TIMESYNC_2_8.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_TIMESYNC_2_9.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_TIMESYNC_3_1.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_TestEventTrigger.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TestBatchInvoke.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TestGroupTableReports.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_OPCREDS_3_1.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_OPCREDS_3_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_OPSTATE_2_1.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_OPSTATE_2_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_OPSTATE_2_3.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_OPSTATE_2_4.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_OPSTATE_2_5.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_OVENOPSTATE_2_1.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_OVENOPSTATE_2_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_OVENOPSTATE_2_3.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_OVENOPSTATE_2_4.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_OVENOPSTATE_2_5.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_MWOCTRL_2_1.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_MWOCTRL_2_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_MWOCTRL_2_4.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_MWOM_1_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_RVCRUNM_1_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_RVCRUNM_2_1.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_RVCRUNM_2_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_RVCCLEANM_1_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_RVCCLEANM_2_1.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_RVCCLEANM_2_2.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_RVCOPSTATE_2_1.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_RVCOPSTATE_2_3.py'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_RVCOPSTATE_2_4.py'
scripts/run_in_python_env.sh out/venv 'python3 ./src/python_testing/execute_python_tests.py --env-file /tmp/test_env.yaml --search-directory src/python_testing'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --script "src/python_testing/TestConformanceSupport.py" --script-args "--trace-to json:out/trace_data/test-{SCRIPT_BASE_NAME}.json --trace-to perfetto:out/trace_data/test-{SCRIPT_BASE_NAME}.perfetto"'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --script "src/python_testing/TestMatterTestingSupport.py" --script-args "--trace-to json:out/trace_data/test-{SCRIPT_BASE_NAME}.json --trace-to perfetto:out/trace_data/test-{SCRIPT_BASE_NAME}.perfetto"'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --script "src/python_testing/TestSpecParsingSupport.py" --script-args "--trace-to json:out/trace_data/test-{SCRIPT_BASE_NAME}.json --trace-to perfetto:out/trace_data/test-{SCRIPT_BASE_NAME}.perfetto"'
Expand Down
116 changes: 116 additions & 0 deletions src/python_testing/execute_python_tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
#
# Copyright (c) 2024 Project CHIP Authors
# All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

import argparse
import glob
import os
import subprocess

# Function to load environment variables from a YAML file


def load_env_from_yaml(file_path):
"""
Load environment variables from the specified YAML file.

The YAML file contains key-value pairs that define --app environment variables
required for the test scripts to run. These variables configurations needed during the test execution.

This function reads the YAML file and sets the environment variables
in the current process's environment using os.environ.

Args:
file_path (str): The path to the YAML file containing the environment variables.
"""
with open(file_path, 'r') as file:
for line in file:
if line.strip(): # Skip empty lines
key, value = line.strip().split(': ', 1)
os.environ[key] = value


def main(search_directory, env_file):
# Determine the root directory of the CHIP project
chip_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))

# Load environment variables from the specified file
load_env_from_yaml(env_file)

# Define the base command to run tests
base_command = os.path.join(chip_root, "scripts/tests/run_python_test.py")

# Define the files and patterns to exclude
excluded_patterns = {
"MinimalRepresentation.py",
"TC_ACL_2_2.py",
"TC_CNET_4_4.py",
"TC_DGGEN_3_2.py",
"TC_EEVSE_Utils.py",
"TC_EnergyReporting_Utils.py",
"TC_OpstateCommon.py",
"TC_TMP_2_1.py",
"TC_pics_checker.py",
"TestCommissioningTimeSync.py",
"TestConformanceSupport.py",
"TestIdChecks.py",
"TestMatterTestingSupport.py",
"TestSpecParsingSupport.py",
"TestTimeSyncTrustedTimeSource.py",
"basic_composition_support.py",
"conformance_support.py",
"drlk_2_x_common.py",
"execute_python_tests.py",
"global_attribute_ids.py",
"hello_external_runner.py",
"hello_test.py",
"matter_testing_support.py",
"pics_support.py",
"spec_parsing_support.py",
"taglist_and_topology_test_support.py",
"test_plan_support.py",
"test_plan_table_generator.py"
}

"""
Explanation for excluded files:
The above list of files are excluded from the tests as they are either not app-specific tests
or are run through a different block of tests mentioned in tests.yaml.
This is to ensure that only relevant test scripts are executed, avoiding redundancy and ensuring
the correctness of the test suite.
"""

# Get all .py files in the directory
all_python_files = glob.glob(os.path.join(search_directory, "*.py"))

# Filter out the files matching the excluded patterns
python_files = [file for file in all_python_files if os.path.basename(file) not in excluded_patterns]

# Run each script with the base command
for script in python_files:
full_command = f"{base_command} --load-from-env {env_file} --script {script}"
print(f"Running command: {full_command}")
subprocess.run(full_command, shell=True, check=True)


if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Run Python test scripts.")
parser.add_argument("--search-directory", type=str, default="src/python_testing",
help="Directory to search for Python scripts.")
parser.add_argument("--env-file", type=str, default="/tmp/test_env.yaml", help="Path to the environment variables file.")

args = parser.parse_args()
main(args.search_directory, args.env_file)
Loading