Skip to content

Commit f8d71b9

Browse files
mkasenbergmkasenberg
committedJan 12, 2024
docs: Add tutorial on how to add new test cases
A short guide that can save new contributors some time and retain knowledge for the future.
1 parent b464f7f commit f8d71b9

16 files changed

+451
-0
lines changed
 

‎README.md

+6
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
* [Running in Client/Server Mode](#running-in-clientserver-mode)
99
* [Running AutoPTSClientBot](#running-autoptsclientbot)
1010
* [Zephyr with AutoPTS step-by-step setup tutorial](#zephyr-with-autopts-step-by-step-setup-tutorial)
11+
* [Tutorials](#tutorials)
1112
* [More examples of run and tips](#more-examples-of-run-and-tips)
1213
* [Slack Channel](#slack-channel)
1314

@@ -307,6 +308,11 @@ Check out the guide how to set up AutoPTS for Zephyr + nRF52 under:
307308

308309
https://docs.zephyrproject.org/latest/connectivity/bluetooth/autopts/autopts-win10.html
309310

311+
# Tutorials
312+
313+
- [How to create PTS workspace](doc/tutorials/create_workspace.md)
314+
- [How to add support for a PTS profile](doc/tutorials/add_test_case.md)
315+
310316
# More examples of run and tips
311317

312318
**Run many instances of autoptsserver on one Windows**

‎doc/tutorials/add_test_case.md

+353
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,353 @@
1+
# Add support for a PTS profile
2+
3+
- [Plan the BTP interface](#plan-the-btp-interface)
4+
- [Get familiar with tools](#get-familiar-with-tools)
5+
- [Add a new profile](#add-a-new-profile)
6+
- [Add a new test case](#add-a-new-test-case)
7+
- [Synchronize multiple Lower Testers](#synchronize-multiple-lower-testers)
8+
- [Keep global variables in Stack](#keep-global-variables-in-stack)
9+
10+
## Plan the BTP interface
11+
12+
PTS running in automation mode sends MMI requests to AutoPTS which handles them
13+
by using [BTP protocol](../../doc/overview.txt). Each PTS profile has its own
14+
set of MMI messages that can be found in ATS files. The links to ATS files can
15+
be accessed in PTS GUI -> Start Page -> Reference Documents.
16+
17+
![](images/new_profile_ats_files.png)
18+
19+
Reviewing MMI messages of a profile allows to gain a bigger picture of how the BTP
20+
interface for that profile should be designed.
21+
22+
![](images/new_profile_ats_mmi.png)
23+
24+
Planning the BTP interface before starting the implementation process will:
25+
- avoid IUT specific implementation,
26+
- make the BTP interface more generic in terms of reusing across multiple
27+
profiles, reducing the amount of code on the IUT tester side,
28+
- reduce the number of frequent BTP changes, so multiple projects could enjoy
29+
the AutoPTS without disturbing their Bluetooth product development process.
30+
31+
Adding support for all test cases of a profile at once should result with
32+
a complete BTP interface for the profile. If a partial BTP interface is
33+
being contributed, with a few BTP commands, then those commands should be
34+
well-thought-out so that they will not need to be updated in the future to
35+
complete the remaining test cases.
36+
37+
Changing the parameters of a BTP command/response/event that has been used
38+
for a while is difficult, because it might already have been adopted by many
39+
and the change may rise objections. Adding a new helper command to the completed
40+
BTP interfaces is more acceptable as it does not affect the existing IUT tester
41+
implementations.
42+
43+
## Get familiar with tools
44+
45+
There are few log sources that will be helpful:
46+
47+
- PTS and Bluetooth Protocol Viewer logs,
48+
- AutoPTS client and server logs,
49+
- Jlink RTT logs,
50+
- Btmon logs,
51+
- Ellisys logs.
52+
53+
### PTS and Bluetooth Protocol Viewer logs
54+
55+
The logs allow to peek what happens on the Host side of the PTS dongle during
56+
a test case.
57+
58+
AutoPTS saves the logs in the directory of the PTS workspace (the .pqw6 file).
59+
Inside a test case sub folder there should be 4 files:
60+
- `.xml` - PTS debug logs. Useful to find out what was the reason of a test case
61+
failure when a problem was related to some failing checks in case some test case
62+
conditions were not met.
63+
- `.cfa`, `.frm` and `.fsc` - can be open only with BPV. They allow you to view
64+
parsed packets exchanged between the host and the controller of the PTS dongle.
65+
66+
### AutoPTS client and server logs
67+
68+
The logs allow to peek what happens on the AutoPTS client or server side:
69+
- what MMI are requested by PTS,
70+
- what response to MMI is sent back to PTS,
71+
- all data and order of the BTP communication,
72+
- AutoPTS framework python exceptions, including missing MMI/WID handlers,
73+
WID implementation bugs, bot features exceptions.
74+
75+
AutoPTS creates a log file named after the autopts client launcher that is used
76+
e.g. `autoptsclient_zephyr_65000.log`, `autoptsclient_bot_65000_65002.log`,
77+
`autoptsserver_65000_65002.log`... These log files contains AutoPTS client logs
78+
from all test cases in the run. For convenience, the client logs are saved one
79+
file per test case too in the `logs/` folder.
80+
81+
### Jlink RTT logs
82+
83+
If an IUT board supports debug logs sent via Jlink RTT logs, AutoPTS can read
84+
them during a test case and save them in the `logs/` folder.
85+
86+
### Btmon logs
87+
88+
If an IUT supports btmon logs, AutoPTS can read them during a test case and
89+
save them in the `logs/` folder in a btsnoop or plain text format.
90+
91+
### Ellisys logs.
92+
93+
If you are lucky user of [Ellisys sniffer](https://www.ellisys.com/better_analysis/bta_previous.php)
94+
you will be able to resolve the tricky cases, where a controller is at fault.
95+
96+
97+
## Add a new profile
98+
99+
To add a new profile to the AutoPTS, we have to start with some boilerplate code.
100+
This can be done with the `./tools/generate_profile.py` generator. An example usage
101+
that creates profile named PROFILE for the `zephyr` project:
102+
```shell
103+
$ python ./tools/generate_profile.py
104+
Enter project name: zephyr
105+
Enter profile name: profile
106+
Enter new BTP service ID (max 255): 26
107+
```
108+
109+
The generator creates all required new files and appends some additional imports
110+
and entries to the existing ones to tie everything together. Let's do a quick
111+
overview of what the generated files and entries are for:
112+
113+
### `doc/btp_profile.txt`
114+
115+
This file is for a description of the profile BTP interface.
116+
117+
### `autopts/pybtp/btp/profile.py`
118+
119+
This file is for implementations of the BTP interface, i.e. functions
120+
for packing the BTP commands, and parsers of the BTP events and responses.
121+
122+
### `autopts/pybtp/defs.py`
123+
124+
New BTP service interface needs an ID, so create a new entry `BTP_SERVICE_ID_PROFILE = 26`
125+
inside the `autopts/pybtp/defs.py`, where the value should be the next available ID value.
126+
Keep all BTP interface opcodes inside the file, i.e. all the BTP commands and events opcodes.
127+
128+
### `autopts/wid/profile.py`
129+
130+
This file is for MMI/WID handlers of the profile, generic for all projects. The handlers
131+
make use of the BTP commands, responses and events. If there is a need for a project specific
132+
handler, then it should be added into the `autopts/ptsprojects/zephyr/profile_wid.py` inside
133+
the project.
134+
135+
### `autopts/ptsprojects/stack/layers/profile.py`
136+
137+
The `class Stack` is a good place for global variables that have to be shared
138+
between MMI/WID handlers, because the Stack instance is reinitialized before
139+
start of each test case. So create a class like `class Profile` and make it a part
140+
of the Stack.
141+
142+
### `autopts/ptsprojects/zephyr/profile.py`
143+
144+
This file has to contain two functions:
145+
`def set_pixits(ptses)` and `def test_cases(ptses)`.
146+
147+
The `def set_pixits(ptses)` configures how AutoPTS should overwrite the default
148+
PIXITs value of a PTS workspace. The function could be as short as:
149+
150+
```python
151+
def set_pixits(ptses):
152+
pts = ptses[0]
153+
154+
pts.set_pixit("BAP", "TSPX_time_guard", "180000")
155+
pts.set_pixit("BAP", "TSPX_use_implicit_send", "TRUE")
156+
157+
if len(ptses) < 2:
158+
return
159+
160+
pts2 = ptses[1]
161+
pts2.set_pixit("BAP", "TSPX_time_guard", "180000")
162+
pts2.set_pixit("BAP", "TSPX_use_implicit_send", "TRUE")
163+
```
164+
165+
The `def test_cases(ptses)` is used to customize the IUT configuration and behavior
166+
before and after running test cases of the profile. A short version of the function
167+
could look like this:
168+
169+
```python
170+
def test_cases(ptses):
171+
"""Returns a list of PROFILE test cases
172+
ptses -- list of PyPTS instances"""
173+
174+
pts = ptses[0]
175+
stack = get_stack()
176+
177+
# Generic preconditions for all test case in the profile
178+
pre_conditions = [
179+
TestFunc(btp.core_reg_svc_profile),
180+
TestFunc(stack.profile_init)
181+
]
182+
183+
test_case_name_list = pts.get_test_case_list('PROFILE')
184+
tc_list = []
185+
186+
# Use the same preconditions and MMI/WID handler for all test cases of the profile
187+
for tc_name in test_case_name_list:
188+
instance = ZTestCase('PROFILE', tc_name, cmds=pre_conditions,
189+
generic_wid_hdl=profile_wid_hdl)
190+
191+
tc_list.append(instance)
192+
193+
return tc_list
194+
```
195+
196+
In almost every profile there are multiple test cases that need a custom preconditions,
197+
e.g. at the start of a test case PTS expects a specific advertisement
198+
in the air, but there is no MMI (WID) for it. Test cases with multiple Lower Testers
199+
(PTS instances) usually need to be synchronized with Synch Points to avoid collision
200+
of BTP commands, so those can be customized here too.
201+
202+
The `profile_wid_hdl` parameter of the TestCase class is a generic handler that
203+
will find a right handler for an MMI (WID) request received from Lower Tester
204+
(PTS instance). In most test cases a project will use handlers only from
205+
`autopts/wid`, but sometimes a custom, project-specific WID is required.
206+
It is convenient to create a separate file in your project for this, e.g:
207+
`autopts/ptsprojects/zephyr/profile_wid.py`:
208+
209+
```python
210+
# profile_wid.py file
211+
import logging
212+
213+
from autopts.wid import generic_wid_hdl
214+
from autopts.pybtp.types import WIDParams
215+
216+
log = logging.debug
217+
218+
219+
def profile_wid_hdl(wid, description, test_case_name):
220+
log(f'{profile_wid_hdl.__name__}, {wid}, {description}, {test_case_name}')
221+
222+
# Try to find the WID handler in this file, if not found then proceed to
223+
# the generic autopts/wid/profile.py.
224+
return generic_wid_hdl(wid, description, test_case_name,
225+
[__name__, 'autopts.wid.profile'])
226+
227+
228+
def hdl_wid_104(_: WIDParams):
229+
"""
230+
Confirm if the Lower Tester received ISO stream.
231+
"""
232+
# Should be handled by the Lower Tester, errata in progress.
233+
234+
return True
235+
```
236+
237+
## Add a new test case
238+
239+
Assuming you have completed the setup of the profile, you are ready to
240+
start implementing BTP interface and missing MMI/WID handlers for a test case.
241+
242+
Depending on the role the IUT is given in the test case, the IUT might have to
243+
start scanning or advertising. If an IUT should act as a client, the PTS can
244+
request a MMI similar to 20100: "Please initiate a GATT connection to the PTS."
245+
It means that the IUT should start scanning and then proceed with establishing
246+
a Link Layer connection with the PTS dongle. This behavior is covered by BTP command
247+
of GAP BTP Service:
248+
`Opcode 0x0e - Connect command/responsecan` defined in `doc/btp_gap.txt` and implemented
249+
in `autopts/pybtp/btp/gap.py`. So the handler of MMI 20100 is implemented as fallow:
250+
251+
```python
252+
def hdl_wid_20100(_: WIDParams):
253+
btp.gap_conn()
254+
return True
255+
```
256+
257+
For test cases with the IUT in a server role, it depends on the test case if the
258+
PTS will ask you with an MMI to start advertising, but if it does not, then you
259+
have to start the action on your own.
260+
261+
This can be scripted in `def test_cases(ptses)`. If this action is used
262+
in just a few test cases, you can hardcord them in `custom_test_cases` list,
263+
but if it applies to most of the profile test cases, then just use `pre_conditions`
264+
to skip listing a hundred of test case names.
265+
266+
```python
267+
def test_cases(ptses):
268+
"""Returns a list of PROFILE test cases
269+
ptses -- list of PyPTS instances"""
270+
271+
pts = ptses[0]
272+
stack = get_stack()
273+
274+
# Generic preconditions for all test case in the profile
275+
pre_conditions = [
276+
TestFunc(btp.core_reg_svc_profile),
277+
TestFunc(stack.profile_init)
278+
]
279+
280+
# Custom preconditions
281+
pre_conditions_server = pre_conditions + [
282+
TestFunc(btp.gap_set_extended_advertising_on),
283+
TestFunc(lambda: btp.gap_adv_ind_on(ad=ad)),
284+
]
285+
286+
test_case_name_list = pts.get_test_case_list('PROFILE')
287+
tc_list = []
288+
289+
ad = {
290+
AdType.name_full: iut_device_name[::1].hex(),
291+
AdType.flags: format(AdFlags.br_edr_not_supp |
292+
AdFlags.le_gen_discov_mode, '02x'),
293+
}
294+
295+
# Use custom preconditions
296+
custom_test_cases = [
297+
ZTestCase("PROFILE", "PROFILE/SR/ADV/BV-01-C",
298+
cmds=pre_conditions_server,
299+
generic_wid_hdl=profile_wid_hdl)
300+
]
301+
302+
for tc_name in test_case_name_list:
303+
instance = ZTestCase('PROFILE', tc_name, cmds=pre_conditions,
304+
generic_wid_hdl=profile_wid_hdl)
305+
306+
# Use custom preconditions
307+
for custom_tc in custom_test_cases:
308+
if tc_name == custom_tc.name:
309+
instance = custom_tc
310+
break
311+
312+
tc_list.append(instance)
313+
314+
return tc_list
315+
```
316+
317+
## Synchronize multiple Lower Testers
318+
319+
In test cases with multiple Lower Testers it is common to synchronize WID execution
320+
to avoid BTP commands collision or interrupting a busy IUT. This can be handled with
321+
Synch Points mechanism.
322+
323+
```python
324+
# In test_cases(ptses) function:
325+
326+
# Use custom preconditions
327+
custom_test_cases = [
328+
ZTestCase("BAP", "BAP/BA/BASS/BV-04-C", cmds=pre_conditions +
329+
[ # Barrier no. 1:
330+
# LT1 will not execute the WID 20100 until LT2 receives
331+
# the WID 100 request. LT2 will not execute WID 100
332+
# until LT1 completes its WID 20100.
333+
TestFunc(get_stack().synch.add_synch_element,
334+
[SynchPoint("BAP/BA/BASS/BV-04-C", 20100),
335+
SynchPoint("BAP/BA/BASS/BV-04-C_LT2", 100)]),
336+
337+
# Barrier no. 2:
338+
# LT2 will not execute the WID 384 until LT1 receives
339+
# the WID 345 request. LT1 will not execute WID 345
340+
# until LT2 completes its WID 384.
341+
TestFunc(get_stack().synch.add_synch_element,
342+
[SynchPoint("BAP/BA/BASS/BV-04-C_LT2", 384),
343+
SynchPoint("BAP/BA/BASS/BV-04-C", 345)])],
344+
generic_wid_hdl=bap_wid_hdl,
345+
lt2="BAP/BA/BASS/BV-04-C_LT2"),
346+
]
347+
348+
test_cases_lt2 = [
349+
ZTestCaseSlave("BAP", "BAP/BA/BASS/BV-04-C_LT2",
350+
cmds=pre_conditions_lt2,
351+
generic_wid_hdl=bap_wid_hdl),
352+
]
353+
```

0 commit comments

Comments
 (0)
Please sign in to comment.