Skip to content

Commit f948532

Browse files
grynspangrynspan
committed
[ST-NNNN] Test cancellation
1 parent 227f3c2 commit f948532

File tree

1 file changed

+365
-0
lines changed

1 file changed

+365
-0
lines changed

‎proposals/testing/NNNN-test-cancellation.md

Lines changed: 365 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,365 @@
1+
# Test cancellation
2+
3+
* Proposal: [ST-NNNN](NNNN-test-cancellation.md)
4+
* Authors: [Jonathan Grynspan](https://github.com/grynspan)
5+
* Review Manager: TBD
6+
* Status: **Awaiting review**
7+
* Bug: [swiftlang/swift-testing#120](https://github.com/swiftlang/swift-testing/issues/120)
8+
* Implementation: [swiftlang/swift-testing#1284](https://github.com/swiftlang/swift-testing/pull/1284)
9+
* Review: ([pitch](https://forums.swift.org/...))
10+
11+
## Introduction
12+
13+
Swift Testing provides the ability to conditionally skip a test before it runs
14+
using the [`.enabled(if:)`](https://developer.apple.com/documentation/testing/trait/enabled(if:_:sourcelocation:)),
15+
[`.disabled(if:)`](https://developer.apple.com/documentation/testing/trait/disabled(if:_:sourcelocation:)),
16+
etc. family of traits:
17+
18+
```swift
19+
@Test(.enabled(if: Tyrannosaurus.isTheLizardKing))
20+
func `Tyrannosaurus is scary`() {
21+
let dino = Tyrannosaurus()
22+
#expect(dino.isScary)
23+
// ...
24+
}
25+
```
26+
27+
This proposal extends that feature to allow cancelling a test after it has
28+
started but before it has ended.
29+
30+
## Motivation
31+
32+
We have received feedback from a number of developers indicating that their
33+
tests have constraints that can only be checked after a test has started, and
34+
they would like the ability to end a test early and see that state change
35+
reflected in their development tools.
36+
37+
To date, we have not provided an API for ending a test's execution early because
38+
we want to encourage developers to use the [`.enabled(if:)`](https://developer.apple.com/documentation/testing/trait/enabled(if:_:sourcelocation:))
39+
_et al._ trait. This trait can be evaluated early and lets Swift Testing plan a
40+
test run more efficiently. However, we recognize that these traits aren't
41+
sufficient. Some test constraints are dependent on data that isn't available
42+
until the test starts, while others only apply to specific test cases in a
43+
parameterized test function.
44+
45+
## Proposed solution
46+
47+
A static `cancel()` function is added to the [`Test`](https://developer.apple.com/documentation/testing/test)
48+
and [`Test.Case`](https://developer.apple.com/documentation/testing/test/case)
49+
types. When a test author calls these functions from within the body of a test
50+
(or from within the implementation of a trait, e.g. from [`prepare(for:)`](https://developer.apple.com/documentation/testing/trait/prepare(for:))),
51+
Swift Testing cancels the currently-running test or test case, respectively.
52+
53+
### Relationship between tasks and tests
54+
55+
Each test runs in its own task during a test run, and each test case in a test
56+
also runs in its own task. Cancelling the current task from within the body of a
57+
test will, therefore, cancel the current test case, but not the current test:
58+
59+
```swift
60+
@Test(arguments: Species.all(in: .dinosauria))
61+
func `Are all dinosaurs extinct?`(_ species: Species) {
62+
if species.in(.aves) {
63+
// Birds aren't extinct (I hope)
64+
withUnsafeCurrentTask { $0?.cancel() }
65+
return
66+
}
67+
// ...
68+
}
69+
```
70+
71+
Using [`withUnsafeCurrentTask(body:)`](https://developer.apple.com/documentation/swift/withunsafecurrenttask(body:)-6gvhl)
72+
here is not ideal. It's not clear that the intent is to cancel the test case,
73+
and [`UnsafeCurrentTask`](https://developer.apple.com/documentation/swift/unsafecurrenttask)
74+
is, unsurprisingly, an unsafe interface.
75+
76+
> [!NOTE]
77+
> The version of Swift Testing included with Swift 6.2 does not correctly handle
78+
> task cancellation under all conditions. See [swiftlang/swift-testing#1289](https://github.com/swiftlang/swift-testing/issues/1289).
79+
80+
## Detailed design
81+
82+
New static members are added to [`Test`](https://developer.apple.com/documentation/testing/test)
83+
and [`Test.Case`](https://developer.apple.com/documentation/testing/test/case):
84+
85+
```swift
86+
extension Test {
87+
/// Cancel the current test.
88+
///
89+
/// - Parameters:
90+
/// - comment: A comment describing why you are cancelling the test.
91+
/// - sourceLocation: The source location to which the testing library will
92+
/// attribute the cancellation.
93+
///
94+
/// - Throws: An error indicating that the current test case has been
95+
/// cancelled.
96+
///
97+
/// The testing library runs each test in its own task. When you call this
98+
/// function, the testing library cancels the task associated with the current
99+
/// test:
100+
///
101+
/// ```swift
102+
/// @Test func `Food truck is well-stocked`() throws {
103+
/// guard businessHours.contains(.now) else {
104+
/// try Test.cancel("We're off the clock.")
105+
/// }
106+
/// // ...
107+
/// }
108+
/// ```
109+
///
110+
/// If the current test is parameterized, all of its pending and running test
111+
/// cases are cancelled. If the current test is a suite, all of its pending
112+
/// and running tests are cancelled. If you have already cancelled the current
113+
/// test or if it has already finished running, this function throws an error
114+
/// but does not attempt to cancel the test a second time.
115+
///
116+
/// - Important: If the current task is not associated with a test (for
117+
/// example, because it was created with [`Task.detached(name:priority:operation:)`](https://developer.apple.com/documentation/swift/task/detached(name:priority:operation:)-795w1))
118+
/// this function records an issue and cancels the current task.
119+
///
120+
/// To cancel the current test case but leave other test cases of the current
121+
/// test alone, call ``Test/Case/cancel(_:sourceLocation:)`` instead.
122+
public static func cancel(_ comment: Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation) throws -> Never
123+
}
124+
125+
extension Test.Case {
126+
/// Cancel the current test case.
127+
///
128+
/// - Parameters:
129+
/// - comment: A comment describing why you are cancelling the test case.
130+
/// - sourceLocation: The source location to which the testing library will
131+
/// attribute the cancellation.
132+
///
133+
/// - Throws: An error indicating that the current test case has been
134+
/// cancelled.
135+
///
136+
/// The testing library runs each test case of a test in its own task. When
137+
/// you call this function, the testing library cancels the task associated
138+
/// with the current test case:
139+
///
140+
/// ```swift
141+
/// @Test(arguments: [Food.burger, .fries, .iceCream])
142+
/// func `Food truck is well-stocked`(_ food: Food) throws {
143+
/// if food == .iceCream && Season.current == .winter {
144+
/// try Test.Case.cancel("It's too cold for ice cream.")
145+
/// }
146+
/// // ...
147+
/// }
148+
/// ```
149+
///
150+
/// If the current test is parameterized, the test's other test cases continue
151+
/// running. If the current test case has already been cancelled, this
152+
/// function throws an error but does not attempt to cancel the test case a
153+
/// second time.
154+
///
155+
/// - Important: If the current task is not associated with a test case (for
156+
/// example, because it was created with [`Task.detached(name:priority:operation:)`](https://developer.apple.com/documentation/swift/task/detached(name:priority:operation:)-795w1))
157+
/// this function records an issue and cancels the current task.
158+
///
159+
/// To cancel all test cases in the current test, call
160+
/// ``Test/cancel(_:sourceLocation:)`` instead.
161+
public static func cancel(_ comment: Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation) throws -> Never
162+
}
163+
```
164+
165+
These functions behave similarly, and are distinguished by the level of the test
166+
to which they apply:
167+
168+
- `Test.cancel()` cancels the current test.
169+
- If the current test is parameterized, it implicitly cancels all running and
170+
pending test cases of said test.
171+
- If the current test is a suite (only applicable during trait evaluation), it
172+
recursively cancels all test suites and test functions within said suite.
173+
- `Test.Case.cancel()` cancels the current test case.
174+
- If the current test is parameterized, other test cases are unaffected.
175+
- If the current test is _not_ parameterized, `Test.Case.cancel()` behaves the
176+
same as `Test.cancel()`.
177+
178+
Cancelling a test or test case implicitly cancels its associated task (and any
179+
child tasks thereof) as if [`Task.cancel()`](https://developer.apple.com/documentation/swift/task/cancel())
180+
were called on that task.
181+
182+
### Throwing semantics
183+
184+
Unlike [`Task.cancel()`](https://developer.apple.com/documentation/swift/task/cancel()),
185+
these functions always throw an error instead of returning. This simplifies
186+
control flow when a test is cancelled; instead of having to write:
187+
188+
```swift
189+
if condition {
190+
theTask.cancel()
191+
return
192+
}
193+
```
194+
195+
A test author need only write:
196+
197+
```swift
198+
if condition {
199+
try Test.cancel()
200+
}
201+
```
202+
203+
The errors these functions throw are of a type internal to Swift Testing that is
204+
semantically similar to [`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror)
205+
but carries additional information (namely the `comment` and `sourceLocation`
206+
arguments to `cancel(_:sourceLocation:)`) that Swift Testing can present to the
207+
user. When Swift Testing catches an error of this type[^cancellationErrorToo],
208+
it does not record an issue for the current test or test case.
209+
210+
[^cancellationErrorToo]: Swift Testing also catches errors of type
211+
[`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror)
212+
if the current task has been cancelled. If the current task has not been
213+
cancelled, errors of this type are still recorded as issues.
214+
215+
Suppressing these errors with `do`/`catch` or `try?` does not uncancel a test,
216+
test case, or task, but can be useful if you have additional local work you need
217+
to do before the test or test case ends.
218+
219+
### Interaction with recorded issues
220+
221+
If you cancel a test or test case that has previously recorded an issue, that
222+
issue is not overridden or nullified. In particular, if the test or test case
223+
has already recorded an issue of severity **error** when you call
224+
`cancel(_:sourceLocation:)`, the test or test case will still fail.
225+
226+
### Example usage
227+
228+
To cancel the current test case and let other test cases run:
229+
230+
```swift
231+
@Test(arguments: Species.all(in: .dinosauria))
232+
func `Are all dinosaurs extinct?`(_ species: Species) throws {
233+
if species.in(.aves) {
234+
try Test.Case.cancel("\(species) is birds!")
235+
}
236+
// ...
237+
}
238+
```
239+
240+
Or, to cancel all remaining test cases in the current test:
241+
242+
```swift
243+
@Test(arguments: Species.all(in: .dinosauria))
244+
func `Are all dinosaurs extinct?`(_ species: Species) throws {
245+
if species.is(.godzilla) {
246+
try Test.cancel("Forget about unit tests! Run for your life!")
247+
}
248+
// ...
249+
}
250+
```
251+
252+
## Source compatibility
253+
254+
This change is additive only.
255+
256+
## Integration with supporting tools
257+
258+
The JSON event stream Swift Testing provides is updated to include two new event
259+
kinds:
260+
261+
```diff
262+
<event-kind> ::= "runStarted" | "testStarted" | "testCaseStarted" |
263+
"issueRecorded" | "testCaseEnded" | "testEnded" | "testSkipped" |
264+
- "runEnded" | "valueAttached"
265+
+ "runEnded" | "valueAttached" | "testCancelled" | "testCaseEnded"
266+
```
267+
268+
And new fields are added to event records to represent the comment and source
269+
location passed to `cancel(_:sourceLocation:)`:
270+
271+
```diff
272+
<event> ::= {
273+
"kind": <event-kind>,
274+
"instant": <instant>, ; when the event occurred
275+
["issue": <issue>,] ; the recorded issue (if "kind" is "issueRecorded")
276+
["attachment": <attachment>,] ; the attachment (if kind is "valueAttached")
277+
"messages": <array:message>,
278+
["testID": <test-id>,]
279+
+ ["comments": <array:string>,]
280+
+ ["sourceLocation": <source-location>,]
281+
}
282+
```
283+
284+
These new fields are populated for the new event kinds as well as other event
285+
kinds that can populate them.
286+
287+
These new event kinds and fields will be included in the next revision of the
288+
JSON schema (currently expected to be schema version `"6.3"`.)
289+
290+
## Future directions
291+
292+
- Adding a corresponding `Test.checkCancellation()` function and/or
293+
`Test.isCancelled` static property. These are beyond the scope of this
294+
proposal, primarily because [`Task.isCancelled`](https://developer.apple.com/documentation/swift/task/iscancelled-swift.type.property)
295+
and [`Task.checkCancellation()`](https://developer.apple.com/documentation/swift/task/checkcancellation())
296+
already work in a test.
297+
298+
## Alternatives considered
299+
300+
- Doing nothing. While we do want test authors to use [`.enabled(if:)`](https://developer.apple.com/documentation/testing/trait/enabled(if:_:sourcelocation:))
301+
_et al._ trait, we recognize it does not provide the full set of functionality
302+
that test authors need.
303+
304+
- Ignoring task cancellation or treating [`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror)
305+
as a normal error even when the current task has been cancelled. It is not
306+
possible for Swift Testing to outright ignore task cancellation, and a
307+
[`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror)
308+
instance thrown from [`Task.checkCancellation()`](https://developer.apple.com/documentation/swift/task/checkcancellation())
309+
is not really a test issue but rather a manifestation of control flow.
310+
311+
- Using the [`XCTSkip`](https://developer.apple.com/documentation/xctest/xctskip-swift.struct)
312+
type from XCTest. Interoperation with XCTest is an area of exploration for us,
313+
but core functionality of Swift Testing needs to be usable without also
314+
importing XCTest.
315+
316+
- Spelling the functions `static func cancel(_:sourceLocation:) -> some Error`
317+
and requiring it be called as `throw Test.cancel()`. This is closer to how
318+
the [`XCTSkip`](https://developer.apple.com/documentation/xctest/xctskip-swift.struct)
319+
type is used in XCTest. We have received indirect feedback about [`XCTSkip`](https://developer.apple.com/documentation/xctest/xctskip-swift.struct)
320+
indicating its usage is unclear, and sometimes need to help developers who
321+
have written:
322+
323+
```swift
324+
if x {
325+
XCTSkip()
326+
}
327+
```
328+
329+
And don't understand why it has failed to stop the test. More broadly, it is
330+
not common practice in Swift for a function to return an error that the caller
331+
is then responsible for throwing.
332+
333+
- Providing additional `cancel(if:)` and `cancel(unless:)` functions. In
334+
Objective-C, XCTest provides the [`XCTSkipIf()`](https://developer.apple.com/documentation/xctest/xctskipif)
335+
and [`XCTSkipUnless()`](https://developer.apple.com/documentation/xctest/xctskipunless)
336+
macros which capture their condition arguments as strings for display to the
337+
test author. This functionality is not available in Swift, but XCTest's Swift
338+
interface provides equivalent throwing functions as conveniences. We could
339+
provide these functions (without any sort of string-capturing ability) too,
340+
but they provide little additional clarity above an `if` or `guard` statement.
341+
342+
- Implementing cancellation using Swift macros so we can capture an `if` or
343+
`unless` argument as a string. A macro for this feature is probably the wrong
344+
tradeoff between compile-time magic and technical debt.
345+
346+
- Relying solely on [`Task.cancel()`](https://developer.apple.com/documentation/swift/task/cancel()).
347+
Ignoring the interplay between tests and test cases, this approach is
348+
difficult for test authors to use because the current [`Task`](https://developer.apple.com/documentation/swift/task)
349+
instance isn't visible _within_ that task. Instead, a test author would need
350+
to use [`withUnsafeCurrentTask(body:)`](https://developer.apple.com/documentation/swift/withunsafecurrenttask(body:)-6gvhl)
351+
to get a temporary reference to the task and cancel _that_ value. We would
352+
also not have the ability to include a comment and source location information
353+
in the test's console output or an IDE's test result interface.
354+
355+
With that said, [`UnsafeCurrentTask.cancel()`](https://developer.apple.com/documentation/swift/unsafecurrenttask/cancel())
356+
_does_ cancel the test or test case associated with the current task.
357+
358+
## Acknowledgments
359+
360+
Thanks team!
361+
362+
Thanks Arthur! That's right, dinosaurs _do_ say "roar!"
363+
364+
And thanks to [@allevato](https://github.com/allevato) for nerd-sniping me into
365+
writing this proposal.

0 commit comments

Comments
 (0)