Skip to content

Commit 561c637

Browse files
committed
Adding Documentation
1 parent 2420d7a commit 561c637

File tree

2 files changed

+177
-0
lines changed

2 files changed

+177
-0
lines changed

docs/guides/BUILDING.md

+20
Original file line numberDiff line numberDiff line change
@@ -382,6 +382,26 @@ They pick up environment variables such as `$CFLAGS`, `$CXXFLAGS` and
382382
383383
You likely want `libfuzzer` + `asan` builds instead for local testing.
384384
385+
### `pw_fuzzer` `FuzzTests`
386+
387+
An Alternative way for writing and running Fuzz Tests is Google's `FuzzTest`
388+
framework, integrated through `pw_fuzzer`. The Tests will have to be built and
389+
executed manually.
390+
391+
```
392+
./scripts/build/build_examples.py --target linux-x64-tests-clang-pw-fuzztest build
393+
```
394+
395+
NOTE: `asan` is enabled by default in FuzzTest, so adding to build_examples
396+
target does not make a difference.
397+
398+
Tests will be located in:
399+
`out/linux-x64-tests-clang-pw-fuzztest/chip_pw_fuzztest/tests/` where
400+
`chip_pw_fuzztest` is the name of the toolchain used.
401+
402+
- Details on How To Run Fuzz Tests in
403+
[Running FuzzTests](https://github.com/project-chip/connectedhomeip/blob/master/docs/testing/fuzz_testing.md)
404+
385405
## Build custom configuration
386406
387407
The build is configured by setting build arguments. These you can set in one of

docs/testing/fuzz_testing.md

+157
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
# Fuzz testing
2+
3+
- Fuzz Testing involves providing random and unexpected data as input to
4+
functions and methods to uncover bugs, security vulnerabilities, or to
5+
determine if the software crashes.
6+
- it is often continuous; the function under test is called iteratively with
7+
thousands of different inputs.
8+
- Fuzz testing is often done with sanitizers enabled; to catch memory errors
9+
and undefined behaviour.
10+
- The most commonly used fuzz testing frameworks for C/C++ are LibFuzzer and
11+
AFL.
12+
- [Google's FuzzTest](https://github.com/google/fuzztest) is a newer framework
13+
that simplifies writing fuzz tests with user-friendly APIs and offers more
14+
control over input generation. It also integrates seamlessly with Google
15+
Test (GTest).
16+
17+
## `Google's FuzzTest`
18+
19+
- Google FuzzTest is integrated through Pigweed's
20+
[pw_fuzzer](https://pigweed.dev/pw_fuzzer/concepts.html).
21+
22+
### Use cases
23+
24+
1. Finding Undefined Behavior with Sanitizers:
25+
26+
- Running fuzz tests while checking if a crash or other sanitizer-detected
27+
error occurs, allowing detection of subtle memory issues like buffer
28+
overflows and use-after-free errors.
29+
30+
2. Find Correctness Bugs using Assertions:
31+
- For example, in Roundtrip Fuzzing, fuzzed input is encoded, decoded, and
32+
then verified to match the original input. An example of this can be found
33+
in src/setup_payload/tests/FuzzBase38PW.cpp.
34+
35+
- More information can be found in the
36+
[FuzzTest Use Cases](https://github.com/google/fuzztest/blob/main/doc/use-cases.md)
37+
documentation.
38+
39+
### Writing FuzzTests
40+
41+
Keywords: Property Function, Input Domain
42+
43+
- FuzzTests are instantiated through the macro call of `FUZZ_TEST`:
44+
45+
```cpp
46+
FUZZ_TEST(TLVReader, FuzzTlvReader).WithDomains(fuzztest::Arbitrary<std::vector<std::uint8_t>>());
47+
```
48+
49+
- The Macro invocation calls the **Property Function**, which is
50+
`FuzzTlvReader` above.
51+
52+
- The **input domains** define the range and type of inputs that the
53+
**property function** will receive during fuzzing, specified using the
54+
`.WithDomains()` clause.
55+
- In the macro above, FuzzTest will generate a wide range of possible byte
56+
vectors to thoroughly test the `FuzzTlvReader` function.
57+
58+
#### The Property Function
59+
60+
```cpp
61+
// The Property Function
62+
void FuzzTlvRead(const std::vector<std::uint8_t> & bytes)
63+
{
64+
TLVReader reader;
65+
reader.Init(bytes.data(), bytes.size());
66+
chip::TLV::Utilities::Iterate(reader, FuzzIterator, nullptr);
67+
}
68+
```
69+
70+
- The Property Functions must return a `void`
71+
- The function will be run with many different inputs in the same process,
72+
trying to trigger a crash.
73+
- It is possible to include Assertions such as during Round-Trip Fuzzing
74+
75+
- More Information:
76+
https://github.com/google/fuzztest/blob/main/doc/fuzz-test-macro.md#the-property-function
77+
78+
#### Input Domains
79+
80+
- FuzzTest Offers many Input Domains, all of which are part of the
81+
`fuzztest::` namespace.
82+
- All native C++ types can be used through `Arbitrary<T>()`:
83+
84+
```cpp
85+
FUZZ_TEST(Base38Decoder, FuzzQRCodeSetupPayloadParser).WithDomains(Arbitrary<std::string>());
86+
```
87+
88+
- using vector domains is one of the most common. It is possible to limit the
89+
size of the input vectors (or any container domain) using `.WithMaxSize()`
90+
or `.WithMinSize()`, as shown below:
91+
92+
```cpp
93+
FUZZ_TEST(MinimalmDNS, TxtResponderFuzz).WithDomains(Arbitrary<vector<uint8_t>>().WithMaxSize(254));
94+
```
95+
96+
- `ElementOf` is particularly useful as it allows us to define a domain by
97+
explicitly enumerating the set of values in it and passing it to FuzzTest
98+
invocation. Example:
99+
100+
```cpp
101+
AnyProtocolID()
102+
{
103+
return ElementOf({ chip::Protocols::SecureChannel::Id, chip::Protocols::InteractionModel::Id, chip::Protocols::BDX::Id,
104+
chip::Protocols::UserDirectedCommissioning::Id });
105+
}
106+
107+
FUZZ_TEST(PayloadDecoder, RunDecodeFuzz).WithDomains(Arbitrary<std::vector<std::uint8_t>>(), AnyProtocolID(), Arbitrary<uint8_t>());
108+
```
109+
110+
- A detailed reference for input domains can be found here:
111+
[FuzzTest Domain Reference](https://github.com/google/fuzztest/blob/main/doc/domains-reference.md#elementof-domains-element-of).
112+
113+
### Running FuzzTests
114+
115+
There are several ways to run the tests:
116+
117+
1. Unit-test mode (where the inputs are only fuzzed for a second):
118+
119+
```bash
120+
./fuzz-chip-cert-pw
121+
```
122+
123+
2. Continuous fuzzing mode; we need to first list the tests, then specify the
124+
FuzzTestCase to run:
125+
126+
```bash
127+
$ ./fuzz-chip-cert-pw --list_fuzz_tests
128+
[.] Sanitizer coverage enabled. Counter map size: 11134, Cmp map size: 262144
129+
[*] Fuzz test: ChipCert.ChipCertFuzzer
130+
[*] Fuzz test: ChipCert.DecodeChipCertFuzzer
131+
132+
$ ./fuzz-chip-cert-pw --fuzz=ChipCert.DecodeChipCertFuzzer
133+
```
134+
135+
3. Running all Tests in a TestSuite for a specific time, e.g for 10 minutes
136+
137+
```bash
138+
#both Fuzz Tests will be run for 10 minutes each
139+
./fuzz-chip-cert-pw --fuzz_for=10m
140+
```
141+
142+
4. For Help
143+
144+
```bash
145+
# FuzzTest related help
146+
./fuzz-chip-cert-pw --helpfull
147+
148+
# gtest related help
149+
./fuzz-chip-cert-pw --help
150+
151+
```
152+
153+
#### TO ADD:
154+
155+
- More Information on Test Fixtures (After issues are resolved)
156+
- How to add FuzzTests to the Build System
157+
- More Information on OSS-FUZZ

0 commit comments

Comments
 (0)