|
| 1 | +# Fuzz testing |
| 2 | + |
| 3 | +- Fuzz Testing involves providing random and unexpected data as input to |
| 4 | + functions and methods to uncover bugs, security vulnerabilities, or to |
| 5 | + determine if the software crashes. |
| 6 | +- it is often continuous; the function under test is called iteratively with |
| 7 | + thousands of different inputs. |
| 8 | +- Fuzz testing is often done with sanitizers enabled; to catch memory errors |
| 9 | + and undefined behaviour. |
| 10 | +- The most commonly used fuzz testing frameworks for C/C++ are LibFuzzer and |
| 11 | + AFL. |
| 12 | +- [Google's FuzzTest](https://github.com/google/fuzztest) is a newer framework |
| 13 | + that simplifies writing fuzz tests with user-friendly APIs and offers more |
| 14 | + control over input generation. It also integrates seamlessly with Google |
| 15 | + Test (GTest). |
| 16 | + |
| 17 | +## `Google's FuzzTest` |
| 18 | + |
| 19 | +- Google FuzzTest is integrated through Pigweed's |
| 20 | + [pw_fuzzer](https://pigweed.dev/pw_fuzzer/concepts.html). |
| 21 | + |
| 22 | +### Use cases |
| 23 | + |
| 24 | +1. Finding Undefined Behavior with Sanitizers: |
| 25 | + |
| 26 | + - Running fuzz tests while checking if a crash or other sanitizer-detected |
| 27 | + error occurs, allowing detection of subtle memory issues like buffer |
| 28 | + overflows and use-after-free errors. |
| 29 | + |
| 30 | +2. Find Correctness Bugs using Assertions: |
| 31 | + - For example, in Roundtrip Fuzzing, fuzzed input is encoded, decoded, and |
| 32 | + then verified to match the original input. An example of this can be found |
| 33 | + in src/setup_payload/tests/FuzzBase38PW.cpp. |
| 34 | + |
| 35 | +- More information can be found in the |
| 36 | + [FuzzTest Use Cases](https://github.com/google/fuzztest/blob/main/doc/use-cases.md) |
| 37 | + documentation. |
| 38 | + |
| 39 | +### Writing FuzzTests |
| 40 | + |
| 41 | +Keywords: Property Function, Input Domain |
| 42 | + |
| 43 | +- FuzzTests are instantiated through the macro call of `FUZZ_TEST`: |
| 44 | + |
| 45 | +```cpp |
| 46 | +FUZZ_TEST(TLVReader, FuzzTlvReader).WithDomains(fuzztest::Arbitrary<std::vector<std::uint8_t>>()); |
| 47 | +``` |
| 48 | +
|
| 49 | +- The Macro invocation calls the **Property Function**, which is |
| 50 | + `FuzzTlvReader` above. |
| 51 | +
|
| 52 | +- The **input domains** define the range and type of inputs that the |
| 53 | + **property function** will receive during fuzzing, specified using the |
| 54 | + `.WithDomains()` clause. |
| 55 | +- In the macro above, FuzzTest will generate a wide range of possible byte |
| 56 | + vectors to thoroughly test the `FuzzTlvReader` function. |
| 57 | +
|
| 58 | +#### The Property Function |
| 59 | +
|
| 60 | +```cpp |
| 61 | +// The Property Function |
| 62 | +void FuzzTlvRead(const std::vector<std::uint8_t> & bytes) |
| 63 | +{ |
| 64 | + TLVReader reader; |
| 65 | + reader.Init(bytes.data(), bytes.size()); |
| 66 | + chip::TLV::Utilities::Iterate(reader, FuzzIterator, nullptr); |
| 67 | +} |
| 68 | +``` |
| 69 | + |
| 70 | +- The Property Functions must return a `void` |
| 71 | +- The function will be run with many different inputs in the same process, |
| 72 | + trying to trigger a crash. |
| 73 | +- It is possible to include Assertions such as during Round-Trip Fuzzing |
| 74 | + |
| 75 | +- More Information: |
| 76 | + https://github.com/google/fuzztest/blob/main/doc/fuzz-test-macro.md#the-property-function |
| 77 | + |
| 78 | +#### Input Domains |
| 79 | + |
| 80 | +- FuzzTest Offers many Input Domains, all of which are part of the |
| 81 | + `fuzztest::` namespace. |
| 82 | +- All native C++ types can be used through `Arbitrary<T>()`: |
| 83 | + |
| 84 | +```cpp |
| 85 | +FUZZ_TEST(Base38Decoder, FuzzQRCodeSetupPayloadParser).WithDomains(Arbitrary<std::string>()); |
| 86 | +``` |
| 87 | +
|
| 88 | +- using vector domains is one of the most common. It is possible to limit the |
| 89 | + size of the input vectors (or any container domain) using `.WithMaxSize()` |
| 90 | + or `.WithMinSize()`, as shown below: |
| 91 | +
|
| 92 | +```cpp |
| 93 | +FUZZ_TEST(MinimalmDNS, TxtResponderFuzz).WithDomains(Arbitrary<vector<uint8_t>>().WithMaxSize(254)); |
| 94 | +``` |
| 95 | + |
| 96 | +- `ElementOf` is particularly useful as it allows us to define a domain by |
| 97 | + explicitly enumerating the set of values in it and passing it to FuzzTest |
| 98 | + invocation. Example: |
| 99 | + |
| 100 | +```cpp |
| 101 | +AnyProtocolID() |
| 102 | +{ |
| 103 | + return ElementOf({ chip::Protocols::SecureChannel::Id, chip::Protocols::InteractionModel::Id, chip::Protocols::BDX::Id, |
| 104 | + chip::Protocols::UserDirectedCommissioning::Id }); |
| 105 | +} |
| 106 | + |
| 107 | +FUZZ_TEST(PayloadDecoder, RunDecodeFuzz).WithDomains(Arbitrary<std::vector<std::uint8_t>>(), AnyProtocolID(), Arbitrary<uint8_t>()); |
| 108 | +``` |
| 109 | +
|
| 110 | +- A detailed reference for input domains can be found here: |
| 111 | + [FuzzTest Domain Reference](https://github.com/google/fuzztest/blob/main/doc/domains-reference.md#elementof-domains-element-of). |
| 112 | +
|
| 113 | +### Running FuzzTests |
| 114 | +
|
| 115 | +There are several ways to run the tests: |
| 116 | +
|
| 117 | +1. Unit-test mode (where the inputs are only fuzzed for a second): |
| 118 | +
|
| 119 | +```bash |
| 120 | +./fuzz-chip-cert-pw |
| 121 | +``` |
| 122 | + |
| 123 | +2. Continuous fuzzing mode; we need to first list the tests, then specify the |
| 124 | + FuzzTestCase to run: |
| 125 | + |
| 126 | +```bash |
| 127 | +$ ./fuzz-chip-cert-pw --list_fuzz_tests |
| 128 | +[.] Sanitizer coverage enabled. Counter map size: 11134, Cmp map size: 262144 |
| 129 | +[*] Fuzz test: ChipCert.ChipCertFuzzer |
| 130 | +[*] Fuzz test: ChipCert.DecodeChipCertFuzzer |
| 131 | + |
| 132 | +$ ./fuzz-chip-cert-pw --fuzz=ChipCert.DecodeChipCertFuzzer |
| 133 | +``` |
| 134 | + |
| 135 | +3. Running all Tests in a TestSuite for a specific time, e.g for 10 minutes |
| 136 | + |
| 137 | +```bash |
| 138 | +#both Fuzz Tests will be run for 10 minutes each |
| 139 | +./fuzz-chip-cert-pw --fuzz_for=10m |
| 140 | +``` |
| 141 | + |
| 142 | +4. For Help |
| 143 | + |
| 144 | +```bash |
| 145 | +# FuzzTest related help |
| 146 | +./fuzz-chip-cert-pw --helpfull |
| 147 | + |
| 148 | +# gtest related help |
| 149 | +./fuzz-chip-cert-pw --help |
| 150 | + |
| 151 | +``` |
| 152 | + |
| 153 | +#### TO ADD: |
| 154 | + |
| 155 | +- More Information on Test Fixtures (After issues are resolved) |
| 156 | +- How to add FuzzTests to the Build System |
| 157 | +- More Information on OSS-FUZZ |
0 commit comments