diff --git a/docs/guides/openiotsdk_unit_tests.md b/docs/guides/openiotsdk_unit_tests.md index 7b9bba56707870..7560d4dbd959a2 100644 --- a/docs/guides/openiotsdk_unit_tests.md +++ b/docs/guides/openiotsdk_unit_tests.md @@ -6,8 +6,8 @@ tests that are located in the `test` directory, e.g. `src/inet/tests`. Those sources are built as a static library that can be linked to the unit test application separately or as a monolithic test library. The common Matter test library collects all test cases and provides the engine based on -[Nest Labs Unit Test](https://github.com/nestlabs/nlunit-test) to run them in -the application. +[Pigweed Unit Test](https://pigweed.dev/pw_unit_test) to run them in the +application. The Open IoT SDK unit tests implementation are located in the `src/test_driver/openiotsdk/unit-tests` directory. This project builds a @@ -27,6 +27,7 @@ ChipCryptoTests CoreTests CredentialsTest DataModelTests +ICDServerTests InetLayerTests MdnsTests MessagingLayerTests @@ -41,7 +42,6 @@ SetupPayloadTests SupportTests SystemLayerTests TestShell -TransportLayerTests UserDirectedCommissioningTests ``` diff --git a/docs/testing/unit_testing.md b/docs/testing/unit_testing.md index c539420f397e92..64046a4c21a9d6 100644 --- a/docs/testing/unit_testing.md +++ b/docs/testing/unit_testing.md @@ -11,85 +11,347 @@ example applications. - e.g. feature combinations not in example apps. -## Unit testing in the SDK - nlUnitTest +## Unit testing in the SDK using pw_unit_test -The following example gives a small demonstration of how to use nlUnitTest to -write a unit test +This SDK uses Pigweed unit test (pw_unit_test), which is an implementation of +GoogleTest. For more information see the +[pw_unit_test documentation](https://pigweed.dev/pw_unit_test/) or the +[GoogleTest documentation](https://google.github.io/googletest/). -``` -#include -#include -#include +### Simple unit tests -class YourTestContext : public Test::AppContext { - ... -}; +The following example demonstrates how to use pw_unit_test to write a simple +unit test. Each test function is defined using `TEST(NameOfFunction)`. The set +of test functions in a given source file is called a "suite". +``` +#include -static void TestName(nlTestSuite * apSuite, void * apContext) { - // If you register the test suite with a context, cast - // apContext as appropriate - YourTestContext * ctx = static_cast(aContext); +TEST(YourTestFunction1) +{ + // Do some test things here, then check the results using EXPECT_* + SomeTypeX foo; + foo.DoSomething(); + EXPECT_EQ(foo.GetResultCount(), 7); + foo.DoSomethingElse(); + EXPECT_EQ(foo.GetResultCount(), 5); - // Do some test things here, then check the results using NL_TEST_ASSERT - NL_TEST_ASSERT(apSuite, ) + // If you want to abort the rest of the test upon failure, use ASSERT_* + SomeTypeY * ptr = foo.GetSomePointer(); + ASSERT_NE(ptr, nullptr); + ptr->DoTheThing(); // Won't reach here if the ASSERT failed. } -static const nlTest sTests[] = +TEST(YourTestFunction2) { - NL_TEST_DEF("TestName", TestName), // Can have multiple of these - NL_TEST_SENTINEL() // If you forget this, you’re going to have a bad time -}; + // Do some test things here, then check the results using EXPECT_* + SomeTypeZ foo; + foo.DoSomething(); + EXPECT_EQ(foo.GetResultCount(), 3); +} +``` -nlTestSuite sSuite = +See +[TestSpan.cpp](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/tests/TestSpan.cpp) +for an example of a simple unit test. + +In the above example there are no fixtures or setup/teardown behavior. + +### Test fixtures and setup/teardown behavior + +If your tests need fixtures or some kind of setup/teardown you will need to +define a test context that derives from `::testing::Test`. Each of your test +functions will be defined with `TEST_F(NameOfTestContext, NameOfFunction)`. The +following example demonstrates how to use pw_unit_test to write a unit test that +uses fixtures and setup/teardown behavior. + +``` +#include + +class YourTestContext : public ::testing::Test { - "TheNameOfYourTestSuite", // Test name - &sTests[0], // The list of tests to run - TestContext::Initialize, // Runs before all the tests (can be nullptr) - TestContext::Finalize // Runs after all the tests (can be nullptr) +public: + // Performs shared setup for all tests in the test suite. Run once for the whole suite. + static void SetUpTestSuite() + { + // Your per-suite setup goes here: + sPerSuiteFixture.Init(); + ASSERT_TRUE(sPerSuiteFixture.WorkingGreat()); + } + + // Performs shared teardown for all tests in the test suite. Run once for the whole suite. + static void TearDownTestSuite() + { + // Your per-suite teardown goes here: + sPerSuiteFixture.Shutdown(); + } + +protected: + // Performs setup for each test in the suite. Run once for each test function. + void SetUp() + { + // Your per-test setup goes here: + mPerTestFixture.Init(); + ASSERT_TRUE(mPerTestFixture.WorkingGreat()); + } + + // Performs teardown for each test in the suite. Run once for each test function. + void TearDown() + { + // Your per-test teardown goes here: + mPerTestFixture.Shutdown(); + } + +private: + // Your per-suite and per-test fixtures are declared here: + static SomeTypeA sPerSuiteFixture; + SomeTypeB mPerTestFixture; }; +// Your per-suite fixtures are defined here: +SomeTypeA YourTestContext::sPerSuiteFixture; -int YourTestSuiteName() +TEST_F(YourTestContext, YourTestFunction1) { - return chip::ExecuteTestsWithContext(&sSuite); // or “without” + // Do some test things here, then check the results using EXPECT_* + mPerTestFixture.DoSomething(); + EXPECT_EQ(mPerTestFixture.GetResultCount(), 7); + sPerSuiteFixture.DoSomething(); + EXPECT_EQ(sPerSuiteFixture.GetResultCount(), 5); + + // If you want to abort the rest of the test upon failure, use ASSERT_* + SomeTypeC * ptr = mPerTestFixture.GetSomePointer(); + ASSERT_NE(ptr, nullptr); + ptr->DoTheThing(); // Won't reach here if the ASSERT failed. } -CHIP_REGISTER_TEST_SUITE(YourTestSuiteName) +TEST_F(YourTestContext, YourTestFunction2) +{ + // Do some test things here, then check the results using EXPECT_* + mPerTestFixture.DoSomethingElse(); + EXPECT_EQ(mPerTestFixture.GetResultCount(), 9); +} ``` -Each test gets an nlTestSuite object (apSuite) that is passed into the test -assertions, and a void\* context (apContext) that is yours to do with as you -require. +### A loopback messaging context for convenience -The apContext should be derived from +If you need messaging, there is a convenience class [Test::AppContext](https://github.com/project-chip/connectedhomeip/blob/master/src/app/tests/AppTestContext.h) +that you can derive your test context from. It provides a network layer and a +system layer and two secure sessions connected with each other. The following +example demonstrates this. -See -[TestSpan.cpp](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/tests/TestSpan.cpp) -for a great example of a good unit test. +``` +#include + +class YourTestContext : public Test::AppContext +{ +public: + // Performs shared setup for all tests in the test suite. Run once for the whole suite. + static void SetUpTestSuite() + { + AppContext::SetUpTestSuite(); // Call parent. + VerifyOrReturn(!HasFailure()); // Stop if parent had a failure. + + // Your per-suite setup goes here: + sPerSuiteFixture.Init(); + ASSERT_TRUE(sPerSuiteFixture.WorkingGreat()); + } + + // Performs shared teardown for all tests in the test suite. Run once for the whole suite. + static void TearDownTestSuite() + { + // Your per-suite teardown goes here: + sPerSuiteFixture.Shutdown(); + + AppContext::TearDownTestSuite(); // Call parent. + } + +protected: + // Performs setup for each test in the suite. Run once for each test function. + void SetUp() + { + AppContext::SetUp(); // Call parent. + VerifyOrReturn(!HasFailure()); // Stop if parent had a failure. + + // Your per-test setup goes here: + mPerTestFixture.Init(); + ASSERT_TRUE(mPerTestFixture.WorkingGreat()); + } + + // Performs teardown for each test in the suite. Run once for each test function. + void TearDown() + { + // Your per-test teardown goes here: + mPerTestFixture.Shutdown(); + + chip::app::EventManagement::DestroyEventManagement(); + AppContext::TearDown(); // Call parent. + } + +private: + // Your per-suite and per-test fixtures are declared here: + static SomeTypeA sPerSuiteFixture; + SomeTypeB mPerTestFixture; +}; +// Your per-suite fixtures are defined here: +SomeTypeA YourTestContext::sPerSuiteFixture; + +TEST_F(YourTestContext, YourTestFunction1) +{ + // Do some test things here, then check the results using EXPECT_* +} + +TEST_F(YourTestContext, YourTestFunction2) +{ + // Do some test things here, then check the results using EXPECT_* +} +``` + +You don't have to override all 4 functions `SetUpTestsuite`, +`TearDownTestSuite`, `SetUp`, `TearDown`. If you don't need any custom behavior +in one of those functions just omit it. + +If you override one of the setup/teardown functions make sure to invoke the +parent's version of the function as well. `AppContext::SetUpTestSuite` and +`AppContext::SetUp` may generate fatal failures, so after you call these from +your overriding function make sure to check `HasFailure()` and return if the +parent function failed. + +If you don't override any of the setup/teardown functions, you can simply make a +type alias: `using YourTestContext = Test::AppContextPW;` instead of defining +your own text context class. + +## Best practices + +- Try to use as specific an assertion as possible. For example use these + + ``` + EXPECT_EQ(result, 3); + EXPECT_GT(result, 1); + EXPECT_STREQ(myString, "hello"); + ``` + + instead of these + + ``` + EXPECT_TRUE(result == 3); + EXPECT_TRUE(result > 1); + EXPECT_EQ(strcmp(myString, "hello"), 0); + ``` + +- If you want a test to abort when an assertion fails, use `ASSERT_*` instead + of `EXPECT_*`. This will cause the current function to return. + +- If a test calls a subroutine which exits due to an ASSERT failing, execution + of the test will continue after the subroutine call. This is because ASSERT + causes the subroutine to return (as opposed to throwing an exception). If + you want to prevent the test from continuing, check the value of + `HasFailure()` and stop execution if true. Example: + + ``` + void Subroutine() + { + ASSERT_EQ(1, 2); // Fatal failure. + } + + TEST(YourTestContext, YourTestFunction1) + { + Subroutine(); // A fatal failure happens in this subroutine... + // ... however execution still continues. + print("This gets executed"); + VerifyOrReturn(!HasFailure()); + print("This does not get executed"); + } + ``` + +- If you want to force a fatal failure use `FAIL()`, which will record a fatal + failure and exit the current function. This is similar to using + `ASSERT_TRUE(false)`. If you want to force a non-fatal failure use + `ADD_FAILURE()`, which will record a non-fatal failure and continue + executing the current function. This is similar to using + `EXPECT_TRUE(false)`. + +- `ASSERT_*` and `FAIL` will only work in functions with void return type, + since they generate a `return;` statement. If you must use these in a + non-void function, instead use `EXPECT_*` or `ADD_FAILURE` and then check + `HasFailure()` afterward and return if needed. + +- If your test requires access to private/protected members of the underlying + class you're testing, you'll need to create an accessor class that performs + these operations and is friended to the underlying class. Please name the + class `chip::Test::SomethingTestAccess` where `Something` is the name of the + underlying class whose private/protected members you're trying to access. + Then add `friend class chip::Test::SomethingTestAccess;` to the underlying + class. Make sure your test's BUILD.gn file contains + `sources = [ "SomethingTestAccess.h" ]`. Before creating a new TestAccess + class, check if one already exists. If it does exist but doesn't expose the + member you need, you can add a function to that class to do so. Note that + you should make these functions as minimal as possible, with no logic + besides just exposing the private/protected member. + - For an example see `ICDConfigurationDataTestAccess` which is defined in + [ICDConfigurationDataTestAccess.h](https://github.com/project-chip/connectedhomeip/blob/master/src/app/icd/server/tests/ICDConfigurationDataTestAccess.h), + friends the underlying class in + [ICDConfigurationData.h](https://github.com/project-chip/connectedhomeip/blob/master/src/app/icd/server/ICDConfigurationData.h), + is included as a source in + [BUILD.gn](https://github.com/project-chip/connectedhomeip/blob/master/src/app/icd/server/tests/BUILD.gn), + and is used by a test in + [TestICDManager.cpp](https://github.com/project-chip/connectedhomeip/blob/master/src/app/icd/server/tests/TestICDManager.cpp). + - For another example see `TCPBaseTestAccess` which is defined in + [TCPBaseTestAccess.h](https://github.com/project-chip/connectedhomeip/blob/master/src/transport/raw/tests/TCPBaseTestAccess.h), + friends the underlying class in + [TCP.h](https://github.com/project-chip/connectedhomeip/blob/master/src/transport/raw/TCP.h), + is included as a source in + [BUILD.gn](https://github.com/project-chip/connectedhomeip/blob/master/src/transport/raw/tests/BUILD.gn), + and is used by a test in + [TestTCP.cpp](https://github.com/project-chip/connectedhomeip/blob/master/src/transport/raw/tests/TestTCP.cpp). + +## Compiling and running + +- Add to `src/some_directory/tests/BUILD.gn` + + - Example + + ``` + chip_test_suite("tests") { + output_name = "libSomethingTests" + + test_sources = [ + "TestSuite1.cpp", + "TestSuite2.cpp", + // Other test source files go here. + ] + + sources = [ + // Non-test source files go here. + ] + + cflags = [ "-Wconversion" ] + + public_deps = [ + // Dependencies go here. + ] + } + ``` -## nlUnitTest - Compiling and running + - Another example: + [src/lib/support/tests/BUILD.gn](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/tests/BUILD.gn) -- Add to src/some_directory/tests/BUILD.gn - - chip_test_suite_using_nltest("tests") - - See for example - [src/lib/support/tests/BUILD.gn](https://github.com/project-chip/connectedhomeip/blob/master/src/lib/support/tests/BUILD.gn) -- [./gn_build.sh](https://github.com/project-chip/connectedhomeip/blob/master/gn_build.sh) - will build and run all tests +- Build and run all tests with + [./gn_build.sh](https://github.com/project-chip/connectedhomeip/blob/master/gn_build.sh) - CI runs this, so any unit tests that get added will automatically be - added to the CI + added to the CI. - Test binaries are compiled into: - - out/debug//tests - - e.g. out/debug/linux_x64_clang/tests -- Tests are run when ./gn_build.sh runs, but you can run them individually in - a debugger from their location. + - `out/debug//tests` + - e.g. `out/debug/linux_x64_clang/tests` +- Tests are run when `./gn_build.sh` runs, but you can run them individually + in a debugger from their location. ## Debugging unit tests -- After running ./gn_build.sh, test binaries are compiled into - - out/debug//tests - - e.g. out/debug/linux_x64_clang/tests -- Individual binaries, can be run through regular tools: +- After running `./gn_build.sh`, test binaries are compiled into + - `out/debug//tests` + - e.g. `out/debug/linux_x64_clang/tests` +- Individual binaries can be run through regular tools: - gdb - valgrind - Your favorite tool that you tell everyone about.