Updated docs for writing custom tests

powershell/public/Invoke-MtGraphRequest.ps1

@@ -16,7 +16,7 @@
     :::
 
  .Example
-    Invoke-MtGraph -RelativeUri "users" -Filter "displayName eq 'John Doe'" -Select "displayName" -Top 10
+    Invoke-MtGraph -RelativeUri "users" -Filter "displayName eq 'John Doe'" -Select "displayName"
 
     Get all users with a display name of "John Doe" and return the first 10 results.
 

website/docs/writing-tests/advanced-concepts.md

@@ -0,0 +1,183 @@
+---
+title: Advanced guide
+sidebar_position: 3
+---
+
+## Overview
+
+In this guide we will cover advanced concepts for writing custom tests with Maester.
+
+## Invoke-MtGraphRequest
+
+Maester provides a function called `Invoke-MtGraphRequest` that allows you to make direct calls to the Microsoft Graph API. This is an enhanced version of the `Invoke-MgGraphRequest` function that has been optimized for Maester's use case to query Microsoft Graph data.
+
+Here's an example of how you can use `Invoke-MtGraphRequest` to get all the users in your tenant.
+
+```powershell
+$users = Invoke-MtGraphRequest -RelativeUri "users"
+```
+
+### Caching: Invoke-MtGraphRequest's secret sauce
+
+`Invoke-MtGraphRequest` has built-in caching to reduce the number of calls to the Microsoft Graph API when running Maester tests.
+
+This way you can write tests that call into any Graph API and if that data has already been fetched in the Maester run, the cached data will be used instead of querying Microsoft Graph. This is one of the reasons we can run multiple tests in a very performant way.
+
+The cache is reset when you run Invoke-Maester to ensure you always have the latest data.
+
+If your tests use Graph cmdlets like `Get-MgUser`, they will not benefit from this caching mechanism and will make a call to the Graph API every time they are run.
+
+### Other key features of `Invoke-MtGraphRequest`:
+
+In addition to caching, `Invoke-MtGraphRequest` has other key features that make it very easy to write tests that query data.
+
+- Automatically handles pagination and get's all of the users by default (you don't need to specific -All)
+- Includes **ConsistencyLevel** by default to all the calls. This works for Maester's read-only use case and allows you to use any of the advanced query filter options without worrying about the consistency flag.
+- Provides automatic support for batching by passing in an array of Object IDs to the `-UniqueId` parameter.
+- Named parameters for `Select`, `Filter` and `QueryParameters` to make it easier to write complex queries.
+
+Here are a few examples.
+
+#### Get selected list of users users with specific properties
+
+Use the `UniqueId` parameter to get specific users by their Object ID and select only the properties you need.
+
+The $usersIds array can have one or hundreds of object IDs. Invoke-MtGraphRequest will optimize the calls by batching and paging through the results.
+
+```powershell
+$userIds = @($globalAdministrators.Id)
+
+Write-Verbose "Requesting users onPremisesSyncEnabled property"
+$users = Invoke-MtGraphRequest -RelativeUri "users" -UniqueId $userIds -Select id, displayName, onPremisesSyncEnabled
+
+```
+
+#### Specify api version, filters, query parameters with expand
+
+This example shows how you can splat the code to make it easier to read when you have a complex query.
+
+```powershell
+$policySplat = @{
+    ApiVersion      = "beta"
+    RelativeUri     = "policies/roleManagementPolicyAssignments"
+    Filter          = "scopeId eq '/' and scopeType eq 'DirectoryRole' and roleDefinitionId eq '$($globalAdministratorsRole.id)'"
+    QueryParameters = @{
+        expand = "policy(expand=rules)"
+    }
+}
+$policy = Invoke-MtGraphRequest @policySplat
+```
+
+To learn more see [Invoke-MtGraphRequest](https://github.com/maester365/maester/blob/main/powershell/public/Invoke-MtGraphRequest.ps1).
+
+## Splitting tests into multiple files
+
+As you write more tests you might find it helpful to split out the markdown part of the tests into a separate file. This helps reduce clutter in the test code and also allows content writers to independently edit the markdown files. Almost all the out of the box Maester tests use this approach of splitting out the markdown content for the test.
+
+Here's an example of how you can split out the markdown content into a separate file.
+
+This custom test checks if there are any users without a manager assigned.
+
+### Step 1: Create the tests file in the `Custom` folder
+
+Create a new file in the `Custom` folder with the `.Tests.ps1` suffix.
+
+#### ContosoUsers.Tests.ps1
+
+```powershell
+BeforeAll {
+    . $PSScriptRoot/Test-ContosoUsersMissingManagers.ps1
+}
+Describe "Contoso" -Tag "Entra", "CustomTests", "Users" {
+    It "CTS.1001: Manager Attribute - All users should have a manager attribute set" {
+        $result = Test-ContosoUsersMissingManagers
+        $result | Should -Be $true -Because "All users should have a manager assigned."
+    }
+}
+```
+
+### Step 2: Create test functions file
+
+Create the test file in the `Custom` folder that was referred to in the `BeforeAll` block in the previous step.
+
+#### Test-ContosoUsersMissingManagers.ps1
+
+```powershell
+function Test-ContosoUsersMissingManagers {
+    $result = $true
+
+    try {
+        # Retrieve all users from Microsoft Graph
+        $users = Invoke-MtGraphRequest -RelativeUri "users" -Filter "userType eq 'Member'"
+
+        # Initialize an array to track users without a manager
+        $usersWithoutManager = @()
+
+        # Loop through each user and ensure they have a manager assigned
+        foreach ($user in $users) {
+            if($user.jobTitle -eq "CEO" -or $user.displayName -eq "On-Premises Directory Synchronization Service Account" ) {
+            continue
+}
+
+            # Fetch the manager for the current user
+            $manager = Get-MgUserManager -UserId $user.Id -ErrorAction SilentlyContinue
+
+            if ([string]::IsNullOrEmpty($manager)) {
+                $result = $false
+                $usersWithoutManager += $user
+            }
+        }
+
+        if ($result) {
+            $TestResults = "Well done! There were no users with out managers assigned."
+        } else {
+            $TestResults += "No managers are assigned for the following users.`n%TestResult%"
+        }
+
+        Add-MtTestResultDetail -Result $TestResults -GraphObjects $usersWithoutManager -GraphObjectType Users
+    } catch {
+        $result = $false
+        Write-Error $_.Exception.Message
+    }
+
+    return $result
+}
+```
+
+:::note
+To use the markdown content from the file, **do not** include the `-Description` parameter when calling `Add-MtTestResultDetail`.
+:::
+
+### Step 3: Create the markdown file
+
+Create a markdown file in the `Custom` folder **with the same name as the test file** but with the `.md` extension.
+
+#### Test-ContosoUsersMissingManagers.md
+
+```md
+This test checks if there are any users without a manager assigned.
+
+Contoso's company policy requires that all users have a manager assigned to them. This is important for accountability and delegation of responsibilities.
+
+**To remediate this issue:**
+
+- Identify the users without a manager.
+- Raise a ticket in Service Now using [Form: Manager Missing - HR Ticket](https://contoso.service-now.com/managermissing) to request the manager assignment for the users identified in this test.
+  - 🔺 If this is not actioned in three days, escalate to the HR manager.
+
+**Learn more:**
+
+- [Manager Missing - HR Ticket](https://contoso.service-now.com/managermissing)
+- [HR Escalation Process](https://contoso.service-now.com/hrescalation)
+
+<!--- Results --->
+
+%TestResult%
+
+```
+
+### Step 4: Run the test
+
+Running the test should now show the markdown content in the test results.
+
+![ContosoUsersMissingManagers](img/advanced-concepts-split-markdown.png)

website/docs/writing-tests/formatting-test-results.md

@@ -0,0 +1,174 @@
+---
+title: Formatting test results
+sidebar_position: 2
+---
+
+## Overview
+
+In this section we will learn how to format test results to provide more context and make them easier to understand for the person viewing the results.
+
+Let's write a test to check if conditional access policies are following the company's standards.
+
+## A custom Maester test to check conditional access policies standards
+
+Our organization has a policy that all disabled conditional access policies should include the reason for the policy being disabled. This is done by adding a note to the display name in the format `Disabled: <reason>`.
+
+To check if the conditional access policies are following this standard, we can write the following custom test and add it to the `ContosoEntra.Tests.ps1` file in the `Custom` folder (see previous article).
+
+You can copy and paste the following code and add it to the end of the `ContosoEntra.Tests.ps1` file.
+
+```powershell
+Describe "ContosoEntraConfig" -Tag "CA", "Contoso" {
+   It "Disabled CA policies must have reason for being disabled" {
+
+       try {
+          $policies = Get-MgIdentityConditionalAccessPolicy -All
+          $disabledWithoutReason = $policies | Where-Object { $_.State -eq "Disabled" -and $_.DisplayName -notlike "*Disabled:*" }
+       } catch {
+           Write-Error $_.Exception.Message
+       }
+       $disabledWithoutReason | Should -Be 0
+    }
+}
+```
+
+You can run the test using `Invoke-Maester` and check the results.
+
+What you will notice is that the test results are not very informative. The test will pass or fail, but you won't know which conditional access policies are not following the standard.
+
+![Test results without formatting](img/unformatted-test-result.png)
+
+## Formatting the test results with Add-MtTestResultDetail
+
+### Basic formatting
+
+To provide more context in the test results, you can use Maester's `Add-MtTestResultDetail` function to provide additional context.
+
+![Test results with basic formatting](img/formatted-test-basic.png)
+
+By providing the `-Description` and `-Result` parameters the test results are now more informative and provide context on what the test is checking and the outcome.
+
+Here's the code for the complete custom test.
+
+Copy and paste this code to any `*.Tests.ps1` file in the `Custom` folder to try it out.
+
+```powershell
+Describe "ContosoEntraConfig" -Tag "Privilege", "Contoso" {
+    It "Disabled CA policies must have reason for being disabled" {
+
+        try {
+            $policies = Get-MgIdentityConditionalAccessPolicy -All
+
+            $disabledWithoutReason = $policies | Where-Object { $_.State -eq "Disabled" -and $_.DisplayName -notlike "*Disabled:*" }
+
+            $testDescription = "Checks if the disabled policies have the reason for being disabled."
+            if ($disabledWithoutReason.Count -gt 0) {
+                $result = "There are $($disabledWithoutReason.Count) disabled policies without a reason for being disabled."
+                Add-MtTestResultDetail -Description $testDescription -Result $result
+            } else {
+                Add-MtTestResultDetail -Description $testDescription -Result "Well done. All disabled policies have a reason for being disabled."
+            }
+        } catch {
+            Write-Error $_.Exception.Message
+        }
+
+        $disabledWithoutReason | Should -Be 0
+    }
+}
+```
+
+### Adding graph objects
+
+The test result now shows that 24 policies are failing the test but doesn't provide the names of the policies. To provide more context, you can add the names of the policies that are failing by passing the policies to the `Add-MtTestResultDetail` function.
+
+![Test results showing graph objects](img/formatted-test-graph.png)
+
+The `-GraphObjects` and `-GraphObjectType` parameters in `Add-MtTestResultDetail` allow you to pass objects to the test results and specify the type of object.
+
+The test results can then display the names of the objects and also provide a deep link to the object in the Microsoft admin portal.
+
+:::note
+When using `-GraphObjects` the `-Result` string parameter needs to include `%TestResult%` at the position where the object names will be inserted.
+
+The `%TestResult%` placeholder will be replaced with the names of the objects in the test results.
+:::
+
+The current list of supported object types includes Users, Groups, Devices, ConditionalAccess, AuthenticationMethod, AuthorizationPolicy, ConsentPolicy, Domains, IdentityProtection and UserRole.
+
+Here's the updated test with the graph objects that you can try out.
+
+```powershell
+Describe "ContosoEntraConfig" -Tag "Privilege", "Contoso" {
+    It "Disabled CA policies must have reason for being disabled" {
+
+        try {
+            $policies = Get-MgIdentityConditionalAccessPolicy -All
+
+            $disabledWithoutReason = $policies | Where-Object { $_.State -eq "Disabled" -and $_.DisplayName -notlike "*Disabled:*" }
+
+            $testDescription = "Checks if the disabled policies have the reason for being disabled."
+            if ($disabledWithoutReason.Count -gt 0) {
+                $result = "There are $($disabledWithoutReason.Count) disabled policies without a reason for being disabled.`n`n%TestResult%"
+                Add-MtTestResultDetail -Description $testDescription -Result $result -GraphObjects $disabledWithoutReason -GraphObjectType ConditionalAccess
+            } else {
+                Add-MtTestResultDetail -Description $testDescription -Result "Well done. All disabled policies have a reason for being disabled."
+            }
+        } catch {
+            Write-Error $_.Exception.Message
+        }
+
+        $disabledWithoutReason | Should -Be 0
+    }
+}
+```
+
+To add support for additional types see [Add-MtTestResultDetail](https://github.com/maester365/maester/blob/main/powershell/public/Add-MtTestResultDetail.ps1) and [Get-GraphObjectMarkdown](https://github.com/maester365/maester/blob/main/powershell/internal/Get-GraphObjectMarkdown.ps1).
+
+#### Adding custom markdown
+
+While the `-GraphObjects` parameter provides an easy option to link to common objects, you can also provide custom markdown to the `-Result` parameter. This allows you to format the test results in any way you like.
+
+Here's an example of how you can use a markdown table to display the results including deep links to the policies in the Microsoft Entra portal.
+
+![Test results with custom markdown](img/formatted-test-custom-markdown.png)
+
+```powershell
+Describe "ContosoEntraConfig" -Tag "Privilege", "Contoso" {
+    It "Disabled CA policies must have reason for being disabled" {
+
+        try {
+            $policies = Get-MgIdentityConditionalAccessPolicy -All
+
+            $disabledWithoutReason = $policies | Where-Object { $_.State -eq "Disabled" -and $_.DisplayName -notlike "*Disabled:*" }
+            $disabledWithReason = $policies | Where-Object { $_.State -eq "Disabled" -and $_.DisplayName -like "*Disabled:*" }
+
+            $testDescription = "Checks if the disabled policies have the reason for being disabled."
+
+            if ($disabledWithoutReason.Count -gt 0 ) {
+                $result = "There are $($disabledWithoutReason.Count) disabled policies without a reason for being disabled."
+            } else {
+                $result = "Well done. All disabled policies have a reason for being disabled."
+            }
+            $portalLink = "https://entra.microsoft.com/#view/Microsoft_AAD_ConditionalAccess/PolicyBlade/policyId/{0}"
+            if($disabledWithReason.Count -gt 0 -or $disabledWithoutReason.Count -gt 0){
+                $result += "`n`n"
+                $result += "| Disabled CA Policy | Reason for disabling policy |`n"
+                $result += "| --- | --- |`n"
+                foreach($policy in $disabledWithReason){
+                    $nameSplit = $policy.DisplayName -split ":Disabled:"
+                    $result += "| ✅ [$($nameSplit[0])]($($portalLink -f $policy.id)) | $($nameSplit[1]) |`n"
+                }
+                foreach($policy in $disabledWithoutReason){
+                    $result += "| ❌ [$($policy.DisplayName)]($($portalLink -f $policy.id)) | No reason provided |`n"
+                }
+            }
+            Add-MtTestResultDetail -Description $testDescription -Result $result
+        } catch {
+            Write-Error $_.Exception.Message
+        }
+
+        $disabledWithoutReason | Should -Be 0
+    }
+}
+```
+

website/docs/writing-tests.md → website/docs/writing-tests/index.mdx

@@ -1,5 +1,6 @@
 ---
-title: Writing custom tests
+title: Getting started
+sidebar_position: 1
 ---
 
 ## Introduction
@@ -35,7 +36,12 @@ Describe "ContosoEntraConfig" -Tag "Privilege", "Contoso" {
 
         $groupId = "e05d094c-a785-4a7c-b7eb-f0ccebbe009e"
 
-        $memberCount = Get-MgGroupTransitiveMemberCount -GroupId $groupId -ConsistencyLevel eventual
+        try {
+            $memberCount = Get-MgGroupTransitiveMemberCount -GroupId $groupId -ConsistencyLevel eventual
+        }
+        catch {
+            Write-Error $_.Exception.Message
+        }
 
         # Test if the group exists and has members
         $memberCount | Should -BeGreaterThan 0