Skip to content

Commit 3357b55

Browse files
Add help screen to slang-test (shader-slang#6611)
This commit adds a help screen to slang-test to improve usability; and update README.md with clearer instructions. The help screen is displayed when: - User explicitly requests help with -h or --help flags - An unknown option is provided - A required argument for an option is missing The help screen provides comprehensive documentation of all available options, organized into sections: - Basic options (e.g. bindir, test-dir) - Output modes (e.g. appveyor, travis, teamcity) - Test prefix usage explanation Fixes: shader-slang#6560 Co-authored-by: Ellie Hermaszewska <ellieh@nvidia.com>
1 parent 18e6611 commit 3357b55

File tree

3 files changed

+171
-140
lines changed

3 files changed

+171
-140
lines changed

tools/slang-test/README.md

+108-140
Original file line numberDiff line numberDiff line change
@@ -1,157 +1,125 @@
11
# Slang Test
22

3-
Slang Test is a command line tool that is used to coordinate tests via other tools. The actual executable is 'slang-test'. It is typically run from the test.bat script in the root directory of the project.
3+
Slang Test (`slang-test`) is a command-line tool that coordinates and runs the Slang test suite. It acts as a test runner hub, executing various types of tests and collecting their results.
44

5-
Slang Test can be thought of as the 'hub' running multiple tests and accumulating the results. In the distribution tests are held in the 'tests' directory. Inside this directory there are tests grouped together via other directories. Inside those directories are the actual tests themselves. The tests exist as .hlsl, .slang and .glsl and other file extensions. The top line of each of these files describe what kind of test will be performed with a specialized comment '//TEST'.
5+
## Basic Usage
66

7-
Most command line options are prefixed by - for both switches and parameter options. On the command line you can specify a 'free parameter', and this value specifies the prefix of the name of the test to run. Note that such a prefix includes the path, with each directory separated via '/'.
8-
9-
An example command line:
10-
11-
```
12-
slang-test -bindir E:\slang\build\Debug\bin -category full tests/compute/array-param
7+
```bash
8+
slang-test [options] [test-prefix...]
139
```
1410

15-
* The -bindir value means that the tools slang-test will use the binaries found in this directory.
16-
* The -category full means that all tests can be run.
17-
* The final 'free parameter' is 'tests/compute/array-param' and means only tests starting with this string will run.
18-
19-
Most types of test use 'test tools' to implement actual tests. There are currently 3 'tools' that are typically used
20-
21-
* slangc
22-
* render-test
23-
* slang-reflection-test
24-
25-
These are typically implemeted as dlls/shared libraries that are loaded when a test is needed. Sometimes it is necessary or useful to just call one of these test tools directly with the parameters the tool takes. This can be achieved by giving the tool as a 'sub command' name on the command line. All of the parameters after the tool name will be passed directly to the tool. For example
11+
If no test prefix is specified, all tests will be run. Test prefixes can be used to filter which tests to run, and include the path with directories separated by '/'.
2612

13+
Example:
14+
```bash
15+
slang-test -bindir path/to/bin -category full tests/compute/array-param
2716
```
28-
slang-test -bindir E:\slang\build\Debug\bin slangc tests/compute/array-param.slang
29-
```
30-
31-
Will run the 'slangc' tool with the parameters listed after 'slangc' on the command line. Any parameters before the sub command will be parsed as usual by slang-test, and if not applicable to invoking the tool will be ignored. bindir will be used for finding the tool directory. This is by design so that the sub command invocation can just be placed after the normal slang-test commands, and removed when no longer needed.
32-
33-
The command line can control which tests are run with a couple of switches
34-
35-
* -api - Overall controls over which apis will be tested against
36-
* -synthesizedTestApi - Controls which apis will have tests synthesized for them from other apis. Tests can be synthesized for dx12 and vulkan.
37-
38-
The parameter afterwards is an 'api expression' used to control which apis will or wont be used. This is somewhat like a mathematical expression with only + and - operations and api names. If the first operation is + or - it will be applied to whatever the default is, otherwise the defaults are ignored.
39-
40-
* vk - just for vulkan
41-
* +vk - Whatever the defaults are including vulkan
42-
* -dx12 - Whatever the defaults are excluding vulkan
43-
* all - for all apis, none - for no apis
44-
* all-vk - all apis but not vulkan (vk)
45-
* all-vk-dx12 - all apis but not vulkan (vk) or directx12 (dx12)
46-
* gl+dx11 - just on opengl (gl) and directx11 (dx11)
47-
* +none - same as defaults
4817

49-
So if you wanted to test for all apis, except opengl you'd put on the command line '-api all-gl'
18+
## Command Line Options
19+
20+
### Core Options
21+
- `-h, --help`: Show help message
22+
- `-bindir <path>`: Set directory for binaries (default: the path to the slang-test executable)
23+
- `-test-dir <path>`: Set directory for test files (default: tests/)
24+
- `-v`: Enable verbose output
25+
- `-verbose-paths`: Use verbose paths in output
26+
- `-hide-ignored`: Hide results from ignored tests
27+
28+
### Test Selection and Categories
29+
- `-category <name>`: Only run tests in specified category
30+
- `-exclude <name>`: Exclude tests in specified category
31+
32+
Available test categories:
33+
- `full`: All tests
34+
- `quick`: Quick tests
35+
- `smoke`: Basic smoke tests
36+
- `render`: Rendering-related tests
37+
- `compute`: Compute shader tests
38+
- `vulkan`: Vulkan-specific tests
39+
- `compatibility-issue`: Tests for compatibility issues
40+
41+
A test may be in one or more categories. The categories are specified on top of a test, for example: //TEST(smoke,compute):COMPARE_COMPUTE:
42+
43+
### API Control Options
44+
- `-api <expr>`: Enable specific APIs (e.g., 'vk+dx12' or '+dx11')
45+
- `-api-only`: Only run tests that use specified APIs
46+
- `-synthesizedTestApi <expr>`: Set APIs for synthesized tests
47+
- `-skip-api-detection`: Skip API availability detection
48+
49+
API expression syntax:
50+
- Use `+` or `-` to add or remove APIs from defaults
51+
- Examples:
52+
- `vk`: Vulkan only
53+
- `+vk`: Add Vulkan to defaults
54+
- `-dx12`: Remove DirectX 12 from defaults
55+
- `all`: All APIs
56+
- `all-vk`: All APIs except Vulkan
57+
- `gl+dx11`: Only OpenGL and DirectX 11
58+
59+
Available APIs:
60+
- OpenGL: `gl`, `ogl`, `opengl`
61+
- Vulkan: `vk`, `vulkan`
62+
- DirectX 12: `dx12`, `d3d12`
63+
- DirectX 11: `dx11`, `d3d11`
64+
65+
### Test Execution Options
66+
- `-server-count <n>`: Set number of test servers (default: 1)
67+
- `-use-shared-library`: Run tests in-process using shared library
68+
- `-use-test-server`: Run tests using test server
69+
- `-use-fully-isolated-test-server`: Run each test in isolated server
70+
71+
### Output Options
72+
- `-appveyor`: Use AppVeyor output format
73+
- `-travis`: Use Travis CI output format
74+
- `-teamcity`: Use TeamCity output format
75+
- `-xunit`: Use xUnit output format
76+
- `-xunit2`: Use xUnit 2 output format
77+
- `-show-adapter-info`: Show detailed adapter information
78+
79+
### Other Options
80+
- `-generate-hlsl-baselines`: Generate HLSL test baselines
81+
- `-emit-spirv-via-glsl`: Emit SPIR-V through GLSL instead of directly
82+
- `-expected-failure-list <file>`: Specify file containing expected failures
5083

51-
The different APIs are
52-
53-
* OpenGL - gl,ogl,opengl
54-
* Vulkan - vk,vulkan
55-
* DirectD3D12 - dx12,d3d12
56-
* DirectD3D11 -dx11,d3d11
57-
58-
59-
It may also be necessary to have the working directory the root directory of the slang distribution - in the example above this would be "E:\slang\".
60-
61-
## Unit tests
62-
63-
In addition to the above test tools, there is also `slang-unit-test-tool`, which is invoked as in the following example.
84+
## Test Types
6485

65-
```
86+
Tests are identified by a special comment at the start of the test file: `//TEST:<type>:`
87+
88+
Available test types:
89+
- `SIMPLE`: Runs the slangc compiler with specified options after the command
90+
- `REFLECTION`: Runs slang-reflection-test with the options specified after the command
91+
- `COMPARE_COMPUTE`: Runs render-test to execute a compute shader and writes the result to a text file. The test passes if the output matches the expected content
92+
- `COMPARE_COMPUTE_EX`: Same as COMPARE_COMPUTE, but supports additional parameter specifications
93+
- `COMPARE_RENDER_COMPUTE`: Runs render-test with "-slang -gcompute" options and compares text file outputs
94+
- `LANG_SERVER`: Tests Language Server Protocol features by sending requests (like completion, hover, signatures) and comparing responses with expected outputs
95+
96+
Deprecated test types (do not create new tests of these kinds, and we need to slowly migrate existing tests to use SIMPLE, COMPARE_COMPUTE(_EX) or COMPARE_RENDER_COMPUTE instead):
97+
- `COMPARE_HLSL`: Runs the slangc compiler with forced DXBC output and compares with a file having the '.expected' extension
98+
- `COMPARE_HLSL_RENDER`: Runs render-test to generate two images - one using HLSL (expected) and one using Slang, saving both as .png files. The test passes if the images match
99+
- `COMPARE_HLSL_CROSS_COMPILE_RENDER`: Runs render-test to generate two images - one using Slang and one using -glsl-cross. The test passes if the images match
100+
- `COMPARE_HLSL_GLSL_RENDER`: Runs render-test to generate two images - one using -hlsl-rewrite and one using -glsl-rewrite. The test passes if the images match
101+
- `COMPARE_GLSL`: Runs the slangc compiler both through Slang and directly, then compares the SPIR-V assembly output
102+
- `HLSL_COMPUTE`: Runs render-test with "-hlsl-rewrite -compute" options and compares text file outputs
103+
- `CROSS_COMPILE`: Compiles using GLSL pass-through and through Slang, then compares the outputs
104+
105+
## Unit Tests
106+
In addition to the above test tools, there are also `slang-unit-test-tool` and `gfx-unit-test-tool`, which are invoked as in the following examples.
107+
108+
### slang-unit-test-tool
109+
```bash
110+
# Regular unit tests
111+
slang-test slang-unit-test-tool/<test-name>
112+
# e.g. run the `byteEncode` test.
66113
slang-test slang-unit-test-tool/byteEncode
67114
```
115+
These tests are located in the [tools/slang-unit-test](https://github.com/shader-slang/slang/tree/master/tools/slang-unit-test) directory, and defined with macros like `SLANG_UNIT_TEST(byteEncode)`.
68116

69-
This will run the `byteEncode` test. The unit tests are located in the [tools/slang-unit-test](https://github.com/shader-slang/slang/tree/master/tools/slang-unit-test) directory, and defined with macros like `SLANG_UNIT_TEST(byteEncode)`.
70-
71-
There are also graphics unit tests, which are run as follows:
117+
### gfx-unit-test-tool
118+
```bash
119+
# Graphics unit tests
120+
slang-test gfx-unit-test-tool/<test-name>
72121

73-
```
122+
# e.g. run the `precompiledTargetModule2Vulkan` test.
74123
slang-test gfx-unit-test-tool/precompiledTargetModule2Vulkan
75124
```
76-
77-
This will run the `precompiledTargetModule2Vulkan` test. These tests are located in [tools/gfx-unit-test](https://github.com/shader-slang/slang/tree/master/tools/gfx-unit-test), and likewise defined using macros like `SLANG_UNIT_TEST(precompiledTargetModule2Vulkan)`.
78-
79-
## Test Categories
80-
81-
There are the following test categories
82-
83-
* full
84-
* quick
85-
* smoke
86-
* render
87-
* compute
88-
* vulkan
89-
* compatibility-issue
90-
91-
A test may be in one or more categories. The categories are specified in the test line, for example:
92-
//TEST(smoke,compute):COMPARE_COMPUTE:
93-
94-
## Command line options
95-
96-
### bindir
97-
98-
Specifies the directory where executables will be found.
99-
100-
Eg -bindir "path/to/slang/output/directory/bin"
101-
102-
### category
103-
104-
The parameter controls what kinds of tests will be run. Categories are listed at the 'test categories' section.
105-
106-
### exclude
107-
108-
Used to specify categories to be excluded during a test.
109-
110-
### appveyor
111-
112-
A flag that makes output suitable for the appveyor automated test suite.
113-
114-
### travis
115-
116-
A flag that makes output suitable for the travis automated test suite.
117-
118-
### Other Command Line Options
119-
120-
The following flags/paramteres can be passed but will be ignored by the tool
121-
122-
* debug (flag)
123-
* release (flag)
124-
* platform (parameter)
125-
126-
## Test Types
127-
128-
Test types are controlled via a comment at the top of a test file, that starts with //TEST:
129-
This is then immediately followed by the test type which is one of the following
130-
131-
* SIMPLE
132-
* Calls the slangc compiler with options after the comment
133-
* REFLECTION
134-
* Runs 'slang-reflection-test' passing in the options as given after the command
135-
* COMPARE_HLSL
136-
* Runs the slangc compiler, forcing dxbc output and compares with file post fixed with '.expected'
137-
* COMPARE_HLSL_RENDER
138-
* Runs 'render-test' rendering two images - one for hlsl (expected), and one for slang saving to .png files. The images must match for the test to pass.
139-
* COMPARE_HLSL_CROSS_COMPILE_RENDER
140-
* Runs 'render-test' rendering two images - one from slag, and the other -glsl-cross. The images must match for the test to pass.
141-
* COMPARE_HLSL_GLSL_RENDER
142-
* Runs 'render-test' rendering two images - one with -hlsl-rewrite and the other -glsl-rewrite. The images must match for test to pass.
143-
* COMPARE_COMPUTE
144-
* Runs 'render-test' producing a compute result written as a text file. Text file contents must be identical.
145-
* COMPARE_COMPUTE_EX
146-
* Same as COMPARE_COMPUTE, but allows specific parameters to be specified.
147-
* HLSL_COMPUTE
148-
* Runs 'render-test' with "-hlsl-rewrite -compute" options. Text files are compared.
149-
* COMPARE_RENDER_COMPUTE
150-
* Runs 'render-test' with "-slang -gcompute" options. Text files are compared.
151-
* COMPARE_GLSL
152-
* Runs the slangc compiler compiling through slang, and without and comparing output in spirv assembly.
153-
* CROSS_COMPILE
154-
* Compiles as glsl pass through and then through slang and comparing output
155-
* EVAL
156-
* Runs 'slang-eval-test' - which runs code on slang VM
157-
125+
These tests are located in [tools/gfx-unit-test](https://github.com/shader-slang/slang/tree/master/tools/gfx-unit-test), and likewise defined using macros like `SLANG_UNIT_TEST(precompiledTargetModule2Vulkan)`.

0 commit comments

Comments
 (0)