Skip to content

Commit 723796a

Browse files
jsmall-zzzTim Foley
jsmall-zzz
and
Tim Foley
authored
LZ4 compression support (shader-slang#1654)
* #include an absolute path didn't work - because paths were taken to always be relative. * Testing out use of lz4. * Added ICompressionSystem, and LZ4 implementation. * Add support for deflate compression. Simplify compression interface - to make more easily work across apis. * WIP on CompressedFileSystem. * ImplicitDirectoryCollector * SubStringIndexMap - > StringSliceIndexMap. * WIP save stdlib in different containers. * Support for different archive types for stdlib. * Fix project. * CompressedFileSystem -> ArchiveFileSystem. Added CompressionSystemType::None * Added ArchiveFileSystem * Fix problem RiffFileSystem load withoug compression system. * Test archive types. Improve diagnostic message. * Fix typo in testing file system archives. * Split out archive detection. * Fix gcc warning issue. * Fix warning. * RiffArchiveFileSystem -> RiffFileSystem Co-authored-by: Tim Foley <tfoleyNV@users.noreply.github.com>
1 parent 5554777 commit 723796a

32 files changed

+1652
-334
lines changed

‎.gitmodules

+3
Original file line numberDiff line numberDiff line change
@@ -22,3 +22,6 @@
2222
[submodule "external/miniz"]
2323
path = external/miniz
2424
url = https://github.com/richgel999/miniz
25+
[submodule "external/lz4"]
26+
path = external/lz4
27+
url = https://github.com/lz4/lz4

‎build/visual-studio/core/core.vcxproj

+9
Original file line numberDiff line numberDiff line change
@@ -171,13 +171,16 @@
171171
</ItemDefinitionGroup>
172172
<ItemGroup>
173173
<ClInclude Include="..\..\..\source\core\slang-allocator.h" />
174+
<ClInclude Include="..\..\..\source\core\slang-archive-file-system.h" />
174175
<ClInclude Include="..\..\..\source\core\slang-array-view.h" />
175176
<ClInclude Include="..\..\..\source\core\slang-array.h" />
176177
<ClInclude Include="..\..\..\source\core\slang-basic.h" />
177178
<ClInclude Include="..\..\..\source\core\slang-blob.h" />
178179
<ClInclude Include="..\..\..\source\core\slang-byte-encode-util.h" />
179180
<ClInclude Include="..\..\..\source\core\slang-char-util.h" />
180181
<ClInclude Include="..\..\..\source\core\slang-common.h" />
182+
<ClInclude Include="..\..\..\source\core\slang-compression-system.h" />
183+
<ClInclude Include="..\..\..\source\core\slang-deflate-compression-system.h" />
181184
<ClInclude Include="..\..\..\source\core\slang-dictionary.h" />
182185
<ClInclude Include="..\..\..\source\core\slang-downstream-compiler.h" />
183186
<ClInclude Include="..\..\..\source\core\slang-exception.h" />
@@ -188,6 +191,7 @@
188191
<ClInclude Include="..\..\..\source\core\slang-io.h" />
189192
<ClInclude Include="..\..\..\source\core\slang-linked-list.h" />
190193
<ClInclude Include="..\..\..\source\core\slang-list.h" />
194+
<ClInclude Include="..\..\..\source\core\slang-lz4-compression-system.h" />
191195
<ClInclude Include="..\..\..\source\core\slang-math.h" />
192196
<ClInclude Include="..\..\..\source\core\slang-memory-arena.h" />
193197
<ClInclude Include="..\..\..\source\core\slang-name-convention-util.h" />
@@ -197,6 +201,7 @@
197201
<ClInclude Include="..\..\..\source\core\slang-process-util.h" />
198202
<ClInclude Include="..\..\..\source\core\slang-random-generator.h" />
199203
<ClInclude Include="..\..\..\source\core\slang-render-api-util.h" />
204+
<ClInclude Include="..\..\..\source\core\slang-riff-file-system.h" />
200205
<ClInclude Include="..\..\..\source\core\slang-riff.h" />
201206
<ClInclude Include="..\..\..\source\core\slang-secure-crt.h" />
202207
<ClInclude Include="..\..\..\source\core\slang-semantic-version.h" />
@@ -220,21 +225,25 @@
220225
<ClInclude Include="..\..\..\source\core\windows\slang-win-visual-studio-util.h" />
221226
</ItemGroup>
222227
<ItemGroup>
228+
<ClCompile Include="..\..\..\source\core\slang-archive-file-system.cpp" />
223229
<ClCompile Include="..\..\..\source\core\slang-blob.cpp" />
224230
<ClCompile Include="..\..\..\source\core\slang-byte-encode-util.cpp" />
225231
<ClCompile Include="..\..\..\source\core\slang-char-util.cpp" />
232+
<ClCompile Include="..\..\..\source\core\slang-deflate-compression-system.cpp" />
226233
<ClCompile Include="..\..\..\source\core\slang-downstream-compiler.cpp" />
227234
<ClCompile Include="..\..\..\source\core\slang-free-list.cpp" />
228235
<ClCompile Include="..\..\..\source\core\slang-gcc-compiler-util.cpp" />
229236
<ClCompile Include="..\..\..\source\core\slang-hex-dump-util.cpp" />
230237
<ClCompile Include="..\..\..\source\core\slang-io.cpp" />
238+
<ClCompile Include="..\..\..\source\core\slang-lz4-compression-system.cpp" />
231239
<ClCompile Include="..\..\..\source\core\slang-memory-arena.cpp" />
232240
<ClCompile Include="..\..\..\source\core\slang-name-convention-util.cpp" />
233241
<ClCompile Include="..\..\..\source\core\slang-nvrtc-compiler.cpp" />
234242
<ClCompile Include="..\..\..\source\core\slang-offset-container.cpp" />
235243
<ClCompile Include="..\..\..\source\core\slang-platform.cpp" />
236244
<ClCompile Include="..\..\..\source\core\slang-random-generator.cpp" />
237245
<ClCompile Include="..\..\..\source\core\slang-render-api-util.cpp" />
246+
<ClCompile Include="..\..\..\source\core\slang-riff-file-system.cpp" />
238247
<ClCompile Include="..\..\..\source\core\slang-riff.cpp" />
239248
<ClCompile Include="..\..\..\source\core\slang-semantic-version.cpp" />
240249
<ClCompile Include="..\..\..\source\core\slang-shared-library.cpp" />

‎build/visual-studio/core/core.vcxproj.filters

+27
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,9 @@
1212
<ClInclude Include="..\..\..\source\core\slang-allocator.h">
1313
<Filter>Header Files</Filter>
1414
</ClInclude>
15+
<ClInclude Include="..\..\..\source\core\slang-archive-file-system.h">
16+
<Filter>Header Files</Filter>
17+
</ClInclude>
1518
<ClInclude Include="..\..\..\source\core\slang-array-view.h">
1619
<Filter>Header Files</Filter>
1720
</ClInclude>
@@ -33,6 +36,12 @@
3336
<ClInclude Include="..\..\..\source\core\slang-common.h">
3437
<Filter>Header Files</Filter>
3538
</ClInclude>
39+
<ClInclude Include="..\..\..\source\core\slang-compression-system.h">
40+
<Filter>Header Files</Filter>
41+
</ClInclude>
42+
<ClInclude Include="..\..\..\source\core\slang-deflate-compression-system.h">
43+
<Filter>Header Files</Filter>
44+
</ClInclude>
3645
<ClInclude Include="..\..\..\source\core\slang-dictionary.h">
3746
<Filter>Header Files</Filter>
3847
</ClInclude>
@@ -63,6 +72,9 @@
6372
<ClInclude Include="..\..\..\source\core\slang-list.h">
6473
<Filter>Header Files</Filter>
6574
</ClInclude>
75+
<ClInclude Include="..\..\..\source\core\slang-lz4-compression-system.h">
76+
<Filter>Header Files</Filter>
77+
</ClInclude>
6678
<ClInclude Include="..\..\..\source\core\slang-math.h">
6779
<Filter>Header Files</Filter>
6880
</ClInclude>
@@ -90,6 +102,9 @@
90102
<ClInclude Include="..\..\..\source\core\slang-render-api-util.h">
91103
<Filter>Header Files</Filter>
92104
</ClInclude>
105+
<ClInclude Include="..\..\..\source\core\slang-riff-file-system.h">
106+
<Filter>Header Files</Filter>
107+
</ClInclude>
93108
<ClInclude Include="..\..\..\source\core\slang-riff.h">
94109
<Filter>Header Files</Filter>
95110
</ClInclude>
@@ -155,6 +170,9 @@
155170
</ClInclude>
156171
</ItemGroup>
157172
<ItemGroup>
173+
<ClCompile Include="..\..\..\source\core\slang-archive-file-system.cpp">
174+
<Filter>Source Files</Filter>
175+
</ClCompile>
158176
<ClCompile Include="..\..\..\source\core\slang-blob.cpp">
159177
<Filter>Source Files</Filter>
160178
</ClCompile>
@@ -164,6 +182,9 @@
164182
<ClCompile Include="..\..\..\source\core\slang-char-util.cpp">
165183
<Filter>Source Files</Filter>
166184
</ClCompile>
185+
<ClCompile Include="..\..\..\source\core\slang-deflate-compression-system.cpp">
186+
<Filter>Source Files</Filter>
187+
</ClCompile>
167188
<ClCompile Include="..\..\..\source\core\slang-downstream-compiler.cpp">
168189
<Filter>Source Files</Filter>
169190
</ClCompile>
@@ -179,6 +200,9 @@
179200
<ClCompile Include="..\..\..\source\core\slang-io.cpp">
180201
<Filter>Source Files</Filter>
181202
</ClCompile>
203+
<ClCompile Include="..\..\..\source\core\slang-lz4-compression-system.cpp">
204+
<Filter>Source Files</Filter>
205+
</ClCompile>
182206
<ClCompile Include="..\..\..\source\core\slang-memory-arena.cpp">
183207
<Filter>Source Files</Filter>
184208
</ClCompile>
@@ -200,6 +224,9 @@
200224
<ClCompile Include="..\..\..\source\core\slang-render-api-util.cpp">
201225
<Filter>Source Files</Filter>
202226
</ClCompile>
227+
<ClCompile Include="..\..\..\source\core\slang-riff-file-system.cpp">
228+
<Filter>Source Files</Filter>
229+
</ClCompile>
203230
<ClCompile Include="..\..\..\source\core\slang-riff.cpp">
204231
<Filter>Source Files</Filter>
205232
</ClCompile>

‎build/visual-studio/slang-test/slang-test.vcxproj

+3
Original file line numberDiff line numberDiff line change
@@ -196,6 +196,9 @@
196196
<ProjectReference Include="..\miniz\miniz.vcxproj">
197197
<Project>{E76ACB11-4A12-4F0A-BE1E-CE0B8836EB7F}</Project>
198198
</ProjectReference>
199+
<ProjectReference Include="..\lz4\lz4.vcxproj">
200+
<Project>{E1EC8075-823E-46E5-BC38-C124CCCDF878}</Project>
201+
</ProjectReference>
199202
</ItemGroup>
200203
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
201204
<ImportGroup Label="ExtensionTargets">

‎build/visual-studio/slang/slang.vcxproj

+3
Original file line numberDiff line numberDiff line change
@@ -443,6 +443,9 @@
443443
<ProjectReference Include="..\miniz\miniz.vcxproj">
444444
<Project>{E76ACB11-4A12-4F0A-BE1E-CE0B8836EB7F}</Project>
445445
</ProjectReference>
446+
<ProjectReference Include="..\lz4\lz4.vcxproj">
447+
<Project>{E1EC8075-823E-46E5-BC38-C124CCCDF878}</Project>
448+
</ProjectReference>
446449
</ItemGroup>
447450
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
448451
<ImportGroup Label="ExtensionTargets">

‎external/lz4

Submodule lz4 added at d443718

‎premake5.lua

+19-5
Original file line numberDiff line numberDiff line change
@@ -658,7 +658,7 @@ tool "slang-embed"
658658
tool "slang-test"
659659
uuid "0C768A18-1D25-4000-9F37-DA5FE99E3B64"
660660
includedirs { "." }
661-
links { "core", "slang", "miniz" }
661+
links { "core", "slang", "miniz", "lz4" }
662662

663663
-- We want to set to the root of the project, but that doesn't seem to work with '.'.
664664
-- So set a path that resolves to the same place.
@@ -982,7 +982,7 @@ if enableEmbedStdLib then
982982
standardProject("slangc-bootstrap", "source/slangc")
983983
uuid "6339BF31-AC99-4819-B719-679B63451EF0"
984984
kind "ConsoleApp"
985-
links { "core", "miniz" }
985+
links { "core", "miniz", "lz4" }
986986

987987
-- We need to run all the generators to be able to build the main
988988
-- slang source in source/slang
@@ -1057,7 +1057,7 @@ if enableEmbedStdLib then
10571057

10581058
buildinputs { "%{cfg.targetdir}/slangc-bootstrap" .. executableSuffix }
10591059

1060-
local buildcmd = '"%{cfg.targetdir}/slangc-bootstrap" -save-stdlib-bin-source %{file.directory}/slang-stdlib-generated.h'
1060+
local buildcmd = '"%{cfg.targetdir}/slangc-bootstrap" -archive-type riff-lz4 -save-stdlib-bin-source %{file.directory}/slang-stdlib-generated.h'
10611061

10621062
buildcommands { buildcmd }
10631063
end
@@ -1082,7 +1082,7 @@ end
10821082
standardProject("slang", "source/slang")
10831083
uuid "DB00DA62-0533-4AFD-B59F-A67D5B3A0808"
10841084
kind "SharedLib"
1085-
links { "core", "miniz"}
1085+
links { "core", "miniz", "lz4"}
10861086
warnings "Extra"
10871087
flags { "FatalWarnings" }
10881088
pic "On"
@@ -1183,7 +1183,7 @@ if enableProfile then
11831183
addSourceDir "source/slang"
11841184

11851185
includedirs { "." }
1186-
links { "core", "miniz"}
1186+
links { "core", "miniz", "lz4"}
11871187

11881188
filter { "system:linux" }
11891189
linkoptions{ "-pg" }
@@ -1208,6 +1208,20 @@ standardProject("miniz", nil)
12081208
filter { "system:linux or macosx" }
12091209
links { "dl"}
12101210

1211+
standardProject("lz4", nil)
1212+
uuid "E1EC8075-823E-46E5-BC38-C124CCCDF878"
1213+
kind "StaticLib"
1214+
pic "On"
1215+
1216+
-- Add the files explicitly
1217+
files
1218+
{
1219+
"external/lz4/lib/lz4.c",
1220+
"external/lz4/lib/lz4.h",
1221+
}
1222+
1223+
filter { "system:linux or macosx" }
1224+
links { "dl"}
12111225

12121226
if buildGlslang then
12131227

‎slang.h

+16-2
Original file line numberDiff line numberDiff line change
@@ -477,7 +477,8 @@ extern "C"
477477
#endif
478478

479479
typedef bool SlangBool;
480-
480+
481+
481482
/*!
482483
@brief Severity of a diagnostic generated by the compiler.
483484
Values come from the enum below, with higher values representing more severe
@@ -558,6 +559,18 @@ extern "C"
558559
SLANG_PASS_THROUGH_COUNT_OF,
559560
};
560561

562+
/* Defines an archive type used to holds a 'file system' type structure. */
563+
typedef int SlangArchiveTypeIntegral;
564+
enum SlangArchiveType : SlangArchiveTypeIntegral
565+
{
566+
SLANG_ARCHIVE_TYPE_UNDEFINED,
567+
SLANG_ARCHIVE_TYPE_ZIP,
568+
SLANG_ARCHIVE_TYPE_RIFF, ///< Riff container with no compression
569+
SLANG_ARCHIVE_TYPE_RIFF_DEFLATE,
570+
SLANG_ARCHIVE_TYPE_RIFF_LZ4,
571+
SLANG_ARCHIVE_TYPE_COUNT_OF,
572+
};
573+
561574
/*!
562575
Flags to control compilation behavior.
563576
*/
@@ -3125,10 +3138,11 @@ namespace slang
31253138
virtual SLANG_NO_THROW SlangResult SLANG_MCALL loadStdLib(const void* stdLib, size_t stdLibSizeInBytes) = 0;
31263139

31273140
/** Save the StdLib modules to the file system
3141+
@param archiveType The type of archive used to hold the stdlib
31283142
@param outBlob The serialized blob containing the standard library
31293143
31303144
NOTE! API is experimental and not ready for production code */
3131-
virtual SLANG_NO_THROW SlangResult SLANG_MCALL saveStdLib(ISlangBlob** outBlob) = 0;
3145+
virtual SLANG_NO_THROW SlangResult SLANG_MCALL saveStdLib(SlangArchiveType archiveType, ISlangBlob** outBlob) = 0;
31323146

31333147
/** Look up the internal ID of a capability by its `name`.
31343148

‎slang.sln

+10
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,8 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "run-generators", "build\vis
2626
{7F773DD9-EB8F-2403-B43C-B49C2014B99C} = {7F773DD9-EB8F-2403-B43C-B49C2014B99C}
2727
EndProjectSection
2828
EndProject
29+
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "lz4", "build\visual-studio\lz4\lz4.vcxproj", "{E1EC8075-823E-46E5-BC38-C124CCCDF878}"
30+
EndProject
2931
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "miniz", "build\visual-studio\miniz\miniz.vcxproj", "{E76ACB11-4A12-4F0A-BE1E-CE0B8836EB7F}"
3032
EndProject
3133
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "slang", "build\visual-studio\slang\slang.vcxproj", "{DB00DA62-0533-4AFD-B59F-A67D5B3A0808}"
@@ -127,6 +129,14 @@ Global
127129
{E145B2B8-CD13-A6BE-B6A7-16E5A2148223}.Release|Win32.Build.0 = Release|Win32
128130
{E145B2B8-CD13-A6BE-B6A7-16E5A2148223}.Release|x64.ActiveCfg = Release|x64
129131
{E145B2B8-CD13-A6BE-B6A7-16E5A2148223}.Release|x64.Build.0 = Release|x64
132+
{E1EC8075-823E-46E5-BC38-C124CCCDF878}.Debug|Win32.ActiveCfg = Debug|Win32
133+
{E1EC8075-823E-46E5-BC38-C124CCCDF878}.Debug|Win32.Build.0 = Debug|Win32
134+
{E1EC8075-823E-46E5-BC38-C124CCCDF878}.Debug|x64.ActiveCfg = Debug|x64
135+
{E1EC8075-823E-46E5-BC38-C124CCCDF878}.Debug|x64.Build.0 = Debug|x64
136+
{E1EC8075-823E-46E5-BC38-C124CCCDF878}.Release|Win32.ActiveCfg = Release|Win32
137+
{E1EC8075-823E-46E5-BC38-C124CCCDF878}.Release|Win32.Build.0 = Release|Win32
138+
{E1EC8075-823E-46E5-BC38-C124CCCDF878}.Release|x64.ActiveCfg = Release|x64
139+
{E1EC8075-823E-46E5-BC38-C124CCCDF878}.Release|x64.Build.0 = Release|x64
130140
{E76ACB11-4A12-4F0A-BE1E-CE0B8836EB7F}.Debug|Win32.ActiveCfg = Debug|Win32
131141
{E76ACB11-4A12-4F0A-BE1E-CE0B8836EB7F}.Debug|Win32.Build.0 = Debug|Win32
132142
{E76ACB11-4A12-4F0A-BE1E-CE0B8836EB7F}.Debug|x64.ActiveCfg = Debug|x64

0 commit comments

Comments
 (0)