Add a pass to support resource return values (shader-slang#1537)

A long-standing problem for the Slang implementation has been that some targets (notably GLSL/SPIR-V) do not support treating resources (textures, buffers, samplers, etc.) as first-class types. Resource types on such platforms are restricted so that they may not be used as the type of:

1. fields of aggregate types (`struct`s)
2. local variables
3. function results or `out`/`inout` parameters

Issue (1) is handled by our "type legalization" pass today, by splitting aggregates that contain resources into separate fields/variables/parameters. Issue (2) is worked around by putting code into SSA form and promoting local variables to SSA temporaries when possible; the net result is that many local variables of texture type are eliminated (that pass is not perfect, though, and it is possible for users to get errors when it doesn't fully clean up local variables of texture type).

Issue (3) is a much more complicated matter, and it is what this change is concerned with.

A typical solution to issue (3) is to simply inline all of the code in a program, at which point function results and `out`/`inout` parameters will no longer exist to cause problems. We reject such solutions for two reasons. First, there are limitations on control-flow structure in HLSL/GLSL/SPIR-V that mean they cannot express certain programs after inlining has been performed. Second, and more importantly, the philosophy of the Slang compiler is to perform as little duplication of code as possible, so that we do not accidentally contribute to binary size bloat.

Instead, this change tackles the problem of functions that output resource types by adding a new specialization pass. The pass detects functions that ought to be specialized (because they have resource-type outputs), and inspects their bodies to see if the values they output have a predicatable structure that can be replicated outside of the function body. The same logic that inspects the function body also rewrites (a copy of) the function to not have the offending outputs. Finally, all the call sites to a function that is rewritten in this way also get rewritten so that instead of using output values from the function itself, they reproduce the expected output value(s) in their own code.

The pass as presented here is intentionally limited in the scope of what it can optimize away (and the test case only touches on that specific functionality). The goal is to get a basic version of this pass in place and evaluated, and then to expand on its functionality incrementally over time.

Author: Tim Foley

source/slang/slang-emit.cpp

@@ -431,16 +431,21 @@ Result linkAndOptimizeIR(
     //
     // Many of our targets place restrictions on how certain
     // resource types can be used, so that having them as
-    // function parameters is invalid. To clean this up,
-    // we will try to specialize called functions based
-    // on the actual resources that are being passed to them
-    // at specific call sites.
+    // function parameters, reults, etc. is invalid.
+    // To clean this up, we apply two kinds of specialization:
+    //
+    // * Specalize call sites based on the actual resources
+    //   that a called function will return/output.
+    //
+    // * Specialize called functions based on teh actual resources
+    //   passed ass input at specific call sites.
     //
     // Because the legalization may depend on what target
     // we are compiling for (certain things might be okay
     // for D3D targets that are not okay for Vulkan), we
     // pass down the target request along with the IR.
     //
+    specializeResourceOutputs(compileRequest, targetRequest, irModule);
     specializeResourceParameters(compileRequest, targetRequest, irModule);
 
     // For GLSL targets, we also want to specialize calls to functions that