Stacks: Values reference other units

[cgetzen]

The problem I am currently facing with the new Stacks RFC is that it is not possible to pass the output of one unit through to the values of another in the terragrunt.stack.hcl file.

While it is possible to do this in the unit definition, this is not extensible enough because, with stacks, the units will be reused with different inputs.

Proposal

Expand the terragrunt.values.hcl file to use dependencies:

# terragrunt.stack.hcl

unit "dep" {
  source = "modules/imadep"
  path   = "imadep"
}

unit "test" {
  source = "modules/test"
  path   = "test"
  values = {
    a = "a"
    b = unit.dep.output_b
  }
}


# .terragrunt-stack/test/terragrunt.values.hcl (auto-generated)

values {
    a = "a"
    b = dependency.dep.outputs.output_b
}

dependency "dep" {
    config_path = "../imadep"
}

There is a POC of this here: #4056

POC Q&A

Recommendation to use a new terragrunt.dependencies.hcl file
I'd be happy to POC this, but I'm curious how you see the relationship between the dependencies file, values file, and terragrunt file.

My current view is that consolodating all dynamic inputs to units in the terragrunt.values.hcl is optimal:

values {
    a = "a"
    b = dependency.dep.outputs.output_b
}

dependency "dep" {
    config_path = "../imadep"
}

Yes, it is more complex than just putting (static) key-value pairs. However:

1. the complexity has to go somewhere
2. Putting this inside of the values file may be the least complex

Why is this the least complex?

• I think it can clearly define interfaces and boundaries between components:terragrunt.values.hcl can be processed (i.e. dependencies resolved) and only export a single values interface. Processing would be self-contained to the terragrunt.values.hcl file, and terragrunt.hcl would only ever see the values object from it.
• I suspect there may be other constructs that users may want to add in the future. Splitting files by block types may lead to many files with different kinds of relationships between them.

What if two dependencies are named the same thing across these different files? What's the behavior? Can the terragrunt.hcl file make reference to them?
I think the correct choice is to make the terragrunt.values.hcl file hide it's internals and only expose only an interface for retrieving the values. This helps with composability (If there is a dependency in the values file with the same name as a dependency in the terragrunt file, they will not interfere).

Are there any alternative designs that you think would work just as well?
The constraint to keep unit definitions backwards compatible (which I agree with!) reduces the solution space. I haven't been able to come up with another design.

How are mock outputs handled?
I touch on one option for this at the bottom of this comment #3313 (comment), but there are benefits to not implementing this: it trades a simple interface for flexibility. However, with additional changes it may be possible to get the flexibility of mocks with the benefits of a simple interface, and it boils down to where the mocks are defined. Instead of defining them at the "dependency" block, mock outputs could be defined at either or both of (1) the referenced units (the unit would define it's own mock outputs), and (2) in a new mock_inputs block, which is decoupled from the dependency block. I think it makes sense for a module to define mock_outputs, because the author should know what format the output takes. At the same time, I can see the value of defining mock_inputs for certain use-cases (if the dependency's output is dynamic enough where a static mock_output isn't enough).

[yhakbar]

I'm curious how you see the relationship between the dependencies file, values file, and terragrunt file.

The way that we've approached extending the terragrunt.hcl file via the availability of the terragrunt.values.hcl file is that the values file is a simple data store, and a way to conveniently introduce dynamicity into units (in a way that is 100% compatible with everything users use to define units without stacks). Users can, for example chose to use zero terragrunt.stack.hcl files, and place terragrunt.values.hcl files next to terragrunt.hcl files, and they'll be respected regardless (although for now, we're gating them both behind the stacks experiment to make sure we can make breaking adjustments prior to general availability).

there are benefits to not implementing this: it trades a simple interface for flexibility.

Unfortunately, as OpenTofu works today, this is a bit of a non-starter. Mock outputs aren't just a convenience, but a requirement for the ability to run a plan against units with unapplied dependencies. OpenTofu has no way of allowing certain inputs to be unknown at the root module (values can only be unknown within state between resources), so we have to retain the ability to mock outputs. We've discussed the possibility of introducing an outputs block to move the responsibility for defining outputs on the unit exporting them, but that is likely too much of a shift to make prior to 1.0.

I think the correct choice is to make the terragrunt.values.hcl file hide it's internals and only expose only an interface for retrieving the values.

Dependencies aren't just a data fetching mechanism, but also a scheduling mechanism. One unit depending on another will result in scheduling changes in the DAG, so Terragrunt needs a holistic view of all dependencies when it's looking to schedule the run for a unit. It needs to be unambiguous what all the dependencies are for a unit, and conflict resolution needs to be resolved when there are conflicts between different dependencies named the same thing.

---

Generally, the design we've been going for with the new terragrunt.stack.hcl file is to approach from the perspective of the assumption that users will want to use different parts of the design for their projects. That's generally what we see in the wild with Terragrunt: People have different parts that they like, and adopt the features that matter the most to them.

Given that, would having the dependency block in the terragrunt.values.hcl file be the best design, if taken in isolation? Assume that someone manually wrote up a terragrunt.values.hcl file with no usage of a terragrunt.stack.hcl file. Would there be any benefit to configuring the dependency in the terragrunt.values.hcl file instead of directly in the terragrunt.hcl file? If there was any benefit, would users get the most benefit out of having it be in the terragrunt.values.hcl file?

[cgetzen]

@yhakbar Thank you for your engagement -- the terragrunt team is unique in how quickly it responds to the community.

BLUF: I have updated the PR so that mock_inputs can be defined, as well as accounting for the dependency graph. I have left the dependencies in the values file for now.

Mock outputs aren't just a convenience, but a requirement for the ability to run a plan against units with unapplied dependencies.

Got it. Here is the adjusted spec to accommodate mock outputs:

# terragrunt.stack.hcl

unit "dep" {
  source = "modules/imadep"
  path   = "imadep"
  mock_outputs = {
    output_b = "mock_b"
  }
}

unit "test" {
  source = "modules/test"
  path   = "test"
  values = {
    a = "a"
    b = unit.dep.output_b
  }
}


# .terragrunt-stack/test/terragrunt.values.hcl (auto-generated)

values {
    a = "a"
    b = dependency.dep.outputs.output_b
}

dependency "dep" {
    config_path = "../imadep"
    mock_outputs = {
      output_b = "mock_b"
    }
}

I have adjusted the PR to accommodate this.

Dependencies aren't just a data fetching mechanism, but also a scheduling mechanism. One unit depending on another will result in scheduling changes in the DAG, so Terragrunt needs a holistic view of all dependencies when it's looking to schedule the run for a unit.

Thank you for bearing with my naivety about terragrunt. This makes a lot of sense - for now, I have adjusted the PR so that any dependency across terragrunt.hcl or terragrunt.values.hcl is respected in the graph.

conflict resolution needs to be resolved when there are conflicts between different dependencies named the same thing.

I'm thinking about this a bit differently. If we continue to assume that dependencies in terragrunt.hcl cannot be used in terragrunt.values.hcl (and vice-versa), it's OK to have dependencies with the same name across these files.

Do you think this is a reasonable way to approach this?

I would be happy to change the implementation, but I currently believe the best interface is to have a single file contain the dynamic / injected data that the unit uses. Why?

1. There is a clear boundary between terragrunt.values.hcl and terragrunt.hcl: the file. There is a clear interface: the values object. Nothing else is shared from a configuration language perspective (even though the dependency graph covers both files).
2. Adding another file to specify just the dependencies may confuse users: does it apply to terragrunt.hcl? terragrunt.values.hcl? both? What is the benefit of having this file apart from the values file?
3. I (weakly) believe that keeping all of these configurations in the single values file will result in future-proofing, to allow for other kinds of blocks to be added.

Assume that someone manually wrote up a terragrunt.values.hcl file with no usage of a terragrunt.stack.hcl file. Would there be any benefit to configuring the dependency in the terragrunt.values.hcl file instead of directly in the terragrunt.hcl file?

I think the answer is the same to the question "Would there be any benefit to configuring the values in the terragrunt.values.hcl file instead of directly in the terragrunt.hcl file's inputs block?": There would be very little benefit, if any. However, adding dependency blocks to this file doesn't hurt users from creating their own values file (with or without dependencies).