Filter Flag

[yhakbar]

Summary

Add a new --filter flag to Terragrunt that allows users to filter units in commands. The value passed to the filter flag will be a flexible query string that allows for different types of filtering, including filtering for changes that have taken place across different git references.

The filter flag will provide some utilities that are important for Terragrunt:

1. Offer a standard mechanism for both including and excluding units from the run queue with one flag.
2. Offer a unified flag across multiple commands for unit targeting.
3. Offer git based change detection and action.
4. Allow for heterogenous runs in a single Terragrunt command.

Motivation

There’s a long-standing problem with Terragrunt orchestration in that it can be difficult to target updates to particular units if you aren’t very familiar with the tool.

Users have to use a combination of the some of the following:

• --queue-exclude-dir
• --queue-excludes-file
• --queue-exclude-external
• --queue-include-dir
• --queue-include-external
• --queue-include-units-including
• --queue-strict-include

Git inspection (and branch changes), careful control over the working directory and potentially multiple invocations of terragrunt to get the right units run.

This is especially tricky when using terragrunt.stack.hcl files and destroys, as those use-cases involve units that aren’t tracked in git, and units that are tracked in different refs, respectively.

Proposal

The syntax for the --filter query will allow for the targeting of units based on the following:

1. The name of units.
2. The path of units.
3. The Git references of units.

Each of these will have their own syntax.

Filtering on names

Filtering on the names of units will look like the following:

# plan any unit named `acme`
$ terragrunt run --filter='acme' plan

Using a valid directory name for a filter will communicate to Terragrunt that all units with that name should be included. If multiple units have the same name, they will all match the filter.

Filtering on paths

Filtering on paths of units will look like the following:

# plan the unit at relative path `./acme`
$ terragrunt run --filter='./acme' plan

# plan the unit at absolute path `/acme`
$ terragrunt run --filter='/acme' plan

# plan any units matching the glob `./acme/**/foo`
$ terragrunt run --filter='./acme/**/foo' plan

Any filter query that starts with a ./ or a / will be considered a path query. They will be evaluated as path globs to see which units match.

Path queries can also be wrapped with {}, which is required when combined with other queries. More on that later.

Filtering on Git references

Filtering on git references of units will look like the following:

$ terragrunt run --filter=[HEAD^1] plan
$ terragrunt run --filter=[a1b2c3d...e4f5g6h] plan
$ terragrunt run --filter=[trunk...feature-branch] plan

Any filter query that is surrounded by [] will be treated as a mechanism for targeting units by Git references. The contents of the [] can be any valid git reference, and using a ... between two references will indicate that all units in the diff between two references should match the filter. Specifying a single reference implies that the comparison is being done against the current state of the repository.

There will also be a special alias for [main...HEAD], which is --filter-affected, similar to [Turborepo](https://turbo.build/repo/docs/reference/run#--affected).

Filtering on Attributes

Filtering on configuration attributes would look like the following:

$ terragrunt run --filter='type=unit' plan
$ terragrunt run --filter='type=stack' plan
$ terragrunt run --filter='name=foo' plan

Attributes here are defined as special configuration that directly relates to different pieces of Terragrunt configuration (e.g. is this configuration for a unit or a stack?).

Every Terragrunt configuration will have some attributes applied to it, including the name, which can be explicitly targeted using the name= syntax. When no special configuration is detected (it’s not a path, git reference or attribute lookup), Terragrunt will assume that the query is for the name of the configuration.

This deviates from the design of Turborepo’s syntax, as they don’t need (or benefit from) this kind of filtering, but it’s very useful in Terragrunt.

Combining filters

Passing multiple --filter flags will indicate that a union of filters should apply, and combining multiple types of filter in one query will refine results.

# Plan all units named foo _or_ bar
$ terragrunt run --filter=foo --filter=bar plan

# Plan all units changed between `main` and `my-feature` _and_ named foo
$ terragrunt run --filter=[main...origin/my-feature]foo plan

Negating filters

Using a ! at the front of the filter will make it so that the filter is negated. Any matches will result in the unit being excluded.

# Plan all units in the ./prod directory _except_ foo
$ terragrunt run --filter=./prod/*** --filter=!foo plan

Note that any other query will automatically exclude by default, except for negated queries.

# This will _only_ run units in the ./prod directory
$ terragrunt run --filter=./prod plan

# This will run _everything_ except foo
$ terragrunt run --filter=!foo plan

Dependencies

Using a ... before or after a query will also match the dependents or dependencies of matched units, respectively.

# Plan foo _and_ any unit that is a dependent of foo (and any of its ancestor dependents).
$ terragrunt run --filter=...foo plan

# Plan baz _and_ any unit that is a dependency of baz (and any of its descendent dependencies).
$ terragrunt run --filter=baz... plan

For a concrete example with terragrunt.hcl files:

# foo/terragrunt.hcl
# Intentionally empty

# bar/terragrunt.hcl
dependency "foo" {
  config_path = "../foo"
}

# baz/terragrunt.hcl
dependency "bar" {
  config_path = "../bar"
}

When running the commands above, they will both pick up all the units.

In the first command, bar is a dependent of foo, and baz is a dependent of bar, so running terragrunt run --filter ...foo updates them all.

In the second command, bar is a dependency of baz, and foo is a dependency of bar, so running terragrunt run --filter baz... updates them all from the opposite direction.

Users can also use ^ to omit the target from selection with ....

# Plan any unit that is a dependent of foo (and any of its ancestor dependents).
$ terragrunt run --filter=...^foo plan
# But not foo.

# Plan any unit that is a dependency of baz (and any of its descendent dependencies).
$ terragrunt run --filter=^baz... plan
# But not baz.

Dependencies in this context also extend to “read files” as described by the [queue-include-units-reading](https://terragrunt.gruntwork.io/docs/reference/cli-options/#terragrunt-queue-include-units-reading) flag. If a path to a file is specified that is read by a unit, include that and its ancestors/descendents as well.

Git Interactions with destroy

Something that is relevant to Terragrunt, but not to Turborepo is the consideration that some operations have to run on Git references where units still exist, even if they’re removed in other references. This is the case with destroy commands.

Assuming a user ran the following:

$ terragrunt run --filter=[main...origin/my-feature] plan

And a unit was removed in the my-feature flag, you wouldn’t be able to perform a Terragrunt plan in the my-feature branch. Instead what needs to be done is a plan -destroy in the main branch for those units.

To support this, runs using the --filter flag will result in heterogeneous runs across matching units. Any unit that is found to be deleted between two commits will result in a -destroy flag being added to the run command performed by Terragrunt in the commit where the unit still exists.

Similarly, when running an apply:

$ terragrunt run --filter=[main...origin/my-feature] apply

Terragrunt will add a -destroy flag to the apply command to trigger a destroy in the unit.

Given that this can be surprising for users, it’s likely a good idea to add an additional flag to opt-in for destroys on applies. Users will receive a warning when they don’t use it during plans, and Terragrunt will exclude removed units from the run queue when the flag isn’t supplied. That flag will be named --filter-allow-destroy.

$ terragrunt run --filter=[main...origin/my-feature] plan
14:19:25.081 WARN [foo] The `foo` unit was removed in the `origin/my-feature` Git reference, but the `--filter-allow-destroy` flag was not used. The unit will be ignored during applies unless provided.

$ terragrunt run --filter=[main...origin/my-feature] --filter-allow-destroy plan

$ terragrunt run --filter=[main...origin/my-feature] apply
14:19:25.081 WARN [foo] The `foo` unit was removed in the `origin/my-feature` Git reference, but the `--filter-allow-destroy` flag was not used. The unit will be ignored during applies unless provided.

$ terragrunt run --filter=[main...origin/my-feature] --filter-allow-destroy apply

Interactions with stacks

The advent of the terragrunt.stack.hcl file complicates the implementation of the —-filter flag.

Most obviously, units defined by terragrunt.stack.hcl files usually won’t be tracked in the Git repository where the terragrunt.stack.hcl file lives. As a consequence, any addition, change or removal of a unit for a terragrunt.stack.hcl file won’t be detected as a diff by Git.

# live/terragrunt.stack.hcl
unit "foo" {
 source = "../units/foo"
 path   = "foo"
}

unit "bar" {
 source = "../units/bar"
 path   = "bar"
}

Removing “foo” wouldn’t result in the detection of a delete in git, so the unit would remain provisioned without being destroyed.

More subtly, units may be called something in a terragrunt.stack.hcl that should be picked up by the --filter flag, even when not using inspecting Git changes.

If a user used --filter foo, the terragrunt.stack.hcl file would have to be generated for the right ref for the unit to match.

For this reason, there’s special logic to address terragrunt.stack.hcl files.

When the --filter flag is used, it’s important that stack generation automatically take place in the current working directory. That will resolve the issue of users targeting the names of units that are generated by terragrunt.stack.hcl files.

When the --filter flag contains Git inspection (it uses []), matches for terragrunt.stack.hcl files will be evaluated.

• Adds can be processed using Git metadata, as the entire stack can be applied to generate the stack at a different reference.
• Deletes will require that a temporary Git worktree of the repository be created, and that the destroys take place there.
• Changes will require that one or two (if two references are being compared) Git worktrees be created all stacks generated within them, then the contents of the units in changed stacks between the worktree(s) are compared to determine what needs to be done.

# Assuming stack `./foo` is added, `./bar` is changed, and `./baz` is deleted
$ terragrunt run --filter=[main...origin/my-feature] apply

# Would effectively result in the following (with some concurrency, probably).

# 1. Run `terragrunt stack run apply` in the `foo` directory.
# 2. Run `git worktree add -f "$tmp_main" main` to get a temporary directory with the contents of `main`.
# 3. Run `terragrunt stack run destroy` in the `baz` directory.
# 4. Run `git worktree add -f "$tmp_my-feature" my-feature` to get a temporary directory with the contents of `my-feature`.
# 5. Run `terragrunt stack generate` in both "$tmp_main" and "$tmp_my-feature".
# 6. Determine list of changes for units in `bar`.
# 7. Run `terragrunt apply` for all units that were added or changed.
# 8. Run `terragrunt destroy` for all units that were deleted.

Interactions with read files

The [queue-include-units-reading](https://terragrunt.gruntwork.io/docs/reference/cli-options/#terragrunt-queue-include-units-reading) flag is handy, as it allows users to let Terragrunt which configurations read other files that have changed, and include them in the run queue.

With the introduction of the --filter flag, this can be a feature that happens automatically, meaning that users won’t need the flag anymore.

# Assuming `_envcommon/ec2.hcl` has changed, and `live/unit/terragrunt.hcl` included it.
$ terragrunt run --filter=[main...origin/my-feature] apply

# Would result in a `terragrunt apply` in `live/unit`.
# Note that this would _not_ be the case if `_envcommon/ec2.hcl` was created or destroyed.

Interactions with --all and --graph

The --all and --graph flags provide two ways to discover units for concurrent Terragrunt runs. The --all flag discovers all units child to the current working directory, while the --graph flag traverses the dependency graph to discover units to include in the run queue.

Most users use the --all flag when doing concurrent execution, so the --filter flag will assume the --all flag by default while constructing the run queue.

When users specify both the --all flag and the --filter flag, the --all flag will be ignored. When the --graph flag is provided, Terragrunt will instead use the --graph mechanism for constructing the run queue.

Filter file

To cover functionality that can be included by the [--excludes-file](https://terragrunt.gruntwork.io/docs/reference/cli-options/#terragrunt-excludes-file), there will also be a file which can be specified with --filters-file for configuration driven control over filters. The default value of this flag is .terragrunt-filters, and Terragrunt will look for it by default. When the file isn’t found, Terragrunt will silently ignore it if it is named .terragrunt-filters, and will throw an error otherwise. Users will be able to explicitly disable the filters file with --filters-disable.

Terragrunt will attempt to find the filters file in any of the current directory, or any parent directory until it reaches the root of a git repository if it is in one. If it is not, it will only look in the current directory. This is to avoid any issues that might arise recursively searching up to the root of the filesystem.

This file will be a newline delimited list of filters that match the same schema used for the --filter flag.

# .terragrunt-filters
foo
bar
[HEAD^]

$ terragrunt run --all apply

# Apply any unit named foo, bar or that has changed since the last commit.

Note that users will have to explicitly use the --all flag here to have the .terragrunt-filters file respected.

Interactions with the DAG

In the event that a heterogenous mix of runs are in the run queue (both applies and destroys from added and removed files), Terragrunt will use the DAG as it always does to sequence the order of operations, it will pull units from the run queue into the runner pool when they don’t have pending dependencies (in the case of applies) or they don’t have pending dependents (in the case of destroys).

If, for whatever reason, Terragrunt would perform a conflicting operation in a unit (both destroying and applying it) because of some combination of filters/flags, Terragrunt will throw an error and require disambiguation via a negation filter (leading !) to prevent it from matching.

In CI systems, this error can be detected, and presented as a notification that pull requests should be split into multiple pieces.

Outside the scope of this RFC, there may be a use-case for disambiguation to be performed via additional flags for the run queue (e.g. --queue-destroy-over-apply, --queue-destroy-before-apply, etc). This use-case would have to be proven out to be practically useful in the wild before committing to this.

Technical Details

Syntax considerations

Filter syntax has to be carefully thought through, as we need to avoid reserving too many special sequences which can be useful for additional tooling in the future, and they should be characters that are invalid for filenames when possible.

In addition, any syntax that deviates from Turborepo should try to remain open to adopting it in the future. There may be a good reason they’ve structured their syntax that way, and we should remain open to adopting it in the future.

One special syntax that we should reserve early is the use of = to indicate key-value lookups in configurations. Terragrunt has strong use-cases for this in type=unit and type=stack, and it’s a simple way to support that.

Filter order

Filters should be applied left to right for performance reasons. It will likely be advantageous to start with a coarse filter first, then get granular with a follow-up filter. This is especially true for cases like stacks and read files that are higher cost filters.

Additional flags

In addition to adding the --filter flag, the following will be added to provide accompanying functionality necessary to make it effective:

• --filter-allow-destroy
• --filter-affected
• --filters-file
• --filters-disable

Deprecation of queue control flags

With the introduction of the --filter flag, the following would be deprecated:

• --queue-exclude-dir - No longer necessary, as users can use ! to negate filters.
• --queue-excludes-file - No longer necessary, as users can use the filters file.
• --queue-exclude-external - No longer necessary, as users have to opt-in to dependents being pulled using ....
• --queue-include-dir - No longer necessary, as users can specify paths they want to include.
• --queue-include-external - No longer necessary, as users can use ....
• --queue-include-units-including - No longer necessary, as users can use ....
• --queue-include-units-reading - No longer necessary, as users can use ....
• --queue-strict-include - No longer necessary, as it will be default behavior.

find & list

The new commands introduced as part of the [CLI Redesign](#3445), find & list, should take advantage of these filter flags to give users the ability to perform dry-runs of filter queries. Without doing any plan, apply, init, etc. users will be able to use these commands to determine exactly which units will match a given filter query.

Moreover, the two give different ways to interact with the results of a filter.

Using filter on a find command will give you a flat set of all the units that are going to be run as part of the run queue. This can be a quick way to determine that the filter used matches the expected set of units the user has in mind.

$ terragrunt find --filter=[main...HEAD]

Using filter on a list command will return a structured response with groupings of units based on the DAG to give an idea of how the run queue will evolve over time. This can provide a more comprehensive outlook on what units are likely to run earlier, and which are bottlenecks for other units.

Press Release

Unit discovery and selection has been overhauled with the introduction of the new --filter flag.

Terragrunt now has a sophisticated unit querying syntax for identifying and running exactly the units you need run in a given command.

It provides the ability to:

1. Search through units in a repository via a flexible syntax that utilizes unit names, paths, dependencies, dependents and Git information for discovery.
2. AND act as a standard mechanism for both including and excluding units from the run queue.
3. AND perform Git based change detection and action.
4. AND Allow for heterogenous runs in a single Terragrunt command.

The --filter flag also allows users to use a single flag to replace the functionality of seven (now deprecated) flags, while providing more functionality:

• --queue-exclude-dir
• --queue-excludes-file
• --queue-exclude-external
• --queue-include-dir
• --queue-include-external
• --queue-include-units-including
• --queue-strict-include

To learn more about the filter flag, click here! ← Will be a link to the extensive docs on the filter flag and filter syntax.

Drawbacks

More Learning

This does require that users effectively have to learn a mini-DSL for targeting units.

The current approach where they cobble together some of the functionality through a slew of flags, Git and more is arguably more complicated, but it’s worth putting it out there.

Deprecations

While this does end up reducing the surface area of the CLI, as we’re removing more flags than we add, that does result in deprecations which are never fun.

Maintenance overhead

This is a complex feature being added, and the ability to select units that have changed between Git references might make it a heavily used one. We will need to support it, and support it well as the primary way to select units.

Breaking change on external includes

This change also has the implicit breaking change that external includes into the run queue will happen automatically. Users will have to explicitly request that they be pulled in.

The advantage of taking this approach is that users have occasionally found external includes being pulled in automatically confusing anyways, and it has performance penalties.

By moving the logic for pulling in external includes into the filter syntax, we can clean up the surface area of the CLI, while making Terragrunt easier to use.

This is a breaking change, however, and coordinating the rollout of that behavior update with a strict control will be involved.

Alternatives

Don’t Deprecate

We could just not deprecate the queue flags. We can treat them as aliases to filter flags, and allow users to continue using them.

It’s quite a lot of bloat in the CLI (at least seven flags), but there’s a good chance that users are using these flags. We could also deprecate, but just target removal in 2.0 instead of 1.0.

Do some of it

It’s possible doing all this with one flag is too ambitious. We could look to leave behind some of the “to be deprecated” flags, and use the filter flag for net-new functionality that they can’t address, like filtering on Git information.

Migration Strategy

This functionality can be rolled out progressively in phases, as the less complicated functionality is easy to get out there, and can get the clock started on deprecations.

1. Support unit name-based filtering.
2. Support path-based filtering.
3. Deprecate the following:
   1. --queue-exclude-dir
   • --queue-excludes-file
   • --queue-exclude-external
   • --queue-include-dir
   • --queue-include-external
   • --queue-include-units-including
   • --queue-strict-include
4. Support Git-based filtering.
5. Support filters file.

Users will need to adopt the --filter flag and learn the filter syntax for replacement of their usage of the now deprecated queue flags.

They will also need to adjust any usage of .terragrunt-excludes to instead use .terragrunt-filters.

We will have a period of deprecation for users to adjust to the new paradigm, and the queue flags will be translated to an equivalent filter under the hood for backwards compatibility in the interim.

Unresolved Questions

• Does the syntax appear accessible? Are there simplifications that could be made?
• How do folks feel about losing queue flags? Did I leave behind the right ones?
  • --queue-ignore-dag-order
  • --queue-ignore-errors
• Would folks feel more empowered setting up CI/CD with these changes?

References

The syntax for the filter flag is directly inspired by Turborepo syntax:

https://turbo.build/repo/docs/reference/run#--filter-string

I think it does a good job, and solves a problem for a similar need. If there’s a discrepancy between Turborepo filter syntax and Terragrunt filter syntax, we should be very intentional about deviating from it, as it’s well tested, and users have been using it for a long time to solve a similar problem at scale.

Proof of Concept Pull Request

No response

Support Level

• I have Terragrunt Enterprise Support
• I am a paying Gruntwork customer

Customer Name

No response