Follow-up to Components announcement: Pythonic Components

[schrockn]

Pythonic Components

Context

Two weeks ago we announced Components and we heavily emphasized the YAML frontend piece of the system. While for a lot of users the YAML frontend is critical for non-technical stakeholders, a lot of you might have thought: I want nothing to do with YAML. I want to program in Python.

Components are not just a YAML interface. They are a new, first-class way you structure and manage the programmatic creation of all definitions (e.g. assets, asset checks, sensors, etc) in a project.

In fact, anyone placing a python file in the defs folder in the dg project layout will be utilizing the components infrastructure under the hood, whether they know it or not, because Components drive the autoloading process.

In this post I’ll discuss how to use Pythonic components and their value proposition.

How do you use a Pythonic Component?

The scaffold command now has a --format option, which includes python. If you run this command, a component.py is produced, rather than component.yaml.

For example

dg scaffold dagster_components.dagster_dbt.DbtProjectComponent --format python_component

Produces a component.py

from dagster_components import component, ComponentLoadContext
from dagster_components.dagster_dbt import DbtProjectComponent

@component
def load(context: ComponentLoadContext) -> DbtProjectComponent: ...

At which point you can implement the body of load and create a DbtProjectComponent directly, rather than use the YAML interface.

from dagster_components import component, ComponentLoadContext
from dagster_components.dagster_dbt import DbtProjectComponent
from dagster_dbt import DbtCliResource

@component
def load(context: ComponentLoadContext) -> DbtProjectComponent:
    return DbtProjectComponent(dbt=DbtCliResource(project_dir="."))

What is the value without the YAML interface?

A reasonable question to ask is “Why use this component thing at all if I’m not going to use YAML?”

Well it’s actually a more core abstraction than that going forward, will be underlying way that we organize Dagster projects and structure the process of loading their definitions.

Eliminate the import circus and improve modularity

If you have written a Dagster project of any scale, you’ll have a file at the root of your project like this:

# definitions.py
import dagster as dg

from .schedules import schedule_one, schedule_two
from .asset_one import build_asset_one
from .asset_two import build_asset_two
from .resources import ResourceUsedByAssetOne, ResourceUsedByAssetTwo

defs = dg.Definitions(
  assets=[build_asset_one(), build_asset_two()],
  schedules=[schedule_one, schedule_two],
  resources={
     "resource_one", ResourceUsedByAssetOne(),
     "resource_two", ResourceUsedByAssetTwo()
  }
)

And this is a simple project. In the complex project this file can be hundreds of lines of code. Worse, it is a central file, often edited by all but owned by no-one and becomes a complete mess.

With components, this top-level file is effectively eliminated entirely (we include a stub definitions.py file right now but will be able to completely eliminate it) and definition creation becomes modular. Your resources are defined next to the assets (and other definitions) that use them, and everything is autoloaded:

So instead everything happening one file, you can define the resources at the same level where your definitions will use them.

# in asset_one.py

@asset
def asset_one() -> None: ...

defs = Definitions(
   assets=[asset_one],
   resources={"resource_one", ResourceUsedByAssetOne()}
)

# in asset_two.py

@asset
def asset_two() -> None: ...

return Definitions(
   assets=[asset_two],
   resources={"resource_two", ResourceUsedByAssetTwo()}
)

We will be adding more support to make this a little more ergonomic in the future, stay tuned.

Avoiding import-time side effects due to asset factories

Right now in guides like [these](https://docs.dagster.io/guides/build/assets/creating-asset-factories) we advocate the use of factory functions in cases where you want to programmatically create definitions from a higher-level API, a DSL, or sourced from external state (like a database or a file). Unfortunately with the way that Dagster worked this meant these computations had to be done at import time which is dangerous and undesirable. We worked around this with some convoluted abstractions but it was suboptimal. There were a number of problems with this. For example, A transient error could completely eliminate the ability to load the python process entirely. Another problem is that it was easy to invoke heavyweight operations during unit tests and other operations.

Factory patterns expressed at components mean that all calls to build_defs can be structured to happen after the python process starts. In the short-term this is very useful for unit tests scenarios. Longer term it opens up space for much more granular control of definitions loading, allowing for faster, more targeted reloading of definitions.

Definitions organization

Components are also a powerful generalized tool for organizing definitions and applying metadata transformations in a coherent way. It is quite common to want to mass apply tags or group names to arbitrary sets of definitions based on business logic. Components and autoloading have built-in features to enable you to do that at any arbitrary level of your folder hierarchy, applying post processing to any collection of definitions.

For example if you had a large, multi-team project laid out like this:

defs/
  team_one/
    domain_one/
    domain_two/
  team_two/
    domain_one/
    domain_one/

And if one team wanted to standardize on some tag, they could do so by dropping a component.yaml at the defs/team_two/component.yaml. We have a pattern of “post processing” that any component can implement, including the “folder” component.

type: dagster_components.dagster.FolderDefsComponent
attributes:
  asset_post_processors:
    - target: "*"
      attributes:
        owner: "team_one"
    - target: "tag:bar_group=true" # can be arbitrary selections
      # some legacy assets encode groups this way and easier to fix this way
      attributes:                  
        group: bar

Future work

We are building or planning a number of features that leverage this new structure

• Env and secrets management. We are in the process of improving our secrets and environment management based on the component systems. We will in a metadata-driven way understand the environment requirements of any particular component, and be able to improve tooling and observability of secrets management. For example you will be able to know, prior to deployment, whether or not the target deployment has all the necessary environment variables set. We will also be able to provide lineage so you understand what environment variables are consumed by what components.
• Scaling large projects: Loading subsets of the projects. With this modular approach to definitions construction under the control of the framework, we will be able to load and reload subsets of projects. As an example, commands like dg list defs could be much faster if targeted to a specific submodule.
• Programmatically constructing definitions from slower backends. Right now the process of managing the construction of definitions whose source of truth comes from external systems is quite harrowing. Within the component hierarchy, we have far more control over the definitions loading process, and will be able to support this use case better. For example we plan on making resources available during the load process so you can utilize them in that context to improve code structure and testability.

Feel free to ask further questions in the discussion thread. Thanks!

[stkbailey]

With components, this top-level file is effectively eliminated entirely (we include a stub definitions.py file right now but will be able to completely eliminate it) and definition creation becomes modular.

👏 👏 👏 👏 👏 👏 👏 👏 👏 🚀 🎸

[stkbailey]

And if one team wanted to standardize on some tag, they could do so by dropping a component.yaml at the defs/team_two/component.yaml.

I wonder if it would make sense to allow for post-processor to be added to the @components class as well, so that Pythonic components don't have to include both the Components class and a YAML file.

[menzenski]

Factory patterns expressed at components mean that all calls to build_defs can be structured to happen after the python process starts. In the short-term this is very useful for unit tests scenarios. Longer term it opens up space for much more granular control of definitions loading, allowing for faster, more targeted reloading of definitions.

We use a lot of factory patterns to support a lot of environment-specific logic (as we have a number of environments and a lot of assets and jobs should only be defined in a subset of them). I'd be extremely interested in how this use case might intersect with this delayed build_defs call.

For example, if I have an environment variable with the environment name (DEPLOYED_ENVIRONMENT=staging, say), could I define my components in such a way that they could key off that environment name? (Maybe asset X can only exist in production, or maybe all schedules should be disabled in the dev environment, or similar).

[schrockn]

We can leverage this system to make this entire scenario much better. We're actively working on it, starting with environment variable management tied to the Components system. This will also give us a structured way to interface with external services (think secrets managers) during definitions construction. We also anticipated building a formal notion of profiles/scopes/environments on top of that.

[menzenski]

I am also wondering if this means that EnvVar references might be resolved before an asset factory is evaluated? That would be very cool - we have some things that we would like to persist to a db during evaluation of an asset factory function (recording config version, etc). If we could use an existing database Resource definition instead of defining our own DB accessor just for this, that'd be a nice benefit too.