How to record an error for a single partition in a multi-partition, single-run backfill?

[polivbr]

I frequently have the need to perform large single-run backfills that take significant amounts of time. Is there a way to report an error for an individual partition without causing the entire backfill run to fail? I would like to be able to continue processing the remaining partitions, even if a single one has an issue.

[dridk]

Something like this ?

@run_failure_sensor(request_job=my_job)
def my_failure_sensor(context: RunFailureSensorContext):
       context.log.debug("Run failed ")

You can get partition information from context.

[polivbr]

Thanks @dridk, but my issue is a little different. For example, let's say I'm backfilling 100 partitions in a single run. I can report successful partition materializations by yielding an AssetMaterialization event. However, as far as I can tell, the only way to report an error is by throwing an exception. The problem with this is that it will abort the backfill run and other partitions won't be processed, even though they may have succeeded. I'd love a way to yield an event, similar to an AssetMateralization, like an AssetMaterializationFailed event, to record that a particular partition failed.

[dridk]

I don't understand.
From my side, when I start a backfill of an asset, I can have some partitions in error displayed in red on dagit. But it doesn't affect other partition materialisation.what I missed?

[polivbr]

Sure, if you're backfilling each partition in a separate run. If you are processing them all in a single run (backfill_policy=BackfillPolicy.single_run()), there's no way to signify that one of the partitions in the run failed while continuing to process the others in the same run.