The documentation on data_interval_end is unclear

[jceresini]

What do you see as an issue?

The documentation for the template variable data_interval_end states simply that its the "End of the data interval". Initially I took that to mean the final second/microsecond (depending on precision) of the data interval, but its actually the start of the next interval.

For example, given a schedule that runs every 5 minutes, the variables are set as follows (for a simple test I just ran):

• data_interval_start: 2022-07-28T16:15:00+00:00
• data_interval_end: 2022-07-28T16:20:00+00:00

As opposed to:

• data_interval_start: 2022-07-28T16:15:00+00:00
• data_interval_end: 2022-07-28T16:19:59+00:00

It makes sense as implemented, but I'd like to see the documentation state the behavior explicitly.

Solving the problem

The documentation I read when using the template variables is https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html

I'm not sure how to word it, but it would be helpful to indicate that data_interval_end is effectively the data_interval_start of the next interval. Or that the interval the DAG is operating on, using mathematical interval notation, is [data_interval_start, data_interval_end)

Anything else

Just an explanation of how we ran into this:

We are running DAGS periodically that pull timeseries data from some API. The jobs query an api with filters like this:

• timestamp >= data_interval_start
• timestamp <= data_interval_end

We noticed we were getting some duplicated data, specifically data that happened exactly on the data_interval_start value. The simple fix (once we saw the behavior of the variables) was to remove the = from the second filter:

• timestamp >= data_interval_start
• timestamp < data_interval_end

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Feel free to explain it better. You are probably the best person in the world to know how to phrase it better as you just struggled with it.

[potiuk]

Assigned you to it.

[uranusjr]

data_interval_end is the end of the data interval, not the start of the next data interval. For a DAG that runs every five minutes, the data interval is the span between that five minutes, not the single second/minute/whatever the DAG is run. This is already explained in the documentation, so I think the appropriate fix would be to link to that section.

https://airflow.apache.org/docs/apache-airflow/stable/dag-run.html#data-interval

[leodrivera]

Looking at the documentation, I also had the same interpretation as @jceresini. However, I agree with @uranusjr that the ideal would be to link the section related to the data-interval.

[uranusjr]

Reading the documentation (linked in my previous comment), one additional thing I would add to clarify against the confusion is that, A data_interval_end is not logically equivalent to the next data interval’s start. This only applies to a continuous schedule, i.e. @daily, where each data interval follows directly after the previous one.

But there are discrete schedules as well, where data intervals are not continuous. In those cases, data_interval_end would not (or not always) be equal to the next data_interval_start.

Anyone’s planning on improving the docs, it’d be most awesome if you come in with a clean head, remember what you know before and after you entered this conversation, and document exactly that. This would be most beneficial to regular Airflow users since they would likely be at the place as you were, and the best way to help them figure this out is to take them through the exact think process you did.

[billcrook]

This is still unclear in the docs. The link proposed by @uranusjr does not mention the template variables data_interval_start or data_interval_end at all. Given the original post by @jceresini, the main point to get across is that data_interval_end is exclusive of data in the interval. It is implied that the intent of the job is to process data between data_interval_start and data_interval_end but it might not be obvious to newcomers. Maybe a table with some examples would help communicate?

schedule	logical date	data_interval_start	data_interval_end
@daily	2025-01-01, 00:00:00 UTC	2025-01-01, 00:00:00 UTC	2025-01-02, 00:00:00 UTC
@quarterly	2025-01-01, 00:00:00 UTC	2025-01-01, 00:00:00 UTC	2025-04-01, 00:00:00 UTC

And maybe some discrete schedule examples.