Docs: Make Taskflow-centric?

[afeld]

What do you see as an issue?

The Overview, Fundamental Concepts, and DAGs pages explain use of Airflow through the traditional paradigm. It's only once you get to Working with TaskFlow that the new paradigm is introduced.

Solving the problem

Assuming Taskflow is the recommended paradigm for new users, I suggest that those introductory pages are rewritten to leverage it.

Anything else

I'm having my students try out Airflow for some data engineering projects, having them go through this documentation. It's confusing that the documentation introduces one paradigm, and then later (seems to?) say "do it this other [simpler] way instead."

While I'm sure a lot of existing users are using the old paradigm, they are presumably less likely to be going through the introductory materials. Therefore, those pages could be rewritten without them noticing.

Granted, I'm new to Airflow myself, so I may be totally misunderstanding the intention/limitations.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[afeld]

Now seeing this might be swept up in #45649; if so, feel free to close.

[eladkal]

Assuming Taskflow is the recommended paradigm for new users,

Generally speaking i am not so convinced about this. taskflow is an option. It's not necessarily the favorite.
It has pros and cons.

In any case the overview page can not use taskflow. Airflow docs are built as a story not as individual pages. You can use taskflow only in pages that come after the taskflow page.

Also noting in the overview/concept pages we are looking for the simplest way to present dag authoring options. In my perspective this specific does are aiming for the lowest common ground of data professionals. Task decorators while offer simpler code (less lines etc...) are not in that domain because you need to understand what Python decorators are and how they operate (Unlike OOP concepts which we can safely assume everyone are familiar with it). Only later in the docs we can aim higher by introducing more "complex" stuff that simplify dag authoring.

Happy to hear your take on this one. Docs are for the users of Airflow if you believe we should take different approach we are happy to hear feedback. cc @cmarteepants as author of #45649

[potiuk]

Agree with @eladkal -> they are not "new" or "traditional". They are "different". There is a completely different reason why you want to use taskflow (write your custom python code to do stuff) and Operators (when you want to use operators as "black box". Both are valid patterns.

[afeld]

they are not "new" or "traditional"

I was using the language from the documentation 🤷

the TaskFlow API paradigm which is introduced as part of Airflow 2.0 and contrasts this with DAGs written using the traditional paradigm

[potiuk]

Yes. That probably should be fixed. PRs are welcome for that. The docs were written when taskflow was introduced and back then it was indeed "new" vs. "old". Now both are "old" and "established" and we should likely (again PRs are most welcome) to change the perception to put them "equal" - and explain the different purpose they serve.

[potiuk]

Converting into a discussion