Airflow 2.10.4 is significantly slower compared to 2.9.3

[Komalpawar22]

Apache Airflow version

2.10.4

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Describe the slowness and when it occurs (e.g., loading the web UI, DAG execution, scheduler performance). I have huge dag which was working very fine but now it is not loading and taking more time to load after i upgrade. I have changed nothing in the code.

What you think should happen instead?

Airflow should perform similarly or better after the upgrade.

How to reproduce

upgrade from 2.9.2 to 2.10.4

Operating System

linux

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

That's interesting, but completely not actionable discussion. If anyone has some data to show and compare what has been slower in their installation before and after, that could be very good start to do somethign about it, but until details, comparisions, measurements and showing the different metrics in those two versions, accompanied with details about deployment and configuration, there is absolutely nothing anyone can do about this report,

Please @Komalpawar22 -> if you want to provide such information, feel free to do so - and also anyone else - but without that details, it's just an interesting observation that is completely not actionable.

[Komalpawar22]

Thank you for your response. I understand the need for concrete data to make this issue actionable. Below are the details of my setup and observations:

System Details
Airflow Version: Upgraded from 2.9.3 → 2.10.4
Executor: KubernetesExecutor
Database: MySQL
Python Version: 3.11
OS: Linux
Deployment Mode: Kubernetes Autopilot Cluster

Old version was taking 7-8 seconds to load the dag
And the latest version take 50-60 seconds or even more than a minute

        resources:
          limits:
            cpu: 100m
            ephemeral-storage: 1000Mi
            memory: 3Gi
          requests:
            cpu: 100m
            ephemeral-storage: 1000Mi
            memory: 3Gi

Using 5 replicas of this configuration, I have deployed airflow with official Helm chart 1.14.0

[potiuk]

Well. you likely have something broken - but what it is - hard to say without more details. If you cannot find any evidences and loggs and compare it to old stuff, there is not much anyone can do about it.

The "7-8 seconds to load a DAG" does not explain what happens. Show logs. Explain what you see and where the slowdown comes from - then you have a chance someone will try to help you to debug your problem. There are a LOT of reasons why things might get slower and there is no point guessing. Ideally - run them side-by side and try to spot the differences you observe in logs and provide some comparision of concrete things that are slower.

[potiuk]

Also - just to put it in perspective - this is NOT a common problem. There are many users using Airflow and if it was 10x slower for everyone, we would have known about it. So something in your environment is causing the slowdown. Likely some configuration issue, k8s configuration, way how you are running stuff.

But without more details and investigation on your side, we have no prior cases and similar reports to even guess.

[Komalpawar22]

Hi @potiuk,

My observation over time is that DAGs with fewer tasks are running fast, whereas I am specifically referring to a DAG with 3000+ tasks, and this number may increase in the future.

How can we improve performance in this case? What do you think could be the possible reasons for this slowdown? Understanding this will help me debug the issue in more depth and provide direction on where to focus my investigation.

[potiuk]

No idea, really - very hard to guess, you need to try to see what is slow. observe resources and timeline of things - look at what scheduler does, where your delays are. that's as much of generic advice I can give.