Weekly Plan (Andrew Lamb) March 24, 2025

[alamb]

This is an attempt to organize myself and make what I plan to work on more visible

Weekly High Level Goals

• Complete making tpch data generator screaming fast to generate parquet with @clflushopt and @scsmithr : Make it easier to run TPCH queries with datafusion-cli #14608
• Get tree explain plans on by default [DISCUSS] Switch to tree explain by default #15343
• Get Enable parquet filter pushdown by default #3463 ready for merge with @XiangpengHao

Other projects I think are strategically important

For a list of projects, see

• March 17, 2025: This week(s) in DataFusion #15269

These are the ones I plan to look at / review PRs in order

• Anything performance related
• Hardening external sorts: A complete solution for stable and safe sort with spill #14692 with @2010YOUY01
• Spark functions: feat: Add datafusion-spark crate #15168
• pushing expressions down into scans: Support Push down expression evaluation in TableProviders #14993
• Change mapping of SQL VARCHAR from Utf8 to Utf8View #15096 (comment)

Weekly Stretch / Nice to have goals

• Try to move an example or two from datafusion-examples

Background

I am putting this list on github because:

1. I like how github renders checklists w/ PR titles so it is easy to track (I currently have a local text file...)
2. I thought others might be interested from seeing what I am doing / planning to do
3. It makes me feel better that I don't have time to review all the PRs 😭

The way I am trying to prioritize PRs is in the following order

1. Bug fixes
2. Documentation / UX / API improvements (things that make DataFusion easier/better to work with)
3. Performance improvements
4. New features with wide appeal
5. New functions

Note new features and functions are deliberately at the bottom

[alamb]

Next week: #15528