Weekly Plan (Andrew Lamb) March 31, 2025

[alamb]

This is an attempt to organize myself and make what I plan to work on more visible

Weekly High Level Goals

• Work on integrating tpch data generator with @clflushopt : Make it easier to run TPCH queries with datafusion-cli #14608
• Avoid resorting/merging [EPIC] Avoid sort for already sorted Parquet files that do not overlap values on condition #6672 with @wiedld @suremarc and @xudong963
• Get Enable parquet filter pushdown by default #3463 ready for merge with @XiangpengHao
• TopK Pushdown Dynamic pruning filters from TopK state (optimize ORDER BY LIMIT queries) #15037 with @adriangb
• Arrow release: Release arrow-rs / parquet major version 55.0.0 (Apr 2025) arrow-rs#7084

Other projects I plan to review

• Bug fixes
• Other performance improvements
• Complete insta test migration [Epic] Add snapshot tests (migrate to insta for tests) #15178 with @blaginin @shruti2522 @qstommyshu and others
• Hardening external sorts: A complete solution for stable and safe sort with spill #14692 with @2010YOUY01
• Set up Spark function library pattern: feat: Add datafusion-spark crate #15168 with @shehabgamin and @andygrove
• Use UTF8 view by default Change mapping of SQL VARCHAR from Utf8 to Utf8View #15096 with @zhuqi-lucas

Background

I am putting this list on github because:

1. I like how github renders checklists w/ PR titles so it is easy to track (I currently have a local text file...)
2. I thought others might be interested from seeing what I am doing / planning to do
3. It makes me feel better that I don't have time to review all the PRs 😭

The way I am trying to prioritize PRs is in the following order

1. Bug fixes
2. Documentation / UX / API improvements (things that make DataFusion easier/better to work with)
3. Performance improvements
4. New features with wide appeal
5. New functions

Note new features and functions are deliberately at the bottom