Make Clickbench Q29 5x faster for datafusion #15524

zhuqi-lucas · 2025-04-01T09:13:39Z

Is your feature request related to a problem or challenge?

Our datafusion is 5x slower than duckdb for q29, it's easy for us to optimize to 5x faster, here is the try:

Extraction of Constants in Multiple AGG Calls

In ClickBench, some SQL queries can be optimized using RBO (Rule-Based Optimization) without changing semantics. For example, Q29 computes SUM(ResolutionWidth + constant) 90 times, requiring 90 columns in execution. Using the distributive property, we can rewrite it as:

Before Optimization

SELECT SUM(ResolutionWidth), SUM(ResolutionWidth + 1), ..., SUM(ResolutionWidth + 89) 
FROM hits;

After Optimization

SELECT SUM(ResolutionWidth), 
       SUM(ResolutionWidth) + 1 * COUNT(*), 
       ..., 
       SUM(ResolutionWidth) + 89 * COUNT(*) 
FROM hits;

This reduces redundant computations and improves execution efficiency.

Testing result:

Before rewrite:

cargo run --profile release-nonlto   --target aarch64-apple-darwin --bin dfbench -- clickbench  -p benchmarks/data/hits_partitioned -q 29
    Finished `release-nonlto` profile [optimized] target(s) in 0.26s
     Running `target/aarch64-apple-darwin/release-nonlto/dfbench clickbench -p benchmarks/data/hits_partitioned -q 29`
Running benchmarks with the following options: RunOpt { query: Some(29), common: CommonOpt { iterations: 3, partitions: None, batch_size: 8192, mem_pool_type: "fair", memory_limit: None, sort_spill_reservation_bytes: None, debug: false }, path: "benchmarks/data/hits_partitioned", queries_path: "benchmarks/queries/clickbench/queries.sql", output_path: None }
Q29: SELECT SUM("ResolutionWidth"), SUM("ResolutionWidth" + 1), SUM("ResolutionWidth" + 2), SUM("ResolutionWidth" + 3), SUM("ResolutionWidth" + 4), SUM("ResolutionWidth" + 5), SUM("ResolutionWidth" + 6), SUM("ResolutionWidth" + 7), SUM("ResolutionWidth" + 8), SUM("ResolutionWidth" + 9), SUM("ResolutionWidth" + 10), SUM("ResolutionWidth" + 11), SUM("ResolutionWidth" + 12), SUM("ResolutionWidth" + 13), SUM("ResolutionWidth" + 14), SUM("ResolutionWidth" + 15), SUM("ResolutionWidth" + 16), SUM("ResolutionWidth" + 17), SUM("ResolutionWidth" + 18), SUM("ResolutionWidth" + 19), SUM("ResolutionWidth" + 20), SUM("ResolutionWidth" + 21), SUM("ResolutionWidth" + 22), SUM("ResolutionWidth" + 23), SUM("ResolutionWidth" + 24), SUM("ResolutionWidth" + 25), SUM("ResolutionWidth" + 26), SUM("ResolutionWidth" + 27), SUM("ResolutionWidth" + 28), SUM("ResolutionWidth" + 29), SUM("ResolutionWidth" + 30), SUM("ResolutionWidth" + 31), SUM("ResolutionWidth" + 32), SUM("ResolutionWidth" + 33), SUM("ResolutionWidth" + 34), SUM("ResolutionWidth" + 35), SUM("ResolutionWidth" + 36), SUM("ResolutionWidth" + 37), SUM("ResolutionWidth" + 38), SUM("ResolutionWidth" + 39), SUM("ResolutionWidth" + 40), SUM("ResolutionWidth" + 41), SUM("ResolutionWidth" + 42), SUM("ResolutionWidth" + 43), SUM("ResolutionWidth" + 44), SUM("ResolutionWidth" + 45), SUM("ResolutionWidth" + 46), SUM("ResolutionWidth" + 47), SUM("ResolutionWidth" + 48), SUM("ResolutionWidth" + 49), SUM("ResolutionWidth" + 50), SUM("ResolutionWidth" + 51), SUM("ResolutionWidth" + 52), SUM("ResolutionWidth" + 53), SUM("ResolutionWidth" + 54), SUM("ResolutionWidth" + 55), SUM("ResolutionWidth" + 56), SUM("ResolutionWidth" + 57), SUM("ResolutionWidth" + 58), SUM("ResolutionWidth" + 59), SUM("ResolutionWidth" + 60), SUM("ResolutionWidth" + 61), SUM("ResolutionWidth" + 62), SUM("ResolutionWidth" + 63), SUM("ResolutionWidth" + 64), SUM("ResolutionWidth" + 65), SUM("ResolutionWidth" + 66), SUM("ResolutionWidth" + 67), SUM("ResolutionWidth" + 68), SUM("ResolutionWidth" + 69), SUM("ResolutionWidth" + 70), SUM("ResolutionWidth" + 71), SUM("ResolutionWidth" + 72), SUM("ResolutionWidth" + 73), SUM("ResolutionWidth" + 74), SUM("ResolutionWidth" + 75), SUM("ResolutionWidth" + 76), SUM("ResolutionWidth" + 77), SUM("ResolutionWidth" + 78), SUM("ResolutionWidth" + 79), SUM("ResolutionWidth" + 80), SUM("ResolutionWidth" + 81), SUM("ResolutionWidth" + 82), SUM("ResolutionWidth" + 83), SUM("ResolutionWidth" + 84), SUM("ResolutionWidth" + 85), SUM("ResolutionWidth" + 86), SUM("ResolutionWidth" + 87), SUM("ResolutionWidth" + 88), SUM("ResolutionWidth" + 89) FROM hits;
Query 29 iteration 0 took 341.5 ms and returned 1 rows
Query 29 iteration 1 took 320.7 ms and returned 1 rows
Query 29 iteration 2 took 303.0 ms and returned 1 rows
Query 29 avg time: 321.73 ms

After rewrite:

cargo run --profile release-nonlto   --target aarch64-apple-darwin --bin dfbench -- clickbench  -p benchmarks/data/hits_partitioned -q 29
    Finished `release-nonlto` profile [optimized] target(s) in 0.26s
     Running `target/aarch64-apple-darwin/release-nonlto/dfbench clickbench -p benchmarks/data/hits_partitioned -q 29`
Running benchmarks with the following options: RunOpt { query: Some(29), common: CommonOpt { iterations: 3, partitions: None, batch_size: 8192, mem_pool_type: "fair", memory_limit: None, sort_spill_reservation_bytes: None, debug: false }, path: "benchmarks/data/hits_partitioned", queries_path: "benchmarks/queries/clickbench/queries.sql", output_path: None }
Q29: SELECT SUM("ResolutionWidth"), SUM("ResolutionWidth") + 1 * COUNT(*), SUM("ResolutionWidth") + 2 * COUNT(*), SUM("ResolutionWidth") + 3 * COUNT(*), SUM("ResolutionWidth") + 4 * COUNT(*), SUM("ResolutionWidth") + 5 * COUNT(*), SUM("ResolutionWidth") + 6 * COUNT(*), SUM("ResolutionWidth") + 7 * COUNT(*), SUM("ResolutionWidth") + 8 * COUNT(*), SUM("ResolutionWidth") + 9 * COUNT(*), SUM("ResolutionWidth") + 10 * COUNT(*), SUM("ResolutionWidth") + 11 * COUNT(*), SUM("ResolutionWidth") + 12 * COUNT(*), SUM("ResolutionWidth") + 13 * COUNT(*), SUM("ResolutionWidth") + 14 * COUNT(*), SUM("ResolutionWidth") + 15 * COUNT(*), SUM("ResolutionWidth") + 16 * COUNT(*), SUM("ResolutionWidth") + 17 * COUNT(*), SUM("ResolutionWidth") + 18 * COUNT(*), SUM("ResolutionWidth") + 19 * COUNT(*), SUM("ResolutionWidth") + 20 * COUNT(*), SUM("ResolutionWidth") + 21 * COUNT(*), SUM("ResolutionWidth") + 22 * COUNT(*), SUM("ResolutionWidth") + 23 * COUNT(*), SUM("ResolutionWidth") + 24 * COUNT(*), SUM("ResolutionWidth") + 25 * COUNT(*), SUM("ResolutionWidth") + 26 * COUNT(*), SUM("ResolutionWidth") + 27 * COUNT(*), SUM("ResolutionWidth") + 28 * COUNT(*), SUM("ResolutionWidth") + 29 * COUNT(*), SUM("ResolutionWidth") + 30 * COUNT(*), SUM("ResolutionWidth") + 31 * COUNT(*), SUM("ResolutionWidth") + 32 * COUNT(*), SUM("ResolutionWidth") + 33 * COUNT(*), SUM("ResolutionWidth") + 34 * COUNT(*), SUM("ResolutionWidth") + 35 * COUNT(*), SUM("ResolutionWidth") + 36 * COUNT(*), SUM("ResolutionWidth") + 37 * COUNT(*), SUM("ResolutionWidth") + 38 * COUNT(*), SUM("ResolutionWidth") + 39 * COUNT(*), SUM("ResolutionWidth") + 40 * COUNT(*), SUM("ResolutionWidth") + 41 * COUNT(*), SUM("ResolutionWidth") + 42 * COUNT(*), SUM("ResolutionWidth") + 43 * COUNT(*), SUM("ResolutionWidth") + 44 * COUNT(*), SUM("ResolutionWidth") + 45 * COUNT(*), SUM("ResolutionWidth") + 46 * COUNT(*), SUM("ResolutionWidth") + 47 * COUNT(*), SUM("ResolutionWidth") + 48 * COUNT(*), SUM("ResolutionWidth") + 49 * COUNT(*), SUM("ResolutionWidth") + 50 * COUNT(*), SUM("ResolutionWidth") + 51 * COUNT(*), SUM("ResolutionWidth") + 52 * COUNT(*), SUM("ResolutionWidth") + 53 * COUNT(*), SUM("ResolutionWidth") + 54 * COUNT(*), SUM("ResolutionWidth") + 55 * COUNT(*), SUM("ResolutionWidth") + 56 * COUNT(*), SUM("ResolutionWidth") + 57 * COUNT(*), SUM("ResolutionWidth") + 58 * COUNT(*), SUM("ResolutionWidth") + 59 * COUNT(*), SUM("ResolutionWidth") + 60 * COUNT(*), SUM("ResolutionWidth") + 61 * COUNT(*), SUM("ResolutionWidth") + 62 * COUNT(*), SUM("ResolutionWidth") + 63 * COUNT(*), SUM("ResolutionWidth") + 64 * COUNT(*), SUM("ResolutionWidth") + 65 * COUNT(*), SUM("ResolutionWidth") + 66 * COUNT(*), SUM("ResolutionWidth") + 67 * COUNT(*), SUM("ResolutionWidth") + 68 * COUNT(*), SUM("ResolutionWidth") + 69 * COUNT(*), SUM("ResolutionWidth") + 70 * COUNT(*), SUM("ResolutionWidth") + 71 * COUNT(*), SUM("ResolutionWidth") + 72 * COUNT(*), SUM("ResolutionWidth") + 73 * COUNT(*), SUM("ResolutionWidth") + 74 * COUNT(*), SUM("ResolutionWidth") + 75 * COUNT(*), SUM("ResolutionWidth") + 76 * COUNT(*), SUM("ResolutionWidth") + 77 * COUNT(*), SUM("ResolutionWidth") + 78 * COUNT(*), SUM("ResolutionWidth") + 79 * COUNT(*), SUM("ResolutionWidth") + 80 * COUNT(*), SUM("ResolutionWidth") + 81 * COUNT(*), SUM("ResolutionWidth") + 82 * COUNT(*), SUM("ResolutionWidth") + 83 * COUNT(*), SUM("ResolutionWidth") + 84 * COUNT(*), SUM("ResolutionWidth") + 85 * COUNT(*), SUM("ResolutionWidth") + 86 * COUNT(*), SUM("ResolutionWidth") + 87 * COUNT(*), SUM("ResolutionWidth") + 88 * COUNT(*), SUM("ResolutionWidth") + 89 * COUNT(*) FROM hits;
Query 29 iteration 0 took 86.9 ms and returned 1 rows
Query 29 iteration 1 took 59.3 ms and returned 1 rows
Query 29 iteration 2 took 42.3 ms and returned 1 rows
Query 29 avg time: 62.85 ms

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

The text was updated successfully, but these errors were encountered:

zhuqi-lucas · 2025-04-01T09:21:57Z

Clickhouse also has the optimization:
https://github.com/ClickHouse/ClickHouse/pull/57853/files

zhuqi-lucas · 2025-04-01T09:22:27Z

take

zhuqi-lucas · 2025-04-01T09:24:28Z

Can anyone guide me how to do this in datafusion, i am not familiar with the rewrite now. So need some code reference.

cc @alamb @jonahgao @jayzhan211
Thanks!

jayzhan211 · 2025-04-01T10:20:07Z

You can implement this in rewrite rule SimplifyExpressions in logical optimizer

datafusion/datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

Lines 730 to 748 in 507f6b6

    
           Ok(match expr { 
        
               // 
        
               // Rules for Eq 
        
               // 
        
               // true = A  --> A 
        
               // false = A --> !A 
        
               // null = A --> null 
        
               Expr::BinaryExpr(BinaryExpr { 
        
                   left, 
        
                   op: Eq, 
        
                   right, 
        
               }) if is_bool_lit(&left) && info.is_boolean_type(&right)? => { 
        
                   Transformed::yes(match as_bool_lit(&left)? { 
        
                       Some(true) => *right, 
        
                       Some(false) => Expr::Not(right), 
        
                       None => lit_bool_null(), 
        
                   }) 
        
               }

if there is sum(column + const_a) then you return sum(column) + const_a * count(*).

zhuqi-lucas · 2025-04-01T11:05:17Z

Thank you @jayzhan211 for the guide, i will try this!

Dandandan · 2025-04-01T12:02:39Z

Nice idea!
I think it makes sense to avoid doing this optimization when there is only a single case of this - otherwise the number of accumulators gets bigger (combined with group by this will lead to higher memory usage).

alamb · 2025-04-01T16:16:33Z

As long as this optimization is plausibly general purpose (not just for ClickBench) and the code required is isolated / doesn't add a bunch of complexity, I am in support of this optimization

Thank you @zhuqi-lucas

alamb · 2025-04-01T22:26:49Z

I found a duckdb implementation of a seemingling similar optimization: https://github.com/duckdb/duckdb/blob/7912713493b38b1eda162f29b7759d5024989a5f/src/optimizer/sum_rewriter.cpp#L25

zhuqi-lucas · 2025-04-02T00:14:25Z

Thank you @Dandandan @alamb for double check and confirm!

Dandandan · 2025-04-02T06:39:12Z

I found the rationale behind adding it in DuckDB was also ClickBench.

duckdb/duckdb#15017

As a randomly selected query, let's run this query from ClickBench:

😆

So we're now officially benchmaxxing!

Dandandan · 2025-04-02T07:03:31Z

I'm wondering if instead of this specialized

SUM(x + c) -> SUM(x) * COUNT() * c

We can do instead the more general (distributive property of summation - doesn't need to be a constant):

SUM(x + y) -> SUM(x) + SUM(y)

And have other optimizations follow (transform SUM(y) -> COUNT() * y when y is constant, common subexpression elimination, etc.)

Dandandan · 2025-04-02T07:12:53Z

The nice part of that is that addition in SUM can be quite common and SUM(x) + SUM(y) is faster than SUM(x + y) (even without transforming to use COUNT)

zhuqi-lucas · 2025-04-02T08:38:42Z

I'm wondering if instead of this specialized

SUM(x + c) -> SUM(x) * COUNT() * c

We can do instead the more general (distributive property of summation - doesn't need to be a constant):
SUM(x + y) -> SUM(x) + SUM(y)
And have other optimizations follow (transform SUM(y) -> COUNT() * y when y is constant, common subexpression elimination, etc.)

Thank you @Dandandan for this good idea, i agree with you, it's a more common solution and it will benefit more cases.

And meanwhile, may be we can do more cases in future, such as when MAX(x) > 0:

MAX(5 * x) => 5 * MAX(x)

etc

berkaysynnada · 2025-04-02T09:01:38Z

I haven't reviewed the PR yet, but I agree with @Dandandan, and I think we can improve this further. We've actually thought about this issue before and sketched out an initial design. Let me share some notes from that:

This simplification should be based on the linearity property, not just SUM() and COUNT() rewrites. Formally:

f(x + y) = f(x) + f(y), if f is a linear function.

So, I believe we should define a "linear function" tag for all such functions.

Consider the same query:

SELECT SUM(id), SUM(id + 1), SUM(id + 2), ..., SUM(id + 89) FROM employees;

LP:

--Aggregate: groupBy=[[]], aggr=[
  sum(__common_expr_1 AS employees.id),
  sum(__common_expr_1 AS employees.id + Int64(1)),
  ...,
  sum(__common_expr_1 AS employees.id + Int64(89))
]
----Projection: CAST(employees.id AS Int64) AS __common_expr_1
------TableScan: employees projection=[id]

PP:

--AggregateExec: mode=Single, gby=[], aggr=[
  sum(employees.id),
  sum(employees.id + Int64(1)),
  ...,
  sum(employees.id + Int64(89))
]
----ProjectionExec: expr=[CAST(id@0 AS Int64) as __common_expr_1]
------MemoryExec: partitions=1, partition_sizes=[1]

We should apply the linearity property here to simplify expressions like SUM(id + n) into SUM(id) + n * COUNT(1), when n is a constant. It doesn't effect the performance of this clickbench query, but we should also properly handle the cases when n is not constant as well.

zhuqi-lucas · 2025-04-02T09:32:52Z

Thank you @berkaysynnada, i agree it's a common linearity property, this is a great idea. I will try to address it, and may be we can start from SUM function. And add more cases to extend it in future, such as add linearity enum and add SUM, etc to it.

alamb · 2025-04-02T10:46:54Z

It looks like there were several follow up tickets in DuckDB (no test changes which is weird)

Avoid sum rewrite for hugeint/uhugeint since it could introduce overflow errors duckdb/duckdb#15040
Sum Rewriter: correctly match only Sum aggregations in sum rewriter duckdb/duckdb#15088

zhuqi-lucas · 2025-04-03T04:32:11Z

Try to address the comments for sum(3a + 2b) => 3sum(a) + 2sum(b), but it's more complex than i expected, and more corner cases and testing failed, i am looking into and try to fix it.

zhuqi-lucas · 2025-04-03T09:56:36Z

Need help, i am not sure if i do the right direction for this ticket:

I am still debugging my draft PR, if folks can cooperate, feel free to add a fix to my PR because it will meet more corner cases i believe, and i am still not fixing the existing error:

#15532

More problems need to be fixed based on my PR, for example:

It seems, simplify_expressions will change the sum(a + b) => sum(a) + sum(b), so the error happens:

logical_plan after simplify_expressions
01)Projection: sum(simple_explain_test.a + simple_explain_test.b)
02)--Aggregate: groupBy=[[]], aggr=[[sum(simple_explain_test.a) + sum(simple_explain_test.b) AS sum(simple_explain_test.a + simple_explain_test.b)]]
03)----TableScan: simple_explain_test

logical_plan
01)Aggregate: groupBy=[[]], aggr=[[sum(simple_explain_test.a) + sum(simple_explain_test.b) AS sum(simple_explain_test.a + simple_explain_test.b)]]
02)--TableScan: simple_explain_test projection=[a, b]
physical_plan_error
01)Internal error: Invalid aggregate expression 'BinaryExpr(BinaryExpr { left: AggregateFunction(AggregateFunction { func: AggregateUDF { inner: Sum { signature: Signature { type_signature: UserDefined, volatility: Immutable } } }, params: AggregateFunctionParams { args: [Column(Column { relation: Some(Bare { table: "simple_explain_test" }), name: "a" })], distinct: false, filter: None, order_by: None, null_treatment: None } }), op: Plus, right: AggregateFunction(AggregateFunction { func: AggregateUDF { inner: Sum { signature: Signature { type_signature: UserDefined, volatility: Immutable } } }, params: AggregateFunctionParams { args: [Column(Column { relation: Some(Bare { table: "simple_explain_test" }), name: "b" })], distinct: false, filter: None, order_by: None, null_treatment: None } }) })'.
02)This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker

Some unit tests are failing, need to investigate.
Need to check if the datatype changes, it is still correct for the rewrite.
etc

@alamb
And it seems duckdb also only do the sum(a + C), i am wandering if we can do similar first besides the sum(a + b) => sum(a) + sum(b)

duckdb/duckdb#15017

I found a duckdb implementation of a seemingling similar optimization: https://github.com/duckdb/duckdb/blob/7912713493b38b1eda162f29b7759d5024989a5f/src/optimizer/sum_rewriter.cpp#L25

alamb · 2025-04-04T15:21:13Z

So in my opinion before we get too fancy we should figure out what optimizaton we are really trying to do

I understand it may be possible to do a more general rewrite such as @berkaysynnada suggests above #15524 (comment)

However, I think it would help to have an actual usecase / real query.

The only usecase I know of is the click bench query

SELECT SUM(ResolutionWidth), SUM(ResolutionWidth + 1), SUM(ResolutionWidth + 2), SUM(ResolutionWidth + 3), SUM(ResolutionWidth + 4), SUM(ResolutionWidth + 5), SUM(ResolutionWidth + 6), SUM(ResolutionWidth + 7), SUM(ResolutionWidth + 8), SUM(ResolutionWidth + 9), SUM(ResolutionWidth + 10), SUM(ResolutionWidth + 11), SUM(ResolutionWidth + 12), SUM(ResolutionWidth + 13), SUM(ResolutionWidth + 14), SUM(ResolutionWidth + 15), SUM(ResolutionWidth + 16), SUM(ResolutionWidth + 17), SUM(ResolutionWidth + 18), SUM(ResolutionWidth + 19), SUM(ResolutionWidth + 20), SUM(ResolutionWidth + 21), SUM(ResolutionWidth + 22), SUM(ResolutionWidth + 23), SUM(ResolutionWidth + 24), SUM(ResolutionWidth + 25), SUM(ResolutionWidth + 26), SUM(ResolutionWidth + 27), SUM(ResolutionWidth + 28), SUM(ResolutionWidth + 29), SUM(ResolutionWidth + 30), SUM(ResolutionWidth + 31), SUM(ResolutionWidth + 32), SUM(ResolutionWidth + 33), SUM(ResolutionWidth + 34), SUM(ResolutionWidth + 35), SUM(ResolutionWidth + 36), SUM(ResolutionWidth + 37), SUM(ResolutionWidth + 38), SUM(ResolutionWidth + 39), SUM(ResolutionWidth + 40), SUM(ResolutionWidth + 41), SUM(ResolutionWidth + 42), SUM(ResolutionWidth + 43), SUM(ResolutionWidth + 44), SUM(ResolutionWidth + 45), SUM(ResolutionWidth + 46), SUM(ResolutionWidth + 47), SUM(ResolutionWidth + 48), SUM(ResolutionWidth + 49), SUM(ResolutionWidth + 50), SUM(ResolutionWidth + 51), SUM(ResolutionWidth + 52), SUM(ResolutionWidth + 53), SUM(ResolutionWidth + 54), SUM(ResolutionWidth + 55), SUM(ResolutionWidth + 56), SUM(ResolutionWidth + 57), SUM(ResolutionWidth + 58), SUM(ResolutionWidth + 59), SUM(ResolutionWidth + 60), SUM(ResolutionWidth + 61), SUM(ResolutionWidth + 62), SUM(ResolutionWidth + 63), SUM(ResolutionWidth + 64), SUM(ResolutionWidth + 65), SUM(ResolutionWidth + 66), SUM(ResolutionWidth + 67), SUM(ResolutionWidth + 68), SUM(ResolutionWidth + 69), SUM(ResolutionWidth + 70), SUM(ResolutionWidth + 71), SUM(ResolutionWidth + 72), SUM(ResolutionWidth + 73), SUM(ResolutionWidth + 74), SUM(ResolutionWidth + 75), SUM(ResolutionWidth + 76), SUM(ResolutionWidth + 77), SUM(ResolutionWidth + 78), SUM(ResolutionWidth + 79), SUM(ResolutionWidth + 80), SUM(ResolutionWidth + 81), SUM(ResolutionWidth + 82), SUM(ResolutionWidth + 83), SUM(ResolutionWidth + 84), SUM(ResolutionWidth + 85), SUM(ResolutionWidth + 86), SUM(ResolutionWidth + 87), SUM(ResolutionWidth + 88), SUM(ResolutionWidth + 89) FROM hits;

I have never seen such a query elsewhere and I would struggle to explain to someone in human language what that was calculating and why it was written that way

Thus for this one, I would personally suggest either:

just following the Duck and make a benchmark specific optimization (and don't try to handle any other cases)
take the high road and say we aren't going to benchmaxx

Alternately we can spend some time trying to do something more general, but I probably don't have much time to help here as I don't think it will be widely applicable

zhuqi-lucas added the enhancement New feature or request label Apr 1, 2025

github-actions bot assigned zhuqi-lucas Apr 1, 2025

zhuqi-lucas mentioned this issue Apr 1, 2025

[EPIC] ClickBench Improvements (Vanity Benchmark) #14586

Open

zhuqi-lucas linked a pull request Apr 1, 2025 that will close this issue

Draft: Make Clickbench Q29 5x faster for datafusion #15532

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make Clickbench Q29 5x faster for datafusion #15524

Make Clickbench Q29 5x faster for datafusion #15524

zhuqi-lucas commented Apr 1, 2025 •

edited

Loading

zhuqi-lucas commented Apr 1, 2025

zhuqi-lucas commented Apr 1, 2025

zhuqi-lucas commented Apr 1, 2025 •

edited

Loading

jayzhan211 commented Apr 1, 2025 •

edited

Loading

zhuqi-lucas commented Apr 1, 2025

Dandandan commented Apr 1, 2025 •

edited

Loading

alamb commented Apr 1, 2025 •

edited

Loading

alamb commented Apr 1, 2025

zhuqi-lucas commented Apr 2, 2025

Dandandan commented Apr 2, 2025 •

edited

Loading

Dandandan commented Apr 2, 2025 •

edited

Loading

Dandandan commented Apr 2, 2025 •

edited

Loading

zhuqi-lucas commented Apr 2, 2025

berkaysynnada commented Apr 2, 2025

zhuqi-lucas commented Apr 2, 2025 •

edited

Loading

alamb commented Apr 2, 2025 •

edited

Loading

zhuqi-lucas commented Apr 3, 2025

zhuqi-lucas commented Apr 3, 2025 •

edited

Loading

alamb commented Apr 4, 2025

Make Clickbench Q29 5x faster for datafusion #15524

Make Clickbench Q29 5x faster for datafusion #15524

Comments

zhuqi-lucas commented Apr 1, 2025 • edited Loading

Is your feature request related to a problem or challenge?

Extraction of Constants in Multiple AGG Calls

Before Optimization

After Optimization

Testing result:

Describe the solution you'd like

Describe alternatives you've considered

Additional context

zhuqi-lucas commented Apr 1, 2025

zhuqi-lucas commented Apr 1, 2025

zhuqi-lucas commented Apr 1, 2025 • edited Loading

jayzhan211 commented Apr 1, 2025 • edited Loading

zhuqi-lucas commented Apr 1, 2025

Dandandan commented Apr 1, 2025 • edited Loading

alamb commented Apr 1, 2025 • edited Loading

alamb commented Apr 1, 2025

zhuqi-lucas commented Apr 2, 2025

Dandandan commented Apr 2, 2025 • edited Loading

Dandandan commented Apr 2, 2025 • edited Loading

Dandandan commented Apr 2, 2025 • edited Loading

zhuqi-lucas commented Apr 2, 2025

berkaysynnada commented Apr 2, 2025

zhuqi-lucas commented Apr 2, 2025 • edited Loading

alamb commented Apr 2, 2025 • edited Loading

zhuqi-lucas commented Apr 3, 2025

zhuqi-lucas commented Apr 3, 2025 • edited Loading

alamb commented Apr 4, 2025

zhuqi-lucas commented Apr 1, 2025 •

edited

Loading

zhuqi-lucas commented Apr 1, 2025 •

edited

Loading

jayzhan211 commented Apr 1, 2025 •

edited

Loading

Dandandan commented Apr 1, 2025 •

edited

Loading

alamb commented Apr 1, 2025 •

edited

Loading

Dandandan commented Apr 2, 2025 •

edited

Loading

Dandandan commented Apr 2, 2025 •

edited

Loading

Dandandan commented Apr 2, 2025 •

edited

Loading

zhuqi-lucas commented Apr 2, 2025 •

edited

Loading

alamb commented Apr 2, 2025 •

edited

Loading

zhuqi-lucas commented Apr 3, 2025 •

edited

Loading