Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Register materialized views with the planner #115

Merged
merged 1 commit into from
Feb 26, 2025

Conversation

ryannedolan
Copy link
Collaborator

Summary

  • Enable catalogs to register materializations with the planner.
  • Delay validation of view SQL until after all views have loaded.
  • Minor API changes to carry around a Connection.

Details

Calcite supports substituting all or parts of a query with a view, which may accelerate queries if the view is materialized. We want Hoptimator to leverage this mechanism when planning a pipeline, which would enable, for example, a pipeline to read from a snapshot rather than a source-of-truth event stream. In that scenario, the snapshot must be registered as a materialization of the event stream.

To that end, these changes enable catalogs to register arbitrary materializations. The K8s catalog naively registers all materialized views this way. We can also easily envision a catalog that registers Iceberg tables corresponding to every Kafka topic, etc.

I haven't been able to verify that the planner actually considers using the materializations, but at least it doesn't complain.

Testing

Existing tests pass. Leaving validation of materializations to future work.

Copy link
Collaborator

@jogrogan jogrogan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool stuff! looks like some tests are failing but otherwise LGTM

@ryannedolan ryannedolan force-pushed the register-materialized-views branch from 8281c3f to 1c2d5ea Compare February 26, 2025 16:53
@ryannedolan ryannedolan enabled auto-merge (squash) February 26, 2025 16:55
@ryannedolan ryannedolan merged commit 975ff9c into main Feb 26, 2025
1 check passed
@ryannedolan ryannedolan deleted the register-materialized-views branch February 26, 2025 17:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants