Add materialized views RFC

[tdcmeehan]

No description provided.

[jainxrohit]

@tdcmeehan Thanks for this RFC. Meta is also planning to resume the work in the materialized views. Back in the days, I had initially tried to build this initially in the planning phase; however, that led to really complex and messy structures, which were very difficult to work with. We probably need even higher level abstraction. Let's align with @arhimondr, @kaikalur and @zation99 on this.

[kaikalur]

@tdcmeehan Thanks for this RFC. Meta is also planning to resume the work in the materialized views. Back in the days, I had initially tried to build this initially in the planning phase; however, that led to really complex and messy structures, which were very difficult to work with. We probably need even higher level abstraction. Let's align with @arhimondr, @kaikalur and @zation99 on this.

Two issues

• First one is to come up with a good definition of the view
• Overhead in the view maintenance

so we need to do some cost benefit analysis. Best might be to keep it narrowly scoped like a "preagg a single table" view for aggregation only queries to start with. Once that stabilizes, we can extend it to other shapes.

[tdcmeehan]

Thanks for the comments @jainxrohit and @kaikalur.

@kaikalur to be clear, this design allows for your recommendation of progressive refinement. This design's default behavior is to be rather simple, exactly as you recommended--single table materialized views with simple stale/fresh checks. Progressive enhancements, such as fine grained refresh or unioning up to date partitions with more up to date data, are optional and can be implemented by connectors as needed.

@jainxrohit thanks for the background. I believe this design abstracts complexity and allows for progressive enhancement. I'm keen to maintain momentum on the implementation. Please let me know if you need any additional information or clarification to speed up the review.

[jainxrohit]

Thanks for the comments @jainxrohit and @kaikalur.

@kaikalur to be clear, this design allows for your recommendation of progressive refinement. This design's default behavior is to be rather simple, exactly as you recommended--single table materialized views with simple stale/fresh checks. Progressive enhancements, such as fine grained refresh or unioning up to date partitions with more up to date data, are optional and can be implemented by connectors as needed.

@jainxrohit thanks for the background. I believe this design abstracts complexity and allows for progressive enhancement. I'm keen to maintain momentum on the implementation. Please let me know if you need any additional information or clarification to speed up the review.

I think the main detail I need to understand better is the proposal to move materialized view optimization from the analysis phase to the planning phase. I believe doing this in the planning phase, especially for the use cases Meta wants to support, is really complex. Should we set up some time to discuss the details further?

[tdcmeehan]

@jainxrohit I'll be happy to brainstorm how this might work for your internal infrastructure.