Add support for ducklake #519
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…lete if granularity is zero TimeRangeObjectState.Validate does not validate TimeRange if granularity is zero
move SortColumnsAlphabetically to sdk comment out doConversionForBatch and add insertIntoDucklakeForBatch Add TableSchemaStatus to check for schema change conversion into ducklake works - compaction breaks it
Add --to to partition delete move compact and delete into new maintenance.go
…etColumnNames, createAndDropEmptyView Collector now has a db instance - it uses for clearing partition NewParquetConverter accepts db Move partition deletion to collector.Collect try to create db at highest level and pass down doCompaction, CompactDataFiles, ListPartitionResources, GetPartitionResource, DeletePartition,ListPartitions, ListTableResources,, GetTableResource accept db remove all references to prev tailpipe db Query command now just opens a duck db instance rather than using generateDbFile rename DuckLakeSchema to DuckLakeCatalog DuckDb now calls USE on ducklacke db simplify introspection to use metadata, remove GetRowCount interactive HandlerInput noow has db, passed from client
remove unnecessary extension load statements delete JSON files after processing Add TAILPIPE_DATA_DIR for manual override of data-dir - can use S3 path e.g. s3://tp-ducklake-test/ Hack - disable partition data deletion for s3 Add CompactDataFilesManual for manual compaction with ordering (not tested yet) limit ducklake conversion to 5 chunks per query partition by month component of date - slightly slower reverted conn change untested synthetic threading changes reverted conn change duckDb stores a conneciton - about to revert working on it adding synthetic Update Connect to return ducklake connection string Deprecate filter params from Connect move DuckLakeCatalog and DuckLakeMetadataCatalog consts to pipe-fittings move ConnectDucklake to pipe-fittings Tidy Add issues to all ducklake TODOs add ducklake changes md re-add call to handlePluginEvent from listenToEvents rather than handling inline as this was not working for some reason Converter.processChunks updated to correctly break out of the loop when there are no further chunks to process converted to new conversion - not receiving chunk events refactor tests simplified conbversion - works in same time as test app propert generic struct code, re-add column mapping add column name mapping to test app custom code for cloudtrail structs about to add mapping remove compaction (for now) remove ordering and row number from temp table remove partition filter (where) update test app to insert columns to ducklake add struct pack to test app and up date cloudtrail schema to all fields about to remove json() from tailpipe and add as to test app time logging for conversion update test schema to match time logging for converison remove mutex remove partitionKeysPerConversion remove out of mem handliung added files added dates, debugging timing added dates, debuggin g timing revert ask convertor to insert into ducklake working on conversion test app test apps
add logging
update logging
Update PartitionDelete status output to shows rows dleeted, not files Update collection timing display to show compation AND conversion separately
graza-io
approved these changes
Aug 21, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.