Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
a30fb08
connect to ducklake
kaidaguerre Jun 24, 2025
7d20dc5
Do not clear end objects in TimeRangeCollectionState.OnCollectionComp…
kaidaguerre Jul 2, 2025
6096a96
Add constants.DuckLakeSchema
kaidaguerre Jul 7, 2025
da4c2a3
Implemented compaction
kaidaguerre Jul 8, 2025
146964b
Partition delete uses ducklake
kaidaguerre Jul 8, 2025
3bc2ee6
remove unnecessary db creations, EnsureDatabaseFile, AddTableViews, g…
kaidaguerre Jul 9, 2025
534f280
update getColumnsRenderFunc to use SortColumnsAlphabetically
kaidaguerre Jul 9, 2025
3782119
works but slow - reduced worker count to 1 for now
kaidaguerre Jul 9, 2025
d3e6809
working on it - works but 5 workers does not - about to try add file
kaidaguerre Jul 9, 2025
fb2033e
readded file creation (for now) - about to remove again
kaidaguerre Jul 11, 2025
7f37662
remove file root provider etc
kaidaguerre Jul 11, 2025
29d85ee
ask convertor to insert into ducklake
kaidaguerre Jul 11, 2025
1321da5
go.mod
kaidaguerre Sep 2, 2025
6dae6b3
Untrack test_apps folder and ignore it going forward
kaidaguerre Jul 14, 2025
5a35539
set maxChunksToProcess to 2
kaidaguerre Jul 15, 2025
92b2f4d
set maxChunksToProcess to 20
kaidaguerre Aug 7, 2025
b4b2e01
move manual compact to file
kaidaguerre Aug 8, 2025
b4158e5
Update TODO comments
kaidaguerre Aug 18, 2025
0b1afcd
Update error handling for initGlobalConfig
kaidaguerre Aug 18, 2025
2975ba2
remove support for deprecated events.Error event
kaidaguerre Aug 18, 2025
9e41072
Update comment and remove TODO from TailpipeConnection.ToProto
kaidaguerre Aug 18, 2025
ec460e9
Add duration to CompactionStatus
kaidaguerre Aug 18, 2025
36f5fcf
UUpdate CompactDataFilesManual to fail early if no matches
kaidaguerre Aug 18, 2025
3f5f522
Fix linting errors
kaidaguerre Aug 18, 2025
da88276
comments
kaidaguerre Aug 18, 2025
6f77ce3
Remove tailpipe metadata dir - put db in data dir
kaidaguerre Aug 20, 2025
f79417b
logging
kaidaguerre Aug 20, 2025
b26ac51
partition by year and month
kaidaguerre Aug 21, 2025
d64ce74
go.sum
kaidaguerre Aug 21, 2025
0b5fd1b
delete files
kaidaguerre Aug 21, 2025
e224133
Update comments for s3 code
kaidaguerre Aug 22, 2025
86c2601
remove memtest and add to gitignore
kaidaguerre Aug 22, 2025
fe39013
remove references to tp_date, not that we are partitioning by year an…
kaidaguerre Aug 22, 2025
0d6ba80
working on compaction - about to try new approach
kaidaguerre Aug 25, 2025
7845315
compact a day at a time
kaidaguerre Aug 25, 2025
436c57f
compact a day at a time
kaidaguerre Aug 26, 2025
386fbb4
compact a maximum of 500K rows at a time
kaidaguerre Aug 26, 2025
e8c6394
update maxCompactionRowsPerChunk to 1M
kaidaguerre Aug 27, 2025
d9b8715
disable merge_adjacent_files for now
kaidaguerre Aug 27, 2025
7dcb996
update go-duck_db
kaidaguerre Aug 27, 2025
dde1a52
working on loading rowcounts upfront for compaction progress
kaidaguerre Aug 27, 2025
6543c19
working on compaction status. About to merge partition keys and parti…
kaidaguerre Aug 29, 2025
3de2a42
progress getting there
kaidaguerre Aug 29, 2025
be4c6e1
fix status display
kaidaguerre Aug 29, 2025
962568a
fix status display
kaidaguerre Aug 29, 2025
08b2c29
update disorder metrics to return overlapping files
kaidaguerre Sep 1, 2025
32ff0d8
identify all overlapping files - need to optimize
kaidaguerre Sep 1, 2025
d6bc1b5
about to simplify compaction
kaidaguerre Sep 1, 2025
771e96c
simplified - seems to work
kaidaguerre Sep 1, 2025
9feb1cf
determineChunkingInterval
kaidaguerre Sep 1, 2025
4afc1ba
post rebase tidy
kaidaguerre Sep 2, 2025
c6520d0
linting
kaidaguerre Sep 2, 2025
20ef7b8
use SanitizeDuckDBIdentifier from pipefittings
kaidaguerre Sep 2, 2025
b0e5ccc
build darwin amd for acc tests for now
pskrbasu Aug 22, 2025
b15525c
use ubuntu-latest for Build job
pskrbasu Aug 22, 2025
dc3ad05
try sysroots
pskrbasu Aug 24, 2025
b9a1d8e
try sysroots (2)
pskrbasu Aug 24, 2025
542e169
combine darwin and linux
pskrbasu Aug 25, 2025
0a7860d
buildvcs
pskrbasu Aug 25, 2025
39baafa
fix filename
pskrbasu Aug 25, 2025
dae3c29
fix release workflow
pskrbasu Aug 25, 2025
295f217
Enable and fix the TestLoadTailpipeConfig test (#540)
Priyanka-Chatterjee-2000 Sep 2, 2025
200fc59
Add status spinner for partition deletion in DuckLake (#541)
Priyanka-Chatterjee-2000 Sep 2, 2025
b0178ab
re-add tp_date
kaidaguerre Sep 2, 2025
c187db8
skip tests - to be reenabled later
pskrbasu Sep 2, 2025
7a088ad
comment
kaidaguerre Sep 2, 2025
5580044
add row id filter in deleteUnorderedEntriesForTimeRange
kaidaguerre Sep 2, 2025
399755d
test
kaidaguerre Sep 2, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions .acceptance.goreleaser.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Acceptance testing configuration - builds only Linux AMD64 for faster testing
version: 2

before:
hooks:
- go mod tidy
Expand All @@ -10,12 +13,17 @@ builds:
- amd64

env:
- CC=x86_64-linux-gnu-gcc
- CXX=x86_64-linux-gnu-g++
- CC=x86_64-linux-gnu-gcc-13
- CXX=x86_64-linux-gnu-g++-13
- CGO_CXXFLAGS=-std=c++17
- CGO_LDFLAGS=-lstdc++ -static-libstdc++

ldflags:
- -s -w -X main.version={{.Version}} -X main.date={{.Date}} -X main.commit={{.Commit}} -X main.builtBy=goreleaser

flags:
- -buildvcs=false

archives:
- id: homebrew
format: tar.gz
Expand Down
20 changes: 20 additions & 0 deletions .cursor/rules/general.mdc
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
description:
globs:
alwaysApply: false
---
# general rules to always apply
## confirmation/avoid too much initiative
- DO not make any change I have not explicitly asked for
- NEVER make any changes if I have only asked you a question but not explicitly asked you to make an action
- Ask for confirmation before making ANY changes, with a summary of what you will do
## format
- Use lower case for sql always
## general attitude
- Use a neutral tone of voice and do not be too positive/enthusiastic.
- When I report a problem, do NOT say "perfect I see the problem" as that sounds like you know the solution
- When you have made a change do NOT say "now everything will be working" until you have confirmation that it does work
- Always look at my ideas and suggestions critically and look for flaws in my logic
-


58 changes: 58 additions & 0 deletions .darwin.goreleaser.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Darwin-only goreleaser configuration
version: 2

before:
hooks:
- go mod tidy

builds:
# Darwin AMD64 build with clang
- id: tailpipe-darwin-amd64
binary: tailpipe
goos:
- darwin
goarch:
- amd64
env:
- CC=o64-clang
- CXX=o64-clang++
ldflags:
- -s -w -X main.version={{.Version}} -X main.date={{.Date}} -X main.commit={{.Commit}} -X main.builtBy=goreleaser
flags:
- -buildvcs=false

# Darwin ARM64 build with clang
- id: tailpipe-darwin-arm64
binary: tailpipe
goos:
- darwin
goarch:
- arm64
env:
- CC=oa64-clang
- CXX=oa64-clang++
ldflags:
- -s -w -X main.version={{.Version}} -X main.date={{.Date}} -X main.commit={{.Commit}} -X main.builtBy=goreleaser
flags:
- -buildvcs=false

archives:
- id: darwin
format: tar.gz
name_template: "{{ .ProjectName }}.{{ .Os }}.{{ .Arch }}"
files:
- none*

checksum:
name_template: 'checksums.txt'

snapshot:
name_template: "{{ incpatch .Version }}-next"

changelog:
disable: true
sort: asc
filters:
exclude:
- '^docs:'
- '^test:'
2 changes: 2 additions & 0 deletions .github/workflows/01-tailpipe-release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,8 @@ jobs:
- name: Release publish
run: |-
cd tailpipe
git config --global user.name "Tailpipe GitHub Actions Bot"
git config --global user.email noreply@github.com
make release

create_pr_in_homebrew:
Expand Down
15 changes: 11 additions & 4 deletions .github/workflows/11-test-acceptance.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,13 @@ jobs:
go clean -testcache
go test -timeout 30s ./... -test.v

- name: Build Custom Goreleaser Cross Image for Linux
run: |-
cd tailpipe
make build-goreleaser-image
echo "=== Verifying GCC 13 ==="
docker run --rm tailpipe-goreleaser-cross:gcc13 gcc-13 --version | head -1

- name: Build
run: |-
cd tailpipe
Expand Down Expand Up @@ -125,12 +132,12 @@ jobs:
name: build-artifact-linux
path: ~/artifacts

- name: Extract Ubuntu Artifacts and Install Binary
- name: Extract Linux Artifacts and Install Binary
if: ${{ matrix.platform == 'ubuntu-latest' }}
run: |
mkdir ~/build
tar -xf ~/artifacts/linux.tar.gz -C ~/build

- name: Set PATH
run: |
echo "PATH=$PATH:$HOME/build:$GTIHUB_WORKSPACE/tailpipe/tests/acceptance/lib/bats-core/libexec" >> $GITHUB_ENV
Expand Down Expand Up @@ -183,8 +190,8 @@ jobs:
name: build-artifact-linux
failOnError: true

- name: Clean up Darwin Build
- name: Clean up Linux Build
uses: geekyeggo/delete-artifact@f275313e70c08f6120db482d7a6b98377786765b # v5.1.0
with:
name: build-artifact-darwin
name: build-artifact-linux
failOnError: true
8 changes: 6 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@
*.dll
*.so
*.dylib

/test_apps/
/memtest
# Editor cache and lock files
*.swp
*.swo
Expand All @@ -28,4 +29,7 @@
go.work

# Dist directory is created by goreleaser
/dist
/dist

# Sysroot directory is created by make build-sysroot
/sysroot
1 change: 1 addition & 0 deletions .golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,4 @@ run:
issues:
exclude-dirs:
- "tests/acceptance"
- "test_apps"
64 changes: 20 additions & 44 deletions .goreleaser.yml
Original file line number Diff line number Diff line change
@@ -1,67 +1,43 @@
version: 2

builds:
- id: tailpipe-linux-arm64
binary: tailpipe
goos:
- linux
goarch:
- arm64

env:
- CC=aarch64-linux-gnu-gcc
- CXX=aarch64-linux-gnu-g++

# Custom ldflags.
#
# Default: '-s -w -X main.version={{.Version}} -X main.commit={{.Commit}} -X main.date={{.Date}} -X main.builtBy=goreleaser'
# Templates: allowed
ldflags:
# Go Releaser analyzes your Git repository and identifies the most recent Git tag (typically the highest version number) as the version for your release.
# This is how it determines the value of {{.Version}}.
- -s -w -X main.version={{.Version}} -X main.date={{.Date}} -X main.commit={{.Commit}} -X main.builtBy=goreleaser
before:
hooks:
- go mod tidy

builds:
# Linux AMD64 build with GCC 13+
- id: tailpipe-linux-amd64
binary: tailpipe
goos:
- linux
goarch:
- amd64

env:
- CC=x86_64-linux-gnu-gcc
- CXX=x86_64-linux-gnu-g++

- CC=x86_64-linux-gnu-gcc-13
- CXX=x86_64-linux-gnu-g++-13
- CGO_CXXFLAGS=-std=c++17
- CGO_LDFLAGS=-lstdc++ -static-libstdc++
ldflags:
- -s -w -X main.version={{.Version}} -X main.date={{.Date}} -X main.commit={{.Commit}} -X main.builtBy=goreleaser
flags:
- -buildvcs=false

- id: tailpipe-darwin-arm64
# Linux ARM64 build with GCC 13+
- id: tailpipe-linux-arm64
binary: tailpipe
goos:
- darwin
- linux
goarch:
- arm64

env:
- CC=oa64-clang
- CXX=oa64-clang++

ldflags:
- -s -w -X main.version={{.Version}} -X main.date={{.Date}} -X main.commit={{.Commit}} -X main.builtBy=goreleaser

- id: tailpipe-darwin-amd64
binary: tailpipe
goos:
- darwin
goarch:
- amd64

env:
- CC=o64-clang
- CXX=o64-clang++

- CC=aarch64-linux-gnu-gcc-13
- CXX=aarch64-linux-gnu-g++-13
- CGO_CXXFLAGS=-std=c++17
- CGO_LDFLAGS=-lstdc++ -static-libstdc++
ldflags:
- -s -w -X main.version={{.Version}} -X main.date={{.Date}} -X main.commit={{.Commit}} -X main.builtBy=goreleaser
flags:
- -buildvcs=false

release:
prerelease: auto
Expand Down
40 changes: 40 additions & 0 deletions Dockerfile.goreleaser-cross
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Use Ubuntu 24.04 as base to get GCC 13+
FROM ubuntu:noble

# Install essential packages and cross-compilation tools
RUN apt-get update && apt-get install -y \
wget curl git build-essential \
gcc-13 g++-13 \
gcc-aarch64-linux-gnu g++-aarch64-linux-gnu \
gcc-x86-64-linux-gnu g++-x86-64-linux-gnu \
gcc-arm-linux-gnueabihf g++-arm-linux-gnueabihf \
&& rm -rf /var/lib/apt/lists/*

# Install Go 1.24.5
RUN wget https://go.dev/dl/go1.24.5.linux-amd64.tar.gz && \
tar -C /usr/local -xzf go1.24.5.linux-amd64.tar.gz && \
rm go1.24.5.linux-amd64.tar.gz

# Install goreleaser 2.11.2
RUN wget https://github.com/goreleaser/goreleaser/releases/download/v2.11.2/goreleaser_Linux_x86_64.tar.gz && \
tar -xzf goreleaser_Linux_x86_64.tar.gz && \
mv goreleaser /usr/local/bin/ && \
rm goreleaser_Linux_x86_64.tar.gz

# Set up environment
ENV PATH="/usr/local/go/bin:${PATH}"
ENV GOROOT="/usr/local/go"
ENV GOPATH="/go"
ENV CGO_ENABLED=1

# Set default cross-compilation environment variables for Linux AMD64
ENV CC_linux_amd64=x86_64-linux-gnu-gcc-13
ENV CXX_linux_amd64=x86_64-linux-gnu-g++-13
ENV CGO_CXXFLAGS_linux_amd64="-std=c++17"
ENV CGO_LDFLAGS_linux_amd64="-lstdc++ -static-libstdc++"

# Create working directory
WORKDIR /go/src

# Set entrypoint to goreleaser
ENTRYPOINT ["goreleaser"]
Loading