Skip to content

Commit 874ce1a

Browse files
committed
Update collect docs
1 parent e251fa4 commit 874ce1a

File tree

2 files changed

+23
-3
lines changed

2 files changed

+23
-3
lines changed

docs/collect/collect.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,18 @@ Collect a specific partition.
2626
tailpipe collect aws_cloudtrail_log.dev
2727
```
2828

29+
Collect a specific time range.
30+
31+
```bash
32+
tailpipe collect aws_cloudtrail_log.test --from 2024-01-01 --to 2024-01-31
33+
```
34+
35+
Collect and overwrite existing data for a time range.
36+
37+
```bash
38+
tailpipe collect aws_cloudtrail_log.test --from 2024-01-01 --to 2024-01-31 --overwrite
39+
```
40+
2941
See [collect](/docs/reference/cli/collect) for more examples.
3042

3143

@@ -51,4 +63,6 @@ tailpipe collect aws_cloudtrail_log.test --from T-180d
5163
tailpipe collect aws_cloudtrail_log.test --from 2024-01-01
5264
```
5365

66+
When using the `--from` flag, existing partition data is preserved by default. This means that data will not be recollected for time ranges that have already been collected. To recollect data for a time range, use the `--overwrite` flag.
67+
5468
Subsequent collection runs occur chronologically, resuming from the last collection by default, so there are no time gaps while the data is being collected.

docs/reference/cli/collect.md

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ title: tailpipe collect
77
[Run collection](/docs/collect/collect).
88

99

10-
To improve the first-run experience for collection, Tailpipe will only collect the last 7 days during the [initial collection](/docs/collect/collect#initial-collection) (though you can override this behavior woth the `--from` argument). Subsequent collection runs occur chronologically, resuming from the last collection by default, so there are no time gaps while the data is being collected.
10+
To improve the first-run experience for collection, Tailpipe will only collect the last 7 days during the [initial collection](/docs/collect/collect#initial-collection) (though you can override this behavior with the `--from` and `--to` arguments). Subsequent collection runs occur chronologically, resuming from the last collection by default, so there are no time gaps while the data is being collected.
1111

1212

1313

@@ -23,7 +23,9 @@ To improve the first-run experience for collection, Tailpipe will only collect t
2323
| `--compact` | Compact the Parquet files after collection (default true)
2424
| `--from string` | Collect days newer than a relative or absolute date.
2525
| `--help` | Help for collect
26+
| `--overwrite` | Overwrite existing data for the specified time range
2627
| `--progress` | Show active progress of collection, set to `false` to disable (default `true`)
28+
| `--to string` | Collect days older than a relative or absolute date (use with `--from` for time ranges)
2729

2830

2931

@@ -53,13 +55,17 @@ Collect all partitions in the `aws_cloudtrail_log` table for the last 45 days.
5355
tailpipe collect aws_cloudtrail_log --from T-45d
5456
```
5557

56-
<!--
5758
Collect all partitions in the `aws_cloudtrail_log` between January and June.
5859

5960
```bash
6061
tailpipe collect aws_cloudtrail_log --from 1/1/2024 --to 6/30/2024
6162
```
62-
-->
63+
64+
Collect and overwrite existing data for a specific time range.
65+
66+
```bash
67+
tailpipe collect aws_cloudtrail_log --from 1/1/2024 --to 6/30/2024 --overwrite
68+
```
6369

6470
Collect all partitions in the `aws_cloudtrail_log` table and output JSON.
6571

0 commit comments

Comments
 (0)