Skip to content

Commit 3b74d2b

Browse files
committed
initial review updates
1 parent ae10668 commit 3b74d2b

File tree

4 files changed

+5
-5
lines changed

4 files changed

+5
-5
lines changed

docs/collect/configure.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@ Tailpipe uses [hive partitioning](https://duckdb.org/docs/data/partitioning/hive
5353

5454
- The data is written to Parquet files in the workspace directory, with a prescribed directory and filename structure. Each partition is written to a separate directory.
5555

56-
- The `tp_index` is used to partition the data and defaults to `"default"` if not specified. You can configure the `tp_index` in your partition config to specify a different value or expression for the partition index. Be aware that defining a `tp_index` does not always increase performance and may, in fact, decrease it as it can result in many small parquet files.
56+
- The `tp_index` is used to partition the data and defaults to `"default"` if not specified. You can configure the `tp_index` in your partition config to specify a column name as the partition index. Be aware that defining a `tp_index` does not always increase performance and may, in fact, decrease it as it can result in many small parquet files.
5757

5858
The standard partitioning/hive structure enables efficient queries that only need to read subsets of the hive filtered by index or date. Because the data is laid out into partitions, performance is optimized when the partition appears in a `where` or `join` clause. The index provides a way to segment the data to optimize lookup performance in a way that is *optimal for your specific use case*. For example, you might index on account ID for AWS tables, subscription for Azure tables, or project ID for GCP tables.
5959

docs/faq/index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,4 +93,4 @@ partition "aws_cloudtrail_log" "cloudtrail_all" {
9393

9494
## What partition indexes are available for a table?
9595

96-
The `tp_index` value depends on how you have configured it in your partition config. By default, `tp_index` is set to `"default"`, but you can configure it to use any value or expression that makes sense for your data. For AWS tables, you might set it to `account_id`.
96+
The `tp_index` value depends on how you have configured it in your partition config. By default, `tp_index` is set to `"default"`, but you can configure it to specify a column name as the partition index that makes sense for your data. For AWS tables, you might set it to `account_id`.

docs/reference/config-files/partition.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ The partition has two labels:
3232
|----------|--------|-----------|-----------------
3333
| `source` | Block | Required | a [source](#source) from which to collect data.
3434
| `filter` | String | Optional | A SQL `where` clause condition to filter log entries. Supports expressions using table columns.
35-
| `tp_index` | String | Optional | The value or expression to use for the partition index. Defaults to `"default"` if not specified. This is used in the [hive partitioning](/docs/collect/configure#hive-partitioning) scheme.
35+
| `tp_index` | String | Optional | The column name to use as the partition index. Defaults to `"default"` if not specified. This is used in the [hive partitioning](/docs/collect/configure#hive-partitioning) scheme.
3636

3737

3838

@@ -179,7 +179,7 @@ partition "aws_cloudtrail_log" "s3_bucket_us_east_1" {
179179
}
180180
```
181181

182-
You can configure the `tp_index` to use a specific value or expression for the partition index:
182+
You can configure the `tp_index` to use a specific column as the partition index:
183183

184184
```hcl
185185
partition "aws_cloudtrail_log" "account_specific" {

docs/reference/config-files/table.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,7 @@ Tailpipe supports most of the [DuckDB general-purpose data types](https://duckdb
111111

112112
Tailpipe tables include a set of common columns. These mappings enable queries that correlate values across different logs. If you have collected both Cloudtrail and ALB logs, for example, you could query for `tp_ips` to find IP addresses in the `aws_cloudtrail_log` and `aws_alb_access_log` tables using the same syntax.
113113

114-
When creating a custom table, `tp_timestamp` is the only required column; ***you must define a `tp_timestamp` column***. This is because Tailpipe uses the timestamp to [organize the data files](/docs/collect/configure#hive-partitioning). The `tp_index` is also used in the hive partitioning scheme. By default, `tp_index` is set to `"default"`, but you can configure it in your partition config to specify a different value or expression for the partition index.
114+
When creating a custom table, `tp_timestamp` is the only required column; ***you must define a `tp_timestamp` column***. This is because Tailpipe uses the timestamp to [organize the data files](/docs/collect/configure#hive-partitioning). The `tp_index` is also used in the hive partitioning scheme. By default, `tp_index` is set to `"default"`, but you can configure it in your partition config to specify a column name as the partition index.
115115

116116
Some of the common columns (`tp_date`,`tp_id`,`tp_ingest_timestamp`,`tp_partition`,`tp_table`) are automatically set by the plugins - You do not need to create them. Others are optional (but encouraged). If you do not set an optional common column, all values will be `null`.
117117

0 commit comments

Comments
 (0)