Update prereqs to note that to use `duckdb` to directly query a Tailpipe DB you need >= v1.1.3

[cbruno10]

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stalled for 90 days with no activity.

[Priyanka-Chatterjee-2000]

@cbruno10 I have tested it recently with duckdb v1.0.0 it works:

duckdb /Users/priyankachatterjee/.tailpipe/data/default/tailpipe_20250718193917.db
v1.0.0 1f98600c2c
Enter ".help" for usage hints.
D select * from aws_alb_connection_log;
┌─────────────────┬─────────────┬──────────────────────┬──────────────────────┬───┬──────────────────────┬───────────┬──────────────────────┬──────────────┐
│    client_ip    │ client_port │    conn_trace_id     │ leaf_client_cert_s…  │ … │       tp_table       │  tp_tags  │     tp_timestamp     │ tp_usernames │
│     varchar     │    int64    │       varchar        │       varchar        │   │       varchar        │ varchar[] │      timestamp       │  varchar[]   │
├─────────────────┼─────────────┼──────────────────────┼──────────────────────┼───┼──────────────────────┼───────────┼──────────────────────┼──────────────┤
│ 45.79.181.104   │       21260 │ TID_8a6cb03381a204…  │                      │ … │ aws_alb_connection…  │           │ 2025-02-26 11:49:0…  │              │
│ 45.79.181.104   │       21272 │ TID_70074efa782d05…  │                      │ … │ aws_alb_connection…  │           │ 2025-02-26 11:49:0…  │              │
│ 45.79.181.94    │       44280 │ TID_06f0f7c7e5f542…  │                      │ … │ aws_alb_connection…  │           │ 2025-02-26 11:49:0…  │              │
│ 45.79.181.94    │       52274 │ TID_5c0a741c3a74c1…  │                      │ … │ aws_alb_connection…  │           │ 2025-02-26 11:49:0…  │              │
│ 54.196.247.85   │       48752 │ TID_ebee297568297f…  │                      │ … │ aws_alb_connection…  │           │ 2025-02-26 10:30:4…  │              │
│ 54.196.247.85   │       49600 │ TID_ed23ed9911c1f8…  │                      │ … │ aws_alb_connection…  │           │ 2025-02-26 10:31:3…  │              │
│ 54.196.247.85   │       49614 │ TID_315f415e4f833e…  │                      │ … │ aws_alb_connection…  │           │ 2025-02-26 10:31:3…  │              

Since DuckDB already queries the Tailpipe database without issue, it doesn’t seem necessary to document any extra requirements. That said, we could include a brief note on we can directly use duckdb to query a tailpipe database. Would you like me to add that since we do not have that?

[Priyanka-Chatterjee-2000]

@cbruno10, sorry for the confusion. I initially tested on a table without any JSON columns, which worked fine. When I tried DuckDB v1.0.0 on a table with JSON columns, it failed. I’ve since retested using DuckDB v1.1.3 and everything works as expected. I’ve updated the Hub docs in PR

duckdb /Users/priyankachatterjee/.tailpipe/data/default/tailpipe_20250721183700.db
v1.0.0 1f98600c2c
Enter ".help" for usage hints.
D select * from aws_cloudtrail_log;
Binder Error: Contents of view were altered: types don't match! Expected [JSON, VARCHAR, VARCHAR, JSON, VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR, TIMESTAMP, VARCHAR, VARCHAR, BOOLEAN, BOOLEAN, VARCHAR, VARCHAR, JSON, JSON, JSON, BIGINT, JSON, VARCHAR, VARCHAR, VARCHAR, STRUCT(cipher_suite VARCHAR, client_provided_host_header VARCHAR, tls_version VARCHAR), VARCHAR, STRUCT(access_key_id VARCHAR, account_id VARCHAR, arn VARCHAR, identity_provider VARCHAR, invoked_by VARCHAR, principal_id VARCHAR, session_context STRUCT(attributes STRUCT(creation_date BIGINT, mfa_authenticated VARCHAR), ec2_role_delivery VARCHAR, session_issuer STRUCT(account_id VARCHAR, arn VARCHAR, principal_id VARCHAR, "type" VARCHAR, user_name VARCHAR), web_id_federation_data STRUCT(attributes JSON, federated_provider VARCHAR)), "type" VARCHAR, user_name VARCHAR), VARCHAR, VARCHAR[], DATE, VARCHAR, VARCHAR[], VARCHAR[], VARCHAR, VARCHAR, TIMESTAMP, VARCHAR[], VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR[], TIMESTAMP, VARCHAR[]], but found [VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR, TIMESTAMP, VARCHAR, VARCHAR, BOOLEAN, BOOLEAN, VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR, BIGINT, VARCHAR, VARCHAR, VARCHAR, VARCHAR, STRUCT(cipher_suite VARCHAR, client_provided_host_header VARCHAR, tls_version VARCHAR), VARCHAR, STRUCT(access_key_id VARCHAR, account_id VARCHAR, arn VARCHAR, identity_provider VARCHAR, invoked_by VARCHAR, principal_id VARCHAR, session_context STRUCT(attributes STRUCT(creation_date BIGINT, mfa_authenticated VARCHAR), ec2_role_delivery VARCHAR, session_issuer STRUCT(account_id VARCHAR, arn VARCHAR, principal_id VARCHAR, "type" VARCHAR, user_name VARCHAR), web_id_federation_data STRUCT(attributes VARCHAR, federated_provider VARCHAR)), "type" VARCHAR, user_name VARCHAR), VARCHAR, VARCHAR[], DATE, VARCHAR, VARCHAR[], VARCHAR[], VARCHAR, VARCHAR, TIMESTAMP, VARCHAR[], VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR[], TIMESTAMP, VARCHAR[]] instead

duckdb /Users/priyankachatterjee/.tailpipe/data/default/tailpipe_20250721183700.db
v1.1.3 19864453f7
Enter ".help" for usage hints.
D select user_identity from aws_cloudtrail_log;
┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                                 user_identity                                                                                                 │
│ struct(access_key_id varchar, account_id varchar, arn varchar, identity_provider varchar, invoked_by varchar, principal_id varchar, session_context struct(attributes struct(creation_date bigint, mfa_auth…  │
├───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ {'access_key_id': ASIAZGW7IOFIEN4EWI74, 'account_id': 888888888888, 'arn': arn:aws:sts::888888888888:assumed-role/AWSServiceRoleForConfig/ConfigResourceCompositionSession, 'identity_provider': NULL, 'inv…  │
│ {'access_key_id': ASIAZGW7IOFIGWKSG6AK, 'account_id': 888888888888, 'arn': arn:aws:sts::888888888888:assumed-role/AWSServiceRoleForConfig/ConfigResourceCompositionSession, 'identity_provider': NULL, 'inv…  │