Skip to content

Conversation

zhongzc
Copy link
Contributor

@zhongzc zhongzc commented Aug 30, 2025

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

Older regions didn’t persist partition_expr in engine metadata; new regions do. To unify behavior and support online upgrade, we backfill partition_expr on region open and persist it to the manifest.

Added on-open backfill to persist partition_expr for legacy regions.

Backfill runs only when metadata.partition_expr is None; no impact on regions already carrying the field.

Manual Test

SQL on release/0.15

CREATE TABLE my_table (
  a INT PRIMARY KEY,
  b STRING,
  ts TIMESTAMP TIME INDEX,
)
PARTITION ON COLUMNS (a) (
  a < 1000,
  a >= 1000 AND a < 2000,
  a >= 2000
);

manifests:

{"actions":[{"Change":{"metadata":{"column_metadatas":[{"column_schema":{"name":"a","data_type":{"Int32":{}},"is_nullable":true,"is_time_index":false,"default_constraint":null,"metadata":{}},"semantic_type":"Tag","column_id":0},{"column_schema":{"name":"b","data_type":{"String":null},"is_nullable":true,"is_time_index":false,"default_constraint":null,"metadata":{}},"semantic_type":"Field","column_id":1},{"column_schema":{"name":"ts","data_type":{"Timestamp":{"Millisecond":null}},"is_nullable":false,"is_time_index":true,"default_constraint":null,"metadata":{"greptime:time_index":"true"}},"semantic_type":"Timestamp","column_id":2}],"primary_key":[0],"region_id":4398046511104,"schema_version":0,"primary_key_encoding":"dense"}}}]}

Restart with this patch, manifests:

{"actions":[{"Change":{"metadata":{"column_metadatas":[{"column_schema":{"name":"a","data_type":{"Int32":{}},"is_nullable":true,"is_time_index":false,"default_constraint":null,"metadata":{}},"semantic_type":"Tag","column_id":0},{"column_schema":{"name":"b","data_type":{"String":null},"is_nullable":true,"is_time_index":false,"default_constraint":null,"metadata":{}},"semantic_type":"Field","column_id":1},{"column_schema":{"name":"ts","data_type":{"Timestamp":{"Millisecond":null}},"is_nullable":false,"is_time_index":true,"default_constraint":null,"metadata":{"greptime:time_index":"true"}},"semantic_type":"Timestamp","column_id":2}],"primary_key":[0],"region_id":4398046511104,"schema_version":0,"primary_key_encoding":"dense","partition_expr":"{\"Expr\":{\"lhs\":{\"Column\":\"a\"},\"op\":\"Lt\",\"rhs\":{\"Value\":{\"Int32\":1000}}}}"}}}]}

PR Checklist

Please convert it to a draft if some of the following conditions are not met.

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR requires documentation updates.
  • API changes are backward compatible.
  • Schema or data changes are backward compatible.

@zhongzc zhongzc requested review from evenyag, v0y4g3r, waynexia and a team as code owners August 30, 2025 02:17
@github-actions github-actions bot added size/M docs-not-required This change does not impact docs. labels Aug 30, 2025
@zhongzc zhongzc force-pushed the zhongzc/mito-backfill-partition-expr branch from af7952c to 1d992e8 Compare August 30, 2025 02:18
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
@zhongzc zhongzc force-pushed the zhongzc/mito-backfill-partition-expr branch from 1d992e8 to 35a064d Compare August 30, 2025 02:20
@zhongzc zhongzc force-pushed the zhongzc/mito-backfill-partition-expr branch from db87ee1 to 8ab1e74 Compare September 1, 2025 03:32
Copy link
Collaborator

@fengjiachun fengjiachun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
@zhongzc zhongzc force-pushed the zhongzc/mito-backfill-partition-expr branch from 970a65f to 2f98ea0 Compare September 4, 2025 10:56
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
@zhongzc zhongzc force-pushed the zhongzc/mito-backfill-partition-expr branch from 7f17b2b to 2e84da8 Compare September 4, 2025 11:18
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Copy link
Member

@WenyXu WenyXu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
@zhongzc zhongzc added this pull request to the merge queue Sep 9, 2025
Merged via the queue into main with commit 9fe84f6 Sep 9, 2025
42 checks passed
@zhongzc zhongzc deleted the zhongzc/mito-backfill-partition-expr branch September 9, 2025 03:20
waynexia pushed a commit to waynexia/greptimedb that referenced this pull request Sep 9, 2025
* feat(mito): backfill partition expr on region open

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* tiny polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* only writable leader persists

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* catchup needs backfill

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* add log

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* handle distribute mode

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* only when transfer to set gracefully

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix fmt

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs-not-required This change does not impact docs. size/M
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants