[HUDI-8817] fix backward compatibility issue

[Davis-Zhang-Onehouse]

Change Logs

when we specify write table version as 6

CREATE TABLE hudi_14_table_sql_005 (
    event_id INT,
    event_date STRING,
    event_name STRING,
    event_ts STRING,
    event_type STRING
) USING hudi
 OPTIONS(
    type = 'cow', -- or 'mor'
    primaryKey = 'event_id,event_date',
    preCombileField = 'event_ts',
    hoodie.write.table.version = 6
)
PARTITIONED BY (event_type)
LOCATION 's3://<some_bucket>/hudi_14_table_sql_005';

the hoodie.properties said the version is 8.

It is because at SQL create table code path we didn't use hoodie.write.table.version to config hoodie.table.version. Now we have the following rules:

• If both hoodie.write.table.version and hoodie.table.version are set, hoodie.table.version prevails
• If only one of them is set explicitly, that one will be used to configure hoodie.table.version.

Impact

hoodie.write.table.version will also contribute to hoodie.table.version.

Risk level (write none, low medium or high below)

none

Documentation Update

none

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[hudi-bot]

CI report:

• 640ffe5 UNKNOWN
• a64c849 UNKNOWN
• 944075b Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[yihua]

The logic in meta client should not consider write table version. Otherwise, there is no point checking whether hoodie.write.table.version and hoodie.table.version match.

CommonClientUtils#validateTableVersion has the validation. Does Spark SQL writer escape that?

[Davis-Zhang-Onehouse]

yes, SQL writer does not call this function at all for create table stmt