[Feature][Config] Support nested references to external files in configuration files #8984

thirsd · 2025-03-16T03:08:04Z

thanks to @hailin0 https://github.com/hailin0.

reference the design:
#8945 (comment)

Does this PR introduce any user-facing change?

yes
user can use ref config file to reuse config in job config file.

#8941

How was this patch tested?

i have tested unset ref file .
i have tested set ref file and use in mysql to console and sqlite to console.

Check list

[x ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[x ] If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config
[ x] Update the release-note.

…set path of ref_config, then you can use __st_config_ref_key__ in source\transform\sink to merge ref config.

liunaijie · 2025-03-17T01:12:51Z

docs/zh/concept/config.md

+source {
+  Jdbc {
+    __st_config_ref_key__ = "mysql_prod"
+    query="select * from department"


From the code, looks if the mysql_prod ref config also contains query parameter, it will overwrite the existed query parameter.

pluginConfig.withoutPath("__st_config_ref_key__").withFallback(refConfig);

Please add the priority for replacements when there are duplicate parameters.

got it

We can just describe it in the document to let use know this thing. Replace behavior is good to me.

已经补充了配置优先级。

…nces and add descriptions of ref configuration and plugin configuration priorities.

liunaijie · 2025-03-20T00:59:49Z

Please add test case

… ref_config

wuchunfu · 2025-03-27T03:19:23Z

docs/en/concept/config.md

+```hocon
+# 定义 MySQL 连接配置
+mysql_prod {


@thirsd Please describe in English here, thanks.

wuchunfu · 2025-03-27T03:20:07Z

docs/en/concept/config.md

+
+# 定义 Kafka 连接配置
+kafka_test {


litiliu · 2025-03-27T08:48:56Z

...nel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigShadeUtils.java

+            Map<String, Map<String, Object>> refMap = new HashMap<>();
+            // get map element in ref
+            for (String key : configMap.keySet()) {
+                Object ref_config = configMap.get(key);


we should name the variables in the camelcase style.

...nel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigShadeUtils.java

litiliu · 2025-03-27T09:02:25Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/options/EnvCommonOptions.java

@@ -107,4 +107,10 @@ public class EnvCommonOptions {
                    .mapType()
                    .noDefaultValue()
                    .withDescription("Define the worker where the job runs by tag");
+
+    public static Option<String> REF_PATH =
+            Options.key("__st_config_ref_path__")


__st_config_ref_path__ looks weird, how about naming it to config_ref_path?

Not surprising, he cannot conflict with normal attributes

Copilot

Pull Request Overview

This PR introduces configuration reference support, allowing users to reuse shared configuration files in their job configurations. Key changes include adding a new environment option for reference config paths, merging plugin and reference configurations in the job parser, and updating tests and documentation to verify and explain the new behavior.

Reviewed Changes

Copilot reviewed 8 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java	Introduces refMaps field and merging logic for reference configs
seatunnel-engine-client/src/test/java/org/apache/seatunnel/engine/client/MultipleTableJobConfigParserTest.java	Adds tests for parsing job configs using the ref config feature
seatunnel-core/seatunnel-core-starter/src/test/java/org/apache/seatunnel/core/starter/utils/ConfigShadeTest.java	Extends tests to cover parsing and merging of reference configs
seatunnel-core/seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigShadeUtils.java	Adds processing branch to handle configuration references
seatunnel-api/src/main/java/org/apache/seatunnel/api/options/EnvCommonOptions.java	Adds a new environment option for specifying the ref config file path
docs/*	Updates both Chinese and English documentation to explain configuration reference functionality

Files not reviewed (4)

seatunnel-core/seatunnel-core-starter/src/test/resources/mysql_to_console_by_ref.conf: Language not supported
seatunnel-core/seatunnel-core-starter/src/test/resources/ref.conf: Language not supported
seatunnel-engine/seatunnel-engine-client/src/test/resources/fake_to_console_by_ref.conf: Language not supported
seatunnel-engine/seatunnel-engine-client/src/test/resources/ref.conf: Language not supported

Comments suppressed due to low confidence (1)

seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java:164

[nitpick] The variable name 'RefPath' does not follow the Java camelCase naming convention. Consider renaming it to 'refPath'.

String RefPath = envOptions.getOptional(EnvCommonOptions.REF_PATH).orElse(null);

...-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java

…e/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

thirsd added 4 commits March 15, 2025 20:20

Merge remote-tracking branch 'upstream/dev' into dev

9d0a299

support ref_config. in env option, support __st_config_ref_path__ to …

8294c05

…set path of ref_config, then you can use __st_config_ref_key__ in source\transform\sink to merge ref config.

Code Style Format

ed29abf

add doc

8aefcf6

github-actions bot added document core SeaTunnel core module Zeta api labels Mar 16, 2025

thirsd mentioned this pull request Mar 16, 2025

[Feature] Support connection config, and use conn_id to replace conn info in job config #8945

Closed

4 tasks

liunaijie reviewed Mar 17, 2025

View reviewed changes

thirsd added 4 commits March 18, 2025 22:50

update config.md，增加ref配置和plugin配置优先级的描述。

9c0fd04

update docs/en/concept/config.md，Add examples of configuration refere…

73b4e0a

…nces and add descriptions of ref configuration and plugin configuration priorities.

Merge remote-tracking branch 'upstream/dev' into ref_config

5ebd068

Merge branch 'apache:dev' into ref_config

55972ae

thirsd added 5 commits March 23, 2025 00:47

add test examples

580ecd7

Merge remote-tracking branch 'upstream/dev' into ref_config

e4b41f7

Merge branch 'apache:dev' into ref_config

ccc1d22

Merge branch 'ref_config' of https://github.com/thirsd/seatunnel into…

6b1ad30

… ref_config

add license header

7ab6ec7

wuchunfu reviewed Mar 27, 2025

View reviewed changes

docs/en/concept/config.md

Comment on lines +357 to +359

# 定义 Kafka 连接配置

kafka_test {

Copy link

Member

wuchunfu Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

davidzollo added the First-time contributor First-time contributor label Mar 27, 2025

litiliu reviewed Mar 27, 2025

View reviewed changes

...nel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigShadeUtils.java Show resolved Hide resolved

litiliu reviewed Mar 27, 2025

View reviewed changes

hailin0 requested a review from Copilot March 31, 2025 05:17

Copilot AI reviewed Mar 31, 2025

View reviewed changes

...-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java Outdated Show resolved Hide resolved

Update seatunnel-engine/seatunnel-engine-core/src/main/java/org/apach…

2973521

…e/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

hailin0 changed the title ~~Ref config~~ [Feature][Config] Support nested references to external files in configuration files Apr 2, 2025

hailin0 mentioned this pull request Apr 2, 2025

SeaTunnel community meeting topic collect #9100

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Config] Support nested references to external files in configuration files #8984

[Feature][Config] Support nested references to external files in configuration files #8984

thirsd commented Mar 16, 2025

liunaijie Mar 17, 2025

thirsd Mar 18, 2025

liunaijie Mar 18, 2025

thirsd Mar 18, 2025

liunaijie commented Mar 20, 2025

wuchunfu Mar 27, 2025

wuchunfu Mar 27, 2025

litiliu Mar 27, 2025

litiliu Mar 27, 2025

hailin0 Mar 28, 2025

Copilot AI left a comment

[Feature][Config] Support nested references to external files in configuration files #8984

Are you sure you want to change the base?

[Feature][Config] Support nested references to external files in configuration files #8984

Conversation

thirsd commented Mar 16, 2025

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liunaijie commented Mar 20, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes