NIFI-14457 New processor to run xQuery on BaseX #9933

TomaszK-stack · 2025-05-11T10:04:25Z

Summary

Nifi 14457- New processor to run xQuery on BaseX.

Introduced a new EvaluateBaseXQuery for executing XQuery queries against a remote or embedded BaseX XML database instance. This processor allows integration with BaseX for high-performance XML data querying and transformation.

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

Apache NiFi Jira issue created

Pull Request Tracking

Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-XXXXX
Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-XXXXX

Pull Request Formatting

Pull Request based on current revision of the main branch
Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

Build completed using mvn clean install -P contrib-check
- JDK 21

Licensing

BaseX dependency (BSD 3-Clause) verified to be compatible with Apache License 2.0
BaseX dependency added to module-level LICENSE and NOTICE files (nifi-basex-processor-nar)
BaseX added to root nifi-assembly LICENSE and NOTICE files

Documentation

Processor documentation renders correctly
Usage examples and configuration parameters documented

joewitt · 2025-05-11T19:29:56Z

nifi-extension-bundles/nifi-basex-bundle/nifi-basex-nar/pom.xml

+            <url>https://files.basex.org/maven</url>
+        </repository>
+    </repositories>
+    <properties>


these property entries should not be needed and we want to avoid having such things in nars themselves

joewitt · 2025-05-11T19:31:43Z

nifi-extension-bundles/nifi-basex-bundle/nifi-basex-nar/pom.xml

+    <artifactId>nifi-basex-nar</artifactId>
+    <packaging>nar</packaging>
+    <repositories>
+        <repository>


The org.basex artifacts 'appear' to be in maven central. Is it strictly necessary to add a specific one here?

Okay, I changed to the newest one vestion from maven and deleted it.

joewitt · 2025-05-11T19:32:18Z

nifi-extension-bundles/nifi-basex-bundle/nifi-basex-processors/pom.xml

+    <artifactId>nifi-basex-processors</artifactId>
+
+    <properties>
+        <maven.compiler.source>21</maven.compiler.source>


These properties should be removed

joewitt · 2025-05-11T19:32:40Z

nifi-extension-bundles/nifi-basex-bundle/nifi-basex-processors/pom.xml

+        <repository>
+            <id>basex</id>
+            <name>BaseX Maven Repository</name>
+            <url>https://files.basex.org/maven</url>


same comments a repo entry in other pom

joewitt · 2025-05-11T19:33:22Z

nifi-extension-bundles/nifi-basex-bundle/nifi-basex-processors/pom.xml

+        <dependency>
+            <groupId>org.basex</groupId>
+            <artifactId>basex</artifactId>
+            <version>7.6</version>


this version appears to be extremely old. This is a non-starter for inclusion. If we're adding a new component it needs to be on the latest stable release of that component in general or some really strong case to not do so.

Like in the comment above.

joewitt · 2025-05-11T19:39:00Z

...nifi-basex-processors/src/main/java/org/apache/nifi/processors/basex/EvaluateBaseXQuery.java

+                + "'success' (on successful execution), 'failure' (if the script fails), and 'original' (preserves the unmodified FlowFile)."
+)
+@SystemResourceConsiderations({
+        @SystemResourceConsideration(resource = SystemResource.MEMORY, description = "Processing requires reading the entire FlowFile into memory")


it is good this is here - but I'd recommend making it more dramatic in description.

As currently written the entire content of the flowfile is read into memory and I think as written that might be twice the size at least. On serialization it is also at least once but might be twice as well. Either way this is for large XML documents a serious consideration for users.

Okay, I will try to implement it.

joewitt · 2025-05-11T19:47:18Z

nifi-extension-bundles/nifi-basex-bundle/nifi-basex-nar/pom.xml

+    <parent>
+        <groupId>org.apache.nifi</groupId>
+        <artifactId>nifi-basex-bundle</artifactId>
+        <version>2.4.0-SNAPSHOT</version>


all of these will need to be bumped to 2.5.0-SNAPSHOT

ALSO the nifi-extension-bundles pom needs to know about the nifi-basex-bundle pom or it will not be built

joewitt · 2025-05-11T19:49:27Z

...nifi-basex-processors/src/main/java/org/apache/nifi/processors/basex/EvaluateBaseXQuery.java

@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one


[WARNING] src/main/java/org/apache/nifi/processors/basex/EvaluateBaseXQuery.java:[22,43] (imports) AvoidStarImport: Using the '.' form of import should be avoided - org.apache.nifi.annotation.behavior..
[WARNING] src/main/java/org/apache/nifi/processors/basex/EvaluateBaseXQuery.java:[25,34] (imports) AvoidStarImport: Using the '.' form of import should be avoided - org.apache.nifi.components..
[WARNING] src/main/java/org/apache/nifi/processors/basex/EvaluateBaseXQuery.java:[168,13] (whitespace) WhitespaceAround: '}' is not followed by whitespace.
[WARNING] src/main/java/org/apache/nifi/processors/basex/EvaluateBaseXQuery.java:[168,14] (whitespace) WhitespaceAround: 'else' is not preceded with whitespace.
[WARNING] src/main/java/org/apache/nifi/processors/basex/EvaluateBaseXQuery.java:[218,5] (blocks) LeftCurly: '{' at column 5 should be on the previous line.
[WARNING] src/main/java/org/apache/nifi/processors/basex/EvaluateBaseXQuery.java:[233,5] (blocks) LeftCurly: '{' at column 5 should be on the previous line.
[WARNING] src/main/java/org/apache/nifi/processors/basex/EvaluateBaseXQuery.java:[244,88] (whitespace) WhitespaceAround: '{' is not preceded with whitespace.

joewitt · 2025-05-11T20:04:20Z

nifi-extension-bundles/nifi-basex-bundle/nifi-basex-nar/pom.xml

+        <dependency>
+            <groupId>org.apache.nifi</groupId>
+            <artifactId>nifi-standard-shared-nar</artifactId>
+            <version>2.3.0-SNAPSHOT</version>


2.5.0-SNAPSHOT

Okay I changed it.

exceptionfactory

Concurring with feedback from @joewitt, there are a large number of issues to be addressed before this can be considered for inclusion.

The most fundamental issue is the legacy version of the library referenced. The latest version in Maven Central is 11.9 at the time of this writing.

The hard-coded batching approach also raises some implementation questions.

exceptionfactory · 2025-05-12T17:40:39Z

...nifi-basex-processors/src/main/java/org/apache/nifi/processors/basex/EvaluateBaseXQuery.java

+    @Override
+    public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+        Context basexContext = new Context();
+        final List<FlowFile> flowFileBatch = session.get(50);


Is there a particular reason for having this hard-coded batch of 50, as opposed to just handling one FlowFile and supporting batching through annotations?

I copied this strategy from EvaluateXQuery processor, but if it's not needed I can change it to one FlowFile.

joewitt · 2025-05-12T17:44:27Z

Extensive feedback has been provided and considerable work is needed to reconsider for inclusion.

This is very likely a good candidate to exist as its own github project for now to evaluate for inclusion later with adoption/usage and signs of maintenance.

I am closing the PR for now but it can be revisited once feedback has been incorporated.

TomaszK-stack · 2025-05-15T19:31:42Z

After corrections I will prepare new request.

TomaszK-stack added 2 commits May 10, 2025 23:48

Added new EvaluateBaseXQuery Processor in new nifi-basex-module

c7c08f5

updated licenses

b1c7008

joewitt reviewed May 11, 2025

View reviewed changes

exceptionfactory requested changes May 12, 2025

View reviewed changes

joewitt closed this May 12, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NIFI-14457 New processor to run xQuery on BaseX #9933

NIFI-14457 New processor to run xQuery on BaseX #9933

TomaszK-stack commented May 11, 2025

joewitt May 11, 2025

TomaszK-stack May 15, 2025

joewitt May 11, 2025

TomaszK-stack May 15, 2025

joewitt May 11, 2025

TomaszK-stack May 15, 2025

joewitt May 11, 2025

joewitt May 11, 2025

TomaszK-stack May 15, 2025

joewitt May 11, 2025

TomaszK-stack May 15, 2025

joewitt May 11, 2025

TomaszK-stack May 15, 2025

joewitt May 11, 2025

joewitt May 11, 2025

TomaszK-stack May 15, 2025

exceptionfactory left a comment

exceptionfactory May 12, 2025

TomaszK-stack May 15, 2025

joewitt commented May 12, 2025

TomaszK-stack commented May 15, 2025

		@@ -0,0 +1,277 @@
		/*
		* Licensed to the Apache Software Foundation (ASF) under one

NIFI-14457 New processor to run xQuery on BaseX #9933

NIFI-14457 New processor to run xQuery on BaseX #9933

Conversation

TomaszK-stack commented May 11, 2025

Summary

Tracking

Issue Tracking

Pull Request Tracking

Pull Request Formatting

Verification

Build

Licensing

Documentation

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

exceptionfactory left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joewitt commented May 12, 2025

TomaszK-stack commented May 15, 2025