Skip to content

[OPA Plugin] Support splitting OPA filter requests in batch mode by batch size #25753

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

thinaih
Copy link
Member

@thinaih thinaih commented May 8, 2025

Description

This PR aims to address issue #25748

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( X ) Release notes are required. Please propose a release note for me.
() Release notes are required, with the following suggested text:

## Section
* Fix some things. ({issue}`issuenumber`)

Copy link

cla-bot bot commented May 8, 2025

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Thinhinane Ihadadene.
This is most likely caused by a git client misconfiguration; please make sure to:

  1. check if your git client is configured with an email to sign commits git config --list | grep email
  2. If not, set it up using git config --global user.email email@example.com
  3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

@thinaih thinaih force-pushed the chunk-batch-requests branch from 7634089 to 069d83d Compare May 8, 2025 09:57
@cla-bot cla-bot bot added the cla-signed label May 8, 2025
@thinaih thinaih changed the title Support chunking for OPA batched queries [OPA Plugin] Support splitting OPA filter requests in batch mode by batch size May 8, 2025
@thinaih thinaih force-pushed the chunk-batch-requests branch 2 times, most recently from b725506 to fe95395 Compare May 8, 2025 10:18
@thinaih
Copy link
Member Author

thinaih commented May 8, 2025

The CI seems to be failing due to a missing JAVA_HOME environment variable, I'd appreciate if someone could help with this.

@thinaih thinaih marked this pull request as ready for review May 8, 2025 11:18
Comment on lines 86 to 94
assertAccessControlMethodBehaviour(
(accessControl, systemSecurityContext, identities) -> accessControl.filterViewQueryOwnedBy(systemSecurityContext.getIdentity(), identities),
inputIdentities,
identityToJsonNode,
new ExpectedRequestProperties(
testCase.batchSizeToBatchNumber,
testCase.numberOfRequest,
"FilterViewQueryOwnedBy",
inputIdentities.stream().map(identityToJsonNode).collect(toImmutableSet())),
testCase.opaConfig);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we avoid using this abstraction and rather have these values explicitly like

assertAccessControlMethodBehaviour(situation1);
assertAccessControlMethodBehaviour(situation2);
...

So it is easier to understand on what are the impacts

Copy link
Member Author

@thinaih thinaih May 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mind clarifying if you mean the TestCase abstraction or ExpectedRequestProperties? I assume you mean TestCase

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I've seen in previous PRs, what Praveen is refering to is doing something like

@Test
public void testFilterViewQueryOwnedBy()
{
    ...
    assertFilterViewQueryOwnedBy(new TestCase(OPA_CONFIG_NO_BATCH_SIZE, ImmutableMap.of(11, 1), 1));
    ...
}

void assertFilterViewQueryOwnedBy(TestCase testCase)
{
            assertAccessControlMethodBehaviour(
                    (accessControl, systemSecurityContext, identities) -> accessControl.filterViewQueryOwnedBy(systemSecurityContext.getIdentity(), identities),
                    inputIdentities,
                    identityToJsonNode,
                    new ExpectedRequestProperties(
                            testCase.batchSizeToBatchNumber,
                            testCase.numberOfRequest,
                            "FilterViewQueryOwnedBy",
                            inputIdentities.stream().map(identityToJsonNode).collect(toImmutableSet())),
                    testCase.opaConfig);
}

The nice thing about this is that when a case fails the traceback will point you the line that failed, so you don't have to figure out which element in the for loop failed

@thinaih thinaih force-pushed the chunk-batch-requests branch 2 times, most recently from abfd602 to af7a59c Compare May 13, 2025 13:47
@thinaih thinaih force-pushed the chunk-batch-requests branch from af7a59c to e15090a Compare May 13, 2025 13:59
Comment on lines 238 to 241
ImmutableSet.of("\"table_one_column_1\"", "\"table_one_column_2\"", "\"table_one_column_3\"", "\"table_two_column_1\"")
.stream()
.map(RequestTestUtilities::toJsonNode)
.collect(toImmutableSet())));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we could add a toJsonNode util that takes an object so we don't have to double quote the string

Comment on lines 86 to 94
assertAccessControlMethodBehaviour(
(accessControl, systemSecurityContext, identities) -> accessControl.filterViewQueryOwnedBy(systemSecurityContext.getIdentity(), identities),
inputIdentities,
identityToJsonNode,
new ExpectedRequestProperties(
testCase.batchSizeToBatchNumber,
testCase.numberOfRequest,
"FilterViewQueryOwnedBy",
inputIdentities.stream().map(identityToJsonNode).collect(toImmutableSet())),
testCase.opaConfig);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I've seen in previous PRs, what Praveen is refering to is doing something like

@Test
public void testFilterViewQueryOwnedBy()
{
    ...
    assertFilterViewQueryOwnedBy(new TestCase(OPA_CONFIG_NO_BATCH_SIZE, ImmutableMap.of(11, 1), 1));
    ...
}

void assertFilterViewQueryOwnedBy(TestCase testCase)
{
            assertAccessControlMethodBehaviour(
                    (accessControl, systemSecurityContext, identities) -> accessControl.filterViewQueryOwnedBy(systemSecurityContext.getIdentity(), identities),
                    inputIdentities,
                    identityToJsonNode,
                    new ExpectedRequestProperties(
                            testCase.batchSizeToBatchNumber,
                            testCase.numberOfRequest,
                            "FilterViewQueryOwnedBy",
                            inputIdentities.stream().map(identityToJsonNode).collect(toImmutableSet())),
                    testCase.opaConfig);
}

The nice thing about this is that when a case fails the traceback will point you the line that failed, so you don't have to figure out which element in the for loop failed

@thinaih thinaih requested a review from mosiac1 May 19, 2025 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

3 participants