fix(oci): allow unlimited layer size

[KarstenSiemer]

This fix gathers the value for unlimited from the tar lib and passes it to the untar function to allow layer sizes bigger than the default of 100MB.

fixes #1511 as discussed.
Currently I did not implement tests. If this is wanted, I'll gladly add them if you would be okay with v1beta2 of github.com/fluxcd/source-controller/api.

I have this code running now with OCIs as big as 500MB for about 2 weeks. It is running flawless, I am really happy with this controller. Thanks for all the efforts, this makes a huge difference for me :-)

Thanks for your consideration.

[akselleirv]

Hello @KarstenSiemer, thank you for your contribution and the research you have done to verify that this works as intended. Yes, please feel free to add tests with OCIRepository using the v1beta2 version.

[KarstenSiemer]

Hello @akselleirv,

first of all my question regarding v1beta2 was very redundant. I should have spend more time looking into it before asking the question.

I have added the test and kept close to #16 where a test for the bucket source was implemented and I thought that this makes the most sense to test ocirepository in general as we gain a longer term testing template for other sources which might be implemented in the future.

As direct testing for this exact code change, setting the layer size limit to unlimited, I think we have this covered quite well with the overall testing suite, as most of them abstact away from the source controller via a tar archive. This means that all included tests that do something with a tar as source (Git, OCI, Bucket), test this very change.

Last note for me is that I required to set skip_requesting_account_id to true in the aws backend in test tc000111, as the backend would otherwise error and pointed out a link to the aws provider where you get advised to do this change. As I don't have an AWS account, nor tools installed which have any information about it, the functions in the backend would still try to look it up in the test, which results in the failure, unless setting skip_requesting_account_id.

Error without skip_requesting_account_id = true:

=== RUN   Test_000111_with_backend_s3_no_outputs_test
SPEC: This spec describes the behaviour of a Terraform resource with backend configured, and `auto` approve.
IT should be reconciled from the plan state, to the apply state and have a correct TFSTATE stored inside the cluster as a Secret.
GIVEN: a GitRepository
BY: defining a new Git repository resource.
BY: creating the GitRepository resource in the cluster.
IT should be created successfully.
GIVEN: the GitRepository's reconciled status.
BY: setting the GitRepository's status, with the downloadable BLOB's URL, and the correct checksum.
IT should be updated successfully.
IT should be retrievable by the k8s client.
BY: getting it with the k8s client and succeed.
BY: preparing s3-backend-configs secret
time="2025-02-12T16:48:33+01:00" level=info msg="starting localstack"
time="2025-02-12T16:48:33+01:00" level=info msg="waiting for localstack to start..."
time="2025-02-12T16:48:40+01:00" level=info msg="localstack: finished waiting"
BY: creating the S3 bucket.
BY: creating the DynamoDB table.
GIVEN: a Terraform resource with auto approve, backend configured, attached to the given GitRepository.
BY: creating a new TF resource and attaching to the repo via `sourceRef`.
IT should be created and attached successfully.
BY: checking that the TF resource existed inside the cluster.
IT should be reconciled and contain some status conditions.
BY: checking that the TF resource's status conditions has some elements.
IT should apply successfully.
BY: checking that the status of the TF resource is `TerraformAppliedSucceed`.

Error: Retrieving AWS account details: AWS account ID not previously found and failed retrieving via all available methods.

See https://www.terraform.io/docs/providers/aws/index.html#skip_requesting_account_id for workaround and implications.
Errors: 2 errors occurred:
        * retrieving caller identity from STS: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 909fabf0-4bea-4d42-b3eb-b359aaba00a7, api error InvalidClientTokenId: The security token included in the request is invalid.
        * retrieving account information via iam:ListRoles: operation error IAM: ListRoles, https response error StatusCode: 403, RequestID: ff588db9-6895-47cd-ab24-1c3ee78a0aa0, api error InvalidClientTokenId: The security token included in the request is invalid.




Error: Retrieving AWS account details: AWS account ID not previously found and failed retrieving via all available methods.

See https://www.terraform.io/docs/providers/aws/index.html#skip_requesting_account_id for workaround and implications.
Errors: 2 errors occurred:
        * retrieving caller identity from STS: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 05a72646-e265-4521-8084-e5b443fe3350, api error InvalidClientTokenId: The security token included in the request is invalid.
        * retrieving account information via iam:ListRoles: operation error IAM: ListRoles, https response error StatusCode: 403, RequestID: 934e13a5-07df-4ac9-be69-a9cdc3269e2e, api error InvalidClientTokenId: The security token included in the request is invalid.



    tc000111_with_backend_s3_no_outputs_test.go:302: 
        Timed out after 30.000s.
        Expected
            <map[string]interface {} | len:0>: nil
        to equal
            <map[string]interface {} | len:2>: {
                "Type": <string>"Apply",
                "Reason": <string>"TerraformAppliedSucceed",
            }
--- FAIL: Test_000111_with_backend_s3_no_outputs_test (43.50s)

Test with skip_requesting_account_id = true:

=== RUN   Test_000111_with_backend_s3_no_outputs_test
SPEC: This spec describes the behaviour of a Terraform resource with backend configured, and `auto` approve.
IT should be reconciled from the plan state, to the apply state and have a correct TFSTATE stored inside the cluster as a Secret.
GIVEN: a GitRepository
BY: defining a new Git repository resource.
BY: creating the GitRepository resource in the cluster.
IT should be created successfully.
GIVEN: the GitRepository's reconciled status.
BY: setting the GitRepository's status, with the downloadable BLOB's URL, and the correct checksum.
IT should be updated successfully.
IT should be retrievable by the k8s client.
BY: getting it with the k8s client and succeed.
BY: preparing s3-backend-configs secret
time="2025-02-12T16:51:40+01:00" level=info msg="starting localstack"
time="2025-02-12T16:51:40+01:00" level=info msg="waiting for localstack to start..."
time="2025-02-12T16:51:47+01:00" level=info msg="localstack: finished waiting"
BY: creating the S3 bucket.
BY: creating the DynamoDB table.
GIVEN: a Terraform resource with auto approve, backend configured, attached to the given GitRepository.
BY: creating a new TF resource and attaching to the repo via `sourceRef`.
IT should be created and attached successfully.
BY: checking that the TF resource existed inside the cluster.
IT should be reconciled and contain some status conditions.
BY: checking that the TF resource's status conditions has some elements.
IT should apply successfully.
BY: checking that the status of the TF resource is `TerraformAppliedSucceed`.
IT should be reconciled successfully, and have some output available.
BY: checking the output reason of the TF resource being `TerraformOutputsAvailable`.
BY: checking that the .status.availableOutputs contains an output named `hello_world`.
--- PASS: Test_000111_with_backend_s3_no_outputs_test (13.48s)

If you are against that change (Test_000111_with_backend_s3_no_outputs_test), I'll remove the commit from this PR.

Thanks for your consideration. I had great fun working on this

[akselleirv]

LGTM! Thank you very much @KarstenSiemer 🥇

Glad to hear that you enjoyed working on this contribution! I hope you consider making further contributions to this project.

(I'll fix the failing trivy job on main)