fix(sfn): handle pagination and ignore errors for expired executions history

[pdecat]

This PR updates the aws_sfn_state_machine_execution_history table to implement pagination and ignore errors for expired executions history, which are only preserved for a maximum of 90 days.

Without this latter change, one would get errors like this one:

Error: operation error SFN: GetExecutionHistory, https response error StatusCode: 400, RequestID: 68dacbfc-af53-42f5-985e-a019c51b679f, ExecutionDoesNotExist: Execution Does Not Exist: 'arn:aw
s:states:eu-west-3:012345678901:execution:sfn-test-012345678901:b1b58243-1276-4195-863a-e83bed3dd603' (SQLSTATE HV000)

Integration test logs

Logs

Add passing integration test logs here

Example query results

Results

Add example SQL query results here (please include the input queries as well)

[cbruno10]

@pdecat Thanks for opening this PR! Can you please see my comments/questions?

[cbruno10]

Is there a known error code for this error type? If so, could we add it to an IgnoreConfig in the ListConfig, similar to

steampipe-plugin-aws/aws/table_aws_ec2_instance.go

Lines 29 to 31 in d7dac4d

	IgnoreConfig: &plugin.IgnoreConfig{
	ShouldIgnoreErrorFunc: shouldIgnoreErrors([]string{"InvalidInstanceID.NotFound", "InvalidInstanceID.Unavailable", "InvalidInstanceID.Malformed"}),
	},

(this example is in a GetConfig, but ListConfig supports adding the same thing)?

The ShouldIgnoreErrorFunc in the config has a similar effect of returning nil if we encounter from the function.

[pdecat]

I guess ExecutionDoesNotExist should be known: https://github.com/aws/aws-sdk-go-v2/blob/v1.21.2/service/sfn/types/errors.go#L166

[pdecat]

I've tried this change:

diff --git a/aws/table_aws_sfn_state_machine_execution_history.go b/aws/table_aws_sfn_state_machine_execution_history.go
index c6ae08ce..9384b73a 100644
--- a/aws/table_aws_sfn_state_machine_execution_history.go
+++ b/aws/table_aws_sfn_state_machine_execution_history.go
@@ -2,7 +2,6 @@ package aws

 import (
        "context"
-       "errors"
        "strconv"
        "strings"
        "sync"
@@ -10,7 +9,6 @@ import (
        "github.com/aws/aws-sdk-go-v2/aws"
        "github.com/aws/aws-sdk-go-v2/service/sfn"
        "github.com/aws/aws-sdk-go-v2/service/sfn/types"
-       "github.com/aws/smithy-go"

        sfnv1 "github.com/aws/aws-sdk-go/service/sfn"

@@ -27,6 +25,9 @@ func tableAwsStepFunctionsStateMachineExecutionHistory(_ context.Context) *plugi
                        Hydrate:       listStepFunctionsStateMachineExecutionHistories,
                        Tags:          map[string]string{"service": "states", "action": "ListExecutions"},
                        ParentHydrate: listStepFunctionsStateMachines,
+                       IgnoreConfig: &plugin.IgnoreConfig{
+                               ShouldIgnoreErrorFunc: shouldIgnoreErrors([]string{"ExecutionDoesNotExist"}),
+                       },
                },
                GetMatrixItemFunc: SupportedRegionMatrix(sfnv1.EndpointsID),
                Columns: awsRegionalColumns([]*plugin.Column{
@@ -373,15 +374,6 @@ func getRowDataForExecutionHistory(ctx context.Context, d *plugin.QueryData, arn
                plugin.Logger(ctx).Trace("aws_sfn_state_machine_execution_history.getRowDataForExecutionHistory", "api_call GetExecutionHistory", arn)
                output, err := paginator.NextPage(ctx)
                if err != nil {
-                       var apiErr smithy.APIError
-                       if errors.As(err, &apiErr) {
-                               switch apiErr.(type) {
-                               case *types.ExecutionDoesNotExist:
-                                       // Ignore expired executions for which history is no longer available
-                                       plugin.Logger(ctx).Trace("aws_sfn_state_machine_execution_history.getRowDataForExecutionHistory", "api_error ignore_expired", err)
-                                       return nil, nil
-                               }
-                       }
                        plugin.Logger(ctx).Error("aws_sfn_state_machine_execution_history.getRowDataForExecutionHistory", "api_error", err)
                        return nil, err
                }

But it does not seem to work:

> with constants (r, s_arn, e_arn) as (
    values (
            'eu-west-3',
            'arn:aws:states:eu-west-3:012345678901:stateMachine:mysfn',
            'arn:aws:states:eu-west-3:012345678901:execution:mysfn:%'
        )
)
select id,
    execution_arn,
    execution_failed_event_details
from aws_sfn_state_machine_execution_history,
    constants
where region = r
    and execution_arn in (
        select execution_arn
        from aws_sfn_state_machine_execution,
            constants
        where region = r
            and state_machine_arn = s_arn
            and execution_arn like e_arn
    )
    and execution_failed_event_details is null;

Error: operation error SFN: GetExecutionHistory, https response error StatusCode: 400, RequestID: 0ce49d57-9f15-4636-8398-8ddaf01af24b, ExecutionDoesNotExist: Execution Does Not Exist: 'arn:aws:states:eu-west-3:012345678901:execution:mysfn:a4fdeb5a-0f8f-4bee-b36b-2ae036665817' (SQLSTATE HV000)

+----+---------------+--------------------------------+
| id | execution_arn | execution_failed_event_details |
+----+---------------+--------------------------------+
+----+---------------+--------------------------------+

[cbruno10]

Thanks @pdecat for testing (sorry for the long response time!), I think this is due to a bug where IgnoreConfig doesn't work when a ParentHydrate is used (turbot/steampipe-plugin-sdk#544), so can you please add a comment mentioning why we need to handle errors here?

[pdecat]

Done!

[cbruno10]

Does the AWS SDK offer any pagination with svc.GetExecutionHistory? Or was this table never iterating through pages before?

[pdecat]

This table did not implement the required pagination handling before: https://docs.aws.amazon.com/sdk-for-go/api/service/sfn/#SFN.GetExecutionHistory

[github-actions]

This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[pdecat]

Not stale