Ensure compatibility with latest botocore S3 client customizations

[bpandola]

The latest version of botocore adds some customizations to the S3 client that break some of our S3 tests:

• S3 client behavior is updated to always calculate a CRC32 checksum by default for operations that support it (such as PutObject or UploadPart), or require it (such as DeleteObjects).
• Botocore will no longer automatically compute and populate the Content-MD5 header.

This PR removes assertions around the default checksum behavior (as it now varies across botocore versions). It also manually supplies the ContentMD5 parameter for API calls that require it, as botocore no longer automatically computes it.

This PR also contains minor request.body processing tweaks for S3 as well as during proxy or recording mode, due to the new aws-chunked encoding and the AwsChunkedWrapper object that now wraps some S3 request bodies sent from boto3/botocore.

Note

This PR fixes the moto build breakages that occurred as a result of the new "default integrity protections" introduced in the AWS SDK for Python v1.36.0. It also maintains existing compatibility with earlier SDK versions. There may be additional issues encountered by users of moto as a result of these AWS changes, which will have to be addressed in subsequent PRs as they arise.

Ref: boto/boto3#4392
Ref: boto/botocore@590103b

[codecov]

Codecov Report

Attention: Patch coverage is 71.42857% with 4 lines in your changes missing coverage. Please review.

Project coverage is 92.63%. Comparing base (67b915f) to head (794725e).
Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
moto/moto_proxy/proxy3.py	0.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8495      +/-   ##
==========================================
- Coverage   92.63%   92.63%   -0.01%     
==========================================
  Files        1225     1224       -1     
  Lines      105921   105922       +1     
==========================================
- Hits        98123    98120       -3     
- Misses       7798     7802       +4     

Flag	Coverage Δ
servertests	27.75% <7.14%> (-0.02%)	⬇️
unittests	92.60% <71.42%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[bblommers]

Awesome, thank you for tackling this @bpandola!

[jeking3]

Where Content-MD5 was required before, it is not any more, but only on S3 proper. Third party S3-like services will still require it. I'm not sure the third party S3-like services support trailing checksums either, and certainly won't support the new CRC64 mode.

Do we need a way for whatever is running moto to specify the "flavor" of S3 it is emulating now?

[jeking3]

Caused by: boto/boto3#4392

[bpandola]

The diff here looks more complicated than it truly is. I just reordered the if statements so that the chunked encoding takes precedence over the Content-Length header. The code within these conditionals was not changed at all.

This change was necessary because some of the intercepted S3 requests from boto3/botocore now default to a chunked encoding (that also include a Content-Length header).

[bpandola]

Where Content-MD5 was required before, it is not any more, but only on S3 proper. Third party S3-like services will still require it. I'm not sure the third party S3-like services support trailing checksums either, and certainly won't support the new CRC64 mode.

Do we need a way for whatever is running moto to specify the "flavor" of S3 it is emulating now?

It's not clear to me if anything changed server-side at AWS. The changes in question all seem to affect the client requests generated by boto3/botocore. As such, this PR just updates moto to deal with the new client request format (for some S3 API endpoints). This PR does not make any modifications to any moto responses.

If you run into a specific issue with moto, related to this change or not, please open a new issue with the details. Thanks!

[bblommers]

I've released moto==5.0.27.dev16 if people want to try this out