CRT: client silently reports success for failed multipart uploads when S3 returns HTTP 200 with error body, causing undetectable data loss

[ruiarodrigues]

Describe the bug

When an S3-compatible server returns HTTP 200 in the response header but an <Error> XML payload in the body for CompleteMultipartUpload — a documented and valid S3 API behavior — the CRT-backed S3AsyncClient silently reports success. The upload is never committed to storage, the client has no way to detect the failure, and data is lost with no exception thrown.

The CompleteMultipartUpload API reference explicitly documents this behavior:

Note: Processing of a Complete Multipart Upload request could take several minutes to complete. After Amazon S3 begins processing the request, it sends an HTTP response header that contains a 200 OK response. While processing is in progress, Amazon S3 periodically sends white space characters to keep the connection from timing out. Because a request could fail after the initial 200 OK response has been sent, it is important that you check the response body to determine whether the request succeeded.

If you use AWS SDKs, SDKs handle this condition. The SDK will detect this error and retry the request.

The SDK does not handle this condition when using the CRT client. However the non-CRT client handles it properly (see tests to reproduce)

Root cause

The native CRT library (aws-c-s3) correctly detects this case. In s3_meta_request.c, s_should_check_for_error_despite_200_OK() returns true for all request types except GET_OBJECT, causing s_s3_meta_request_error_code_from_response() to parse the response body for XML errors even on HTTP 200. For CompleteMultipartUpload returning <Error><Code>ServiceUnavailable</Code>..., this returns AWS_ERROR_S3_NON_RECOVERABLE_ASYNC_ERROR. The JNI bridge in aws-crt-java also correctly propagates this non-zero error code to the Java onFinished callback.

The bug is in S3CrtResponseHandlerAdapter.handleServiceError():

// services/s3/src/main/java/software/amazon/awssdk/services/s3/internal/crt/S3CrtResponseHandlerAdapter.java

private void handleServiceError(int responseStatus, HttpHeader[] headers, byte[] errorPayload) {
    SdkHttpResponse.Builder errorResponse = populateSdkHttpResponse(..., responseStatus, headers);
    if (requestFailedMidwayOfOtherError(responseStatus)) {
        // Correctly builds and throws an exception
        ...
        failResponseHandlerAndFuture(s3Exception);
    } else {
        initiateResponseHandling(errorResponse.build()); // presents HTTP 200 to Java handler
        onErrorResponseComplete(errorPayload);           // ends with resultFuture.complete(null) ← BUG
    }
}

private boolean requestFailedMidwayOfOtherError(int responseStatus) {
    return responseHandlingInitiated && initialHeadersResponse.statusCode() != responseStatus;
}

For the 200+error-in-body case:

• responseHandlingInitiated is false — the CRT stores the error body internally and never calls onResponseBody, so the Java layer never initiates response handling
• requestFailedMidwayOfOtherError(200) = false && ... = false

The method falls through to onErrorResponseComplete, which routes the error XML through responsePublisher as if it were a successful response body. Since PutObjectResponse has no body schema, the XML is silently consumed, and the call ends with resultFuture.complete(null) — success.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Failure should be reported in this situation when uploading a file. The non-CRT client behaves correctly

Current Behavior

Upload is reported as successful but the file is not available

Reproduction Steps

A self-contained JUnit 5 test using WireMock demonstrates the bug. The stub returns HTTP 200 with <Error><Code>ServiceUnavailable</Code>...</Error> for CompleteMultipartUpload. The CRT client completes without throwing; the Java-based client (multipartEnabled(true)) correctly throws:

// CRT client — assertion FAILS: no exception thrown despite upload silently discarded
assertThatThrownBy(() -> crtClient.putObject(...).join())
    .isInstanceOf(Exception.class);

// Java client — assertion PASSES: correctly throws
assertThatThrownBy(() -> javaClient.putObject(...).join())
    .isInstanceOf(Exception.class);

---

S3CrtMultipartBugReproducerTest.java

Possible Solution

Possible solution (by Claude)

private void handleServiceError(int responseStatus, HttpHeader[] headers, byte[] errorPayload) {
    SdkHttpResponse.Builder errorResponse = populateSdkHttpResponse(..., responseStatus, headers);
    // handleServiceError is only called from handleError, which is only called when crtCode != SUCCESS.
    // If the confirmed error has a 2xx response status, this is the S3 "200 with error in body" case.
    // We must fail directly — onErrorResponseComplete would silently complete the future as success.
    if (requestFailedMidwayOfOtherError(responseStatus) || isSuccessStatus(responseStatus)) {
        AwsServiceException s3Exception = buildS3Exception(responseStatus, errorPayload, errorResponse);
        SdkClientException sdkClientException =
            SdkClientException.create("Request failed during the transfer due to an error returned from S3");
        s3Exception.addSuppressed(sdkClientException);
        failResponseHandlerAndFuture(s3Exception);
        notifyResponsePublisherErrorIfNeeded(s3Exception);
    } else {
        initiateResponseHandling(errorResponse.build());
        onErrorResponseComplete(errorPayload);
    }
}

private static boolean isSuccessStatus(int responseStatus) {
    return responseStatus >= 200 && responseStatus < 300;
}

This is backward-compatible: actual 4xx/5xx error responses are unaffected (isSuccessStatus = false), and network-level errors with no HTTP response never reach handleServiceError (they go through handleIoError).

Additional Information/Context

No response

AWS Java SDK version used

bom 2.42.41

JDK version used

21

Operating System and version

RHEL 10

[bhoradc]

Hi @ruiarodrigues

Thanks for submitting a detailed report. We are investigating into this issue, we'll keep you posted.