customer opt-in for ascii expansion by dibahlfi · Pull Request #48914 · Azure/azure-sdk-for-python

dibahlfi · 2026-09-05T22:23:08Z

Problem-
The Python SDK currently serializes JSON request bodies with ASCII escaping enabled. As a result, non-ASCII characters are expanded before being sent to Cosmos DB.

For example:

日 → \u65e5

A character that requires 3 bytes in UTF-8 becomes 6 bytes after escaping. The service measures the 2 MiB limit against the UTF-8 length of the JSON representation it receives, so documents containing large amounts of CJK or other non-ASCII text can exceed that limit even when their compact UTF-8 representation is well below it.

This blocked customers from ingesting otherwise valid documents.

Solution-
This change adds the following opt-in client option:

CosmosClient(
endpoint,
credential,
enable_compact_utf8_item_writes=True,
)

When enabled, item bodies for create, upsert, replace, patch, and transactional batch operations are serialized using compact UTF-8 instead of ASCII escape sequences.
The default remains False , so existing applications retain their current serialization behavior.

The implementation also:

When the option is left at its default, item-body serialization remains byte-for-byte identical to the previous behavior. The disabled path uses the same json.dumps(data, separators=(",", ":")) call and Content-Length calculation as before. Independently, all PATCH requests now explicitly send the registered application/json-patch+json content type..
Keeps the partition-key and other semantic request headers unchanged. PATCH requests now explicitly send Content-Type: application/json-patch+json unless the caller supplies a content type.
Leaves queries, control-plane requests, scripts, responses, and pre-serialized strings unchanged.
Escapes surrogate code units that cannot be represented in valid UTF-8 while leaving valid Unicode characters compact, and performs that rewrite only when a document actually contains surrogate code units.
Preserves the existing UTF-8 Content-Length behavior, reusing the byte length produced while validating the compact body rather than encoding the full body a second time.
Is configured at the client level only; there is no per-request override.

Customer impact-
Customers can opt in to send large Unicode-heavy documents without unnecessary \uXXXX expansion. A document whose escaped JSON exceeds the 2 MiB request size limit may remain below it when sent as compact UTF-8.

The stored item and the values returned on reads are identical either way, since both encodings describe the same JSON document; only the bytes on the wire differ.

Validation-
Added sync and async coverage for:

Default and opt-in serialization.
Create, upsert, replace, patch, and transactional batch operations.
Correct UTF-8 Content-Length, including reuse of the length computed during validation.
Surrogate code units in keys and values.
Unchanged query, control-plane, header, and pre-serialized-string behavior.
Large CJK documents that fail with HTTP 413 when escaped but succeed with compact UTF-8.
Strict client-option validation and sync/async option wiring.

dibahlfi · 2026-09-05T22:23:43Z

@sdkReviewAgent-2

dibahlfi · 2026-09-05T22:23:52Z

/azp run python - cosmos - tests

azure-pipelines · 2026-09-05T22:24:08Z

Azure Pipelines: Successfully started running 3 pipeline(s). 8 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

azure-pipelines · 2026-09-05T22:24:09Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Note

This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.

dibahlfi · 2026-09-06T00:52:33Z

@sdkReviewAgent-2

dibahlfi · 2026-09-06T02:09:06Z

@sdkReviewAgent-2

dibahlfi · 2026-09-06T02:09:25Z

/azp run python - cosmos - tests

azure-pipelines · 2026-09-06T02:09:50Z

Azure Pipelines: Successfully started running 1 pipeline(s).

dibahlfi · 2026-09-06T02:30:57Z

/azp run python - cosmos - tests

azure-pipelines · 2026-09-06T02:31:14Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Note

This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.

dibahlfi · 2026-09-06T03:08:38Z

/azp run python - cosmos - tests

azure-pipelines · 2026-09-06T03:08:53Z

Azure Pipelines: Successfully started running 1 pipeline(s).

dibahlfi · 2026-09-06T03:08:55Z

@sdkReviewAgent-2

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Note

This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.

dibahlfi · 2026-09-06T04:11:09Z

/azp run python - cosmos - tests

dibahlfi · 2026-09-09T04:27:29Z

/azp run python - cosmos - tests

dibahlfi · 2026-09-09T04:27:43Z

@sdkReviewAgent-2

azure-pipelines · 2026-09-09T04:27:46Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot · 2026-09-09T04:30:18Z

One or more custom setup steps configured for this repository failed during this Copilot code review run:

Install azsdk mcp server

Setup steps run before each review. If the review above is missing context, or no review was posted at all, the failing step above may be the cause. See the workflow run for failure details, fix your setup steps configuration, and re-request a review.

Note

You can configure setup steps for Copilot code review separately from Copilot cloud agent with a copilot-code-review.yml file. Read the docs for details.

Copilot

🟢 Approval recommended

The option is correctly scoped and validated, with comprehensive sync, async, transport, resilience, and live coverage.

Review details

Files reviewed: 28/28 changed files
Comments generated: 0 new
Review effort level: Balanced

Annie Liang (xinlian12) · 2026-09-09T15:45:31Z

✅ Review complete (17:22)

Posted 2 inline comment(s).

_{Steps: ✓ context, correctness, cross-sdk, design, history, past-prs, synthesis, test-coverage}

github-actions · 2026-09-09T16:31:12Z

[Pilot] PR Pipeline Failure Analysis

What failed

Two separate, unrelated failures were detected on PR #48914 (head 141109095e24c07e5e0da81ea7e3bde6445c6ede):

Azure Pipelines check: python - cosmos - native-test (and its Build Analyze sub-check) — both report FAILURE.
- Details URL: https://dev.azure.com/azure-sdk/fadad8ea-1e34-4d7a-9016-fd29b14e8b94/_build/results?buildId=6802860
- This is an internal Azure DevOps pipeline. The automated azsdk ci analyze run returned no azure_pipeline_analyses data and no failed-test artifact content for this build, so no test titles, error text, or file:line detail could be retrieved for this failure.
GitHub Actions workflow: "Running Copilot Code Review" — conclusion: failure
- Run: https://github.com/Azure/azure-sdk-for-python/actions/runs/34310893532
- Log shows the "Install azsdk mcp server" setup step failed with a GitHub API error: API rate limit exceeded for 172.184.210.208. This is an unauthenticated-request rate limit hit during tool installation, not a code or test problem.

Relevant pipeline output

2026-09-09T04:26:40.4296637Z Invoke-RestMethod: /home/runner/work/azure-sdk-for-python/azure-sdk-for-python/eng/common/scripts/Helpers/AzSdkTool-Helpers.ps1:142
Line |
 142 |          $releases = Invoke-RestMethod -Uri $releasesUrl
     |                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | {   "message": "API rate limit exceeded for 172.184.210.208. (But
     | here's the good news: Authenticated requests get a higher rate
     | limit. Check out the documentation for more details.)",
     | "documentation_url":
     | "https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting" }
##[error]Process completed with exit code 1.

Recommended next steps

python - cosmos - native-test (Azure Pipelines): this is an internal Azure DevOps pipeline build that this workflow does not have access to beyond the check status, and it does not have the user's Azure DevOps identity to fetch full logs or test results. To analyze it in detail, run the following locally while authenticated to Azure DevOps (e.g. after az login):
```
azsdk ci analyze https://github.com/Azure/azure-sdk-for-python/pull/48914 --output json
```
"Running Copilot Code Review" workflow: this failed due to an unauthenticated GitHub API rate limit during a setup step (installing the azsdk mcp server), not a defect in the PR's code changes. Recommend simply re-running/retrying this workflow.
See https://aka.ms/ci-fix

Copilot detected the failing pipeline and generated the analysis above. To have it attempt a
fix automatically, reply with @copilot please fix the failing pipeline on this PR.

Generated by Pipeline Analysis Next Steps · auto · 60.2 AIC · ⌖ 2.68 AIC · ⊞ 9.2K · ◷

dibahlfi · 2026-09-09T16:39:24Z

@sdkReviewAgent-2

dibahlfi · 2026-09-09T16:39:43Z

/azp run python - cosmos - tests

azure-pipelines · 2026-09-09T16:40:03Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

🟡 Changes recommended

The new PATCH header uses the RFC media type instead of Cosmos DB’s documented application/json_patch+json value.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (3)

Previously missed (2) — in code that hasn't changed since the last review.

sdk/cosmos/azure-cosmos/tests/test_transport_body_encoding.py:289

This assertion codifies the RFC media type rather than Cosmos DB's required PATCH value, application/json_patch+json. Update it so the transport test catches regressions to an unsupported content type.

This issue also appears on line 366 of the same file.
sdk/cosmos/azure-cosmos/CHANGELOG.md:16

The release note names the wrong PATCH media type. Cosmos DB documents application/json_patch+json (underscore), so this should match the corrected runtime header rather than describe the RFC 6902 hyphenated type.

sdk/cosmos/azure-cosmos/tests/test_transport_body_encoding.py:366

The async assertion should also expect Cosmos DB's required application/json_patch+json value; the hyphenated RFC media type is not the service's documented PATCH content type.

                    "application/json-patch+json",

Files reviewed: 28/28 changed files
Comments generated: 1
Review effort level: Balanced

dibahlfi · 2026-09-09T17:07:52Z

@sdkReviewAgent-2

dibahlfi · 2026-09-09T17:08:03Z

/azp run python - cosmos - tests

azure-pipelines · 2026-09-09T17:08:23Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

🔵 Needs a closer look

Wire serialization and HTTP behavior change across both transports and retry paths, warranting final human validation.

Review details

Files reviewed: 28/28 changed files
Comments generated: 0 new
Review effort level: Balanced

customer opt-in for ascii expansion

bc58314

Copilot AI balanced review requested due to automatic review settings September 5, 2026 22:23

dibahlfi requested a review from a team as a code owner September 5, 2026 22:23

github-actions Bot added the Cosmos label Sep 5, 2026

github-project-automation Bot added this to CosmosDB Python Eco-System Sep 5, 2026

Copilot AI reviewed Sep 5, 2026

This comment has been minimized.

Sign in to view

addressing pylint errors

3c0b3b3

Copilot AI review requested due to automatic review settings September 6, 2026 02:05

Merge branch 'main' into users/dibahl/ascii-expansion-fix

adb25de

Copilot AI reviewed Sep 6, 2026

cspell error fix

92c3b51

Copilot AI review requested due to automatic review settings September 6, 2026 03:08

Copilot AI reviewed Sep 6, 2026

adding samples

5af3263

Copilot AI review requested due to automatic review settings September 6, 2026 04:05

Copilot AI review requested due to automatic review settings September 9, 2026 04:24

Copilot started reviewing on behalf of dibahlfi September 9, 2026 04:25 View session

Copilot AI reviewed Sep 9, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

Annie Liang (xinlian12) reviewed Sep 9, 2026

View reviewed changes

Comment thread sdk/cosmos/azure-cosmos/samples/compact_utf8_item_writes.py Outdated

Annie Liang (xinlian12) reviewed Sep 9, 2026

View reviewed changes

Comment thread sdk/cosmos/azure-cosmos/azure/cosmos/_synchronized_request.py Outdated

Simon Moreno (simorenoh) approved these changes Sep 9, 2026

View reviewed changes

addressing comments

7c8de9e

Copilot AI review requested due to automatic review settings September 9, 2026 16:38

Copilot started reviewing on behalf of dibahlfi September 9, 2026 16:39 View session

Simon Moreno (simorenoh) approved these changes Sep 9, 2026

View reviewed changes

Copilot AI reviewed Sep 9, 2026

View reviewed changes

Comment thread sdk/cosmos/azure-cosmos/azure/cosmos/_runtime_constants.py

Simon Moreno (simorenoh) approved these changes Sep 9, 2026

View reviewed changes

cspell fix

f317734

Copilot AI review requested due to automatic review settings September 9, 2026 17:07

Copilot started reviewing on behalf of dibahlfi September 9, 2026 17:07 View session

Simon Moreno (simorenoh) approved these changes Sep 9, 2026

View reviewed changes

Copilot AI reviewed Sep 9, 2026

View reviewed changes

Uh oh!

Conversation

dibahlfi commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dibahlfi commented Sep 5, 2026

Uh oh!

dibahlfi commented Sep 5, 2026

Uh oh!

azure-pipelines Bot commented Sep 5, 2026

Uh oh!

azure-pipelines Bot commented Sep 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

dibahlfi commented Sep 6, 2026

Uh oh!

This comment has been minimized.

dibahlfi commented Sep 6, 2026

Uh oh!

dibahlfi commented Sep 6, 2026

Uh oh!

azure-pipelines Bot commented Sep 6, 2026

Uh oh!

dibahlfi commented Sep 6, 2026

Uh oh!

azure-pipelines Bot commented Sep 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

dibahlfi commented Sep 6, 2026

Uh oh!

azure-pipelines Bot commented Sep 6, 2026

Uh oh!

dibahlfi commented Sep 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

dibahlfi commented Sep 6, 2026

Uh oh!

dibahlfi commented Sep 9, 2026

Uh oh!

dibahlfi commented Sep 9, 2026

Uh oh!

azure-pipelines Bot commented Sep 9, 2026

Uh oh!

Copilot AI commented Sep 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

Annie Liang (xinlian12) commented Sep 9, 2026

Uh oh!

github-actions Bot commented Sep 9, 2026

What failed

Recommended next steps

Uh oh!

dibahlfi commented Sep 9, 2026

Uh oh!

dibahlfi commented Sep 9, 2026

Uh oh!

azure-pipelines Bot commented Sep 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (3)

Uh oh!

Uh oh!

dibahlfi commented Sep 9, 2026

Uh oh!

dibahlfi commented Sep 9, 2026

Uh oh!

dibahlfi commented Sep 5, 2026 •

edited

Loading