Skip to content

Fix UTF-8 offsets in sequence patterns - #873

Closed
yangfan-yf-yf wants to merge 1 commit into
python-rope:masterfrom
yangfan-yf-yf:fix/match-sequence-utf8-offsets
Closed

yangfan-yf-yf wants to merge 1 commit into
python-rope:masterfrom
yangfan-yf-yf:fix/match-sequence-utf8-offsets

Conversation

@yangfan-yf-yf

Copy link
Copy Markdown
Contributor

Description

I reproduced a crash on master (f005ac786da7d556fdf5733e9b9ddae1eaf27242) when patching a sequence pattern containing non-ASCII text:

from rope.refactor import patchedast

source = "match value:\n    case [é, y]:\n        pass\n"
patchedast.get_patched_ast(source, True)

The walker reads : as the closing delimiter and raises MismatchedTokenError. Nested and empty sequence patterns after non-ASCII text are also affected, and some inputs produce incorrect source regions without raising.

AST columns are UTF-8 byte offsets, while the walker slices a Python string using character offsets. I convert the AST coordinates at the sequence-pattern slicing sites, including the empty-pattern path. The generic SourceLinesAdapter coordinate behavior is preserved.

The regression test covers seven patterns, including non-ASCII captures and literals, nested and empty sequences, bare sequences, and multiline input. It checks both source round-tripping and each sequence node's source region against ast.get_source_segment.

Related to #623; follows the sequence delimiter handling introduced in #863. Delimiter warnings for adjacent comments remain outside this change.

Validation on Windows / Python 3.12.14:

  • All seven new regression scenarios fail before the fix and pass after it.
  • Related modules: 311 passed, 10 skipped; seven subtests passed.
  • Full suite: 2117 passed, 12 skipped, 5 xfailed; seven subtests passed.
  • Target-file pre-commit checks and git diff --check pass.

Checklist

  • I have added tests that prove my fix is effective.
  • I have updated CHANGELOG.md.

@lieryan

lieryan commented Sep 15, 2026

Copy link
Copy Markdown
Member

Hi @yangfan-yf-yf, thank you for the contribution here. This issue is already fixed in #872, which had just been merged, so I'm going to close this PR as it is a duplicate. The test case you provided for this issue passes on #872. Please reopen a new issue if there's any other related cases that you might have that didn't get addressed by #872.

@lieryan lieryan closed this Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants