Merge pull request #398 from Integration-Automation/feat/ax-text-batch

Add ax_text: native TextPattern reads (document / selection / visible)

Author: JE-Chen

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-24) — Native Text Reading via the UIA TextPattern (document / selection / visible)
+
+Read the text in multiline editors and document controls where ValuePattern returns nothing. Full reference: [`docs/source/Eng/doc/new_features/v182_features_doc.rst`](docs/source/Eng/doc/new_features/v182_features_doc.rst).
+
+- **`get_control_text` / `get_selected_text` / `get_visible_text`** (`AC_get_control_text`, `AC_get_selected_text`, `AC_get_visible_text`): `control_get_value` reads through UIA ValuePattern, which returns an empty string on multiline edits, RichEdit / document controls and web text areas — exactly the controls whose text you most want. This reads through `TextPattern` instead: `get_control_text` returns the whole `DocumentRange`, `get_selected_text` the current `GetSelection`, `get_visible_text` only the on-screen `GetVisibleRanges`. Dispatched through the injectable `accessibility.backends.get_backend()` seam (headless-testable via a fake backend; real UIA calls in the Windows backend), returning `{text}` from the executor/MCP. No `PySide6`.
+
 ## What's new (2026-06-24) — Extended UIA Control Patterns (Expand / Select / Range / Scroll)
 
 Drive tree nodes, list/combo items, sliders and scroll natively, not by pixel guessing. Full reference: [`docs/source/Eng/doc/new_features/v181_features_doc.rst`](docs/source/Eng/doc/new_features/v181_features_doc.rst).

docs/source/Eng/doc/new_features/v182_features_doc.rst

@@ -0,0 +1,46 @@
+Native Text Reading via the UIA TextPattern (document / selection / visible)
+============================================================================
+
+``control_get_value`` reads a control through UIA ValuePattern, but ValuePattern
+returns an **empty string** on multiline edits, RichEdit / document controls and
+web text areas — exactly the controls whose text you most want to read. UIA
+exposes that text through a different pattern, ``TextPattern``, which models the
+control's content as text ranges. ``ax_text`` adds three reads on top of the
+existing accessibility backend ABC:
+
+* :func:`get_control_text` — the whole document's text (``DocumentRange``),
+* :func:`get_selected_text` — the currently selected text (``GetSelection``),
+* :func:`get_visible_text` — only the on-screen text (``GetVisibleRanges``).
+
+Each function is a thin dispatch onto the injectable
+``accessibility.backends.get_backend()`` seam (the same seam the rest of the
+accessibility module uses), so the headless core is unit-testable on any
+platform by injecting a fake backend; the real UI Automation calls live in the
+Windows backend. Backends that don't implement TextPattern raise
+``AccessibilityNotAvailableError``. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import (get_control_text, get_selected_text,
+                                 get_visible_text)
+
+    # A multiline editor where control_get_value returns "" :
+    text = get_control_text(name="Editor", role="document")
+    selection = get_selected_text(name="Editor")   # "" when nothing selected
+    on_screen = get_visible_text(name="Editor")    # skips scrolled-off lines
+
+All locate the control by ``name`` / ``role`` / ``app_name`` / ``automation_id``
+(same as ``control_get_value`` / ``control_invoke``). Each returns the text as a
+``str``, or ``None`` when the control is not found or exposes no TextPattern;
+``get_selected_text`` returns ``""`` when the control is found but has no
+selection.
+
+Executor commands
+-----------------
+
+``AC_get_control_text`` / ``AC_get_selected_text`` / ``AC_get_visible_text`` each
+return ``{"text": ...}``. They are exposed as the matching read-only ``ac_*`` MCP
+tools and as Script Builder commands under **Native UI**.

docs/source/Zh/doc/new_features/v182_features_doc.rst

@@ -0,0 +1,41 @@
+透過 UIA TextPattern 讀取原生文字(文件 / 選取 / 可見)
+=======================================================
+
+``control_get_value`` 透過 UIA ValuePattern 讀取控制項,但 ValuePattern 在多行編輯框、
+RichEdit / 文件控制項與網頁文字區塊上會回傳**空字串**——而這些正是你最想讀取其文字的控制項。
+UIA 透過另一個模式 ``TextPattern`` 提供這些文字,它把控制項內容建模為文字範圍(text range)。
+``ax_text`` 在既有的無障礙後端 ABC 之上補上三種讀取:
+
+* :func:`get_control_text` ——整份文件的文字(``DocumentRange``),
+* :func:`get_selected_text` ——目前選取的文字(``GetSelection``),
+* :func:`get_visible_text` ——僅螢幕上可見的文字(``GetVisibleRanges``)。
+
+每個函式都是對可注入的 ``accessibility.backends.get_backend()`` 接縫的薄分派(與無障礙模組
+其餘部分相同的接縫),因此無頭核心可在任何平台透過注入 fake backend 單元測試;真正的
+UI Automation 呼叫位於 Windows 後端。未實作 TextPattern 的後端會拋出
+``AccessibilityNotAvailableError``。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import (get_control_text, get_selected_text,
+                                 get_visible_text)
+
+    # 一個 control_get_value 會回傳 "" 的多行編輯框:
+    text = get_control_text(name="Editor", role="document")
+    selection = get_selected_text(name="Editor")   # 沒有選取時回傳 ""
+    on_screen = get_visible_text(name="Editor")    # 略過捲動到畫面外的列
+
+全部以 ``name`` / ``role`` / ``app_name`` / ``automation_id`` 定位控制項(與
+``control_get_value`` / ``control_invoke`` 相同)。各函式以 ``str`` 回傳文字,找不到控制項或
+控制項未提供 TextPattern 時回傳 ``None``;``get_selected_text`` 在找到控制項但沒有選取時
+回傳 ``""``。
+
+執行器指令
+----------
+
+``AC_get_control_text`` / ``AC_get_selected_text`` / ``AC_get_visible_text`` 各自回傳
+``{"text": ...}``。皆以對應的唯讀 ``ac_*`` MCP 工具及 Script Builder 指令(位於 **Native UI**
+分類下)形式提供。

je_auto_control/__init__.py

@@ -53,6 +53,10 @@
     collapse_control, control_expand_state, control_range, expand_control,
     scroll_control_into_view, select_control_item, set_control_range,
 )
+# Native text reads via UIA TextPattern (document / selection / visible)
+from je_auto_control.utils.ax_text import (
+    get_control_text, get_selected_text, get_visible_text,
+)
 # VLM element locator (headless)
 from je_auto_control.utils.vision import (
     VLMNotAvailableError, click_by_description, locate_by_description,
@@ -1617,6 +1621,7 @@ def start_autocontrol_gui(*args, **kwargs):
     "expand_control", "collapse_control", "control_expand_state",
     "select_control_item", "control_range", "set_control_range",
     "scroll_control_into_view",
+    "get_control_text", "get_selected_text", "get_visible_text",
     # VLM locator
     "VLMNotAvailableError", "locate_by_description", "click_by_description",
     "verify_description",

je_auto_control/gui/script_builder/command_schema.py

@@ -1523,6 +1523,21 @@ def _add_native_control_specs(specs: List[CommandSpec]) -> None:
         fields=fields,
         description="Scroll a control into view (ScrollItemPattern).",
     ))
+    specs.append(CommandSpec(
+        "AC_get_control_text", "Native UI", "Get Control Text",
+        fields=fields,
+        description="Read full text via TextPattern (multiline / document safe).",
+    ))
+    specs.append(CommandSpec(
+        "AC_get_selected_text", "Native UI", "Get Selected Text",
+        fields=fields,
+        description="Read the currently selected text via TextPattern.",
+    ))
+    specs.append(CommandSpec(
+        "AC_get_visible_text", "Native UI", "Get Visible Text",
+        fields=fields,
+        description="Read only the on-screen text via TextPattern.GetVisibleRanges.",
+    ))
 
 
 def _add_misc_specs(specs: List[CommandSpec]) -> None:

je_auto_control/utils/accessibility/backends/base.py

@@ -100,6 +100,29 @@ def scroll_into_view(self, name: Optional[str] = None,
         """Scroll the matched control into view (ScrollItemPattern); True on success."""
         self._unsupported("scroll_into_view")
 
+    # --- text patterns (TextPattern reads) ---------------------------------
+
+    def document_text(self, name: Optional[str] = None, role: Optional[str] = None,
+                      app_name: Optional[str] = None,
+                      automation_id: Optional[str] = None) -> Optional[str]:
+        """Return the matched control's full text (TextPattern), or None.
+
+        Reads multiline / document controls where ValuePattern returns ``""``.
+        """
+        self._unsupported("document_text")
+
+    def selected_text(self, name: Optional[str] = None, role: Optional[str] = None,
+                      app_name: Optional[str] = None,
+                      automation_id: Optional[str] = None) -> Optional[str]:
+        """Return the control's currently selected text (TextPattern), or None."""
+        self._unsupported("selected_text")
+
+    def visible_text(self, name: Optional[str] = None, role: Optional[str] = None,
+                     app_name: Optional[str] = None,
+                     automation_id: Optional[str] = None) -> Optional[str]:
+        """Return only the on-screen text of the control (TextPattern), or None."""
+        self._unsupported("visible_text")
+
     def _unsupported(self, operation: str):
         """Raise a clear error for an action this backend can't perform."""
         raise AccessibilityNotAvailableError(

je_auto_control/utils/accessibility/backends/windows_backend.py

@@ -29,6 +29,7 @@
 _UIA_SELECTIONITEM_PATTERN_ID = 10010
 _UIA_RANGEVALUE_PATTERN_ID = 10003
 _UIA_SCROLLITEM_PATTERN_ID = 10017
+_UIA_TEXT_PATTERN_ID = 10014
 _EXPAND_STATES = {0: "collapsed", 1: "expanded", 2: "partial", 3: "leaf"}
 
 
@@ -263,6 +264,50 @@ def get_range(self, name=None, role=None, app_name=None,
         except (OSError, AttributeError, ValueError, TypeError):
             return None
 
+    def _text_pattern(self, name, role, app_name, automation_id):
+        """Find a control and return its IUIAutomationTextPattern, or None."""
+        raw = self._find_raw(name, role, app_name, automation_id)
+        if not raw:
+            return None
+        return self._pattern(raw, _UIA_TEXT_PATTERN_ID,
+                             "IUIAutomationTextPattern")
+
+    def document_text(self, name=None, role=None, app_name=None,
+                      automation_id=None) -> Optional[str]:
+        pattern = self._text_pattern(name, role, app_name, automation_id)
+        if pattern is None:
+            return None
+        try:
+            return str(pattern.DocumentRange.GetText(-1) or "")
+        except (OSError, AttributeError):
+            return None
+
+    def selected_text(self, name=None, role=None, app_name=None,
+                      automation_id=None) -> Optional[str]:
+        pattern = self._text_pattern(name, role, app_name, automation_id)
+        if pattern is None:
+            return None
+        try:
+            selection = pattern.GetSelection()
+            if not selection or int(selection.Length or 0) == 0:
+                return ""
+            return str(selection.GetElement(0).GetText(-1) or "")
+        except (OSError, AttributeError):
+            return None
+
+    def visible_text(self, name=None, role=None, app_name=None,
+                     automation_id=None) -> Optional[str]:
+        pattern = self._text_pattern(name, role, app_name, automation_id)
+        if pattern is None:
+            return None
+        try:
+            ranges = pattern.GetVisibleRanges()
+            count = int(ranges.Length or 0)
+            return "".join(str(ranges.GetElement(i).GetText(-1) or "")
+                           for i in range(count))
+        except (OSError, AttributeError):
+            return None
+
     @staticmethod
     def _read_row(pattern, row: int, cols: int):
         """Read one grid row into a list of cell strings."""

je_auto_control/utils/ax_text/__init__.py

@@ -0,0 +1,8 @@
+"""Native text reading via the UI Automation TextPattern (document/selection/visible)."""
+from je_auto_control.utils.ax_text.ax_text import (
+    get_control_text, get_selected_text, get_visible_text,
+)
+
+__all__ = [
+    "get_control_text", "get_selected_text", "get_visible_text",
+]

je_auto_control/utils/ax_text/ax_text.py

@@ -0,0 +1,61 @@
+"""Native text reading via the UI Automation TextPattern.
+
+``control_get_value`` reads a control through ValuePattern, but ValuePattern
+returns an **empty string** on multiline edits, RichEdit / document controls and
+web text areas — exactly the controls whose text you most want to read. UIA
+exposes that text through a different pattern, ``TextPattern``, which models the
+control's content as text ranges. ``ax_text`` adds three reads on top of the
+existing accessibility backend ABC:
+
+* :func:`get_control_text` — the whole document's text (``DocumentRange``),
+* :func:`get_selected_text` — the currently selected text (``GetSelection``),
+* :func:`get_visible_text` — only the on-screen text (``GetVisibleRanges``).
+
+Each function is a thin dispatch onto the injectable
+``accessibility.backends.get_backend()`` seam (the same seam the rest of the
+accessibility module uses), so the headless core is unit-testable on any
+platform by injecting a fake backend; the real UIA calls live in the Windows
+backend. Imports no ``PySide6``.
+"""
+from typing import Optional
+
+
+def _backend():
+    from je_auto_control.utils.accessibility.backends import get_backend
+    return get_backend()
+
+
+def get_control_text(name: Optional[str] = None, role: Optional[str] = None,
+                     app_name: Optional[str] = None,
+                     automation_id: Optional[str] = None) -> Optional[str]:
+    """Return a control's full text via TextPattern (``None`` if not found).
+
+    Unlike :func:`control_get_value`, this works on multiline edits, RichEdit /
+    document controls and web text areas where ValuePattern returns ``""``.
+    """
+    return _backend().document_text(name=name, role=role, app_name=app_name,
+                                    automation_id=automation_id)
+
+
+def get_selected_text(name: Optional[str] = None, role: Optional[str] = None,
+                      app_name: Optional[str] = None,
+                      automation_id: Optional[str] = None) -> Optional[str]:
+    """Return the control's currently selected text (TextPattern.GetSelection).
+
+    Empty string when nothing is selected; ``None`` if the control is not found
+    or exposes no TextPattern.
+    """
+    return _backend().selected_text(name=name, role=role, app_name=app_name,
+                                    automation_id=automation_id)
+
+
+def get_visible_text(name: Optional[str] = None, role: Optional[str] = None,
+                     app_name: Optional[str] = None,
+                     automation_id: Optional[str] = None) -> Optional[str]:
+    """Return only the on-screen text of a control (TextPattern.GetVisibleRanges).
+
+    Useful for scrolled documents where :func:`get_control_text` would return the
+    whole (possibly huge) buffer. ``None`` if the control is not found.
+    """
+    return _backend().visible_text(name=name, role=role, app_name=app_name,
+                                   automation_id=automation_id)

je_auto_control/utils/executor/action_executor.py

@@ -2416,6 +2416,33 @@ def _scroll_control_into_view(name: Optional[str] = None, role: Optional[str] =
                                     automation_id=automation_id)
 
 
+def _get_control_text(name: Optional[str] = None, role: Optional[str] = None,
+                      app_name: Optional[str] = None,
+                      automation_id: Optional[str] = None) -> Dict[str, Any]:
+    """Adapter: read a control's full text via TextPattern (multiline-safe)."""
+    from je_auto_control.utils.ax_text import get_control_text
+    return {"text": get_control_text(name=name, role=role, app_name=app_name,
+                                     automation_id=automation_id)}
+
+
+def _get_selected_text(name: Optional[str] = None, role: Optional[str] = None,
+                       app_name: Optional[str] = None,
+                       automation_id: Optional[str] = None) -> Dict[str, Any]:
+    """Adapter: read a control's currently selected text (TextPattern)."""
+    from je_auto_control.utils.ax_text import get_selected_text
+    return {"text": get_selected_text(name=name, role=role, app_name=app_name,
+                                      automation_id=automation_id)}
+
+
+def _get_visible_text(name: Optional[str] = None, role: Optional[str] = None,
+                      app_name: Optional[str] = None,
+                      automation_id: Optional[str] = None) -> Dict[str, Any]:
+    """Adapter: read only the on-screen text of a control (TextPattern)."""
+    from je_auto_control.utils.ax_text import get_visible_text
+    return {"text": get_visible_text(name=name, role=role, app_name=app_name,
+                                     automation_id=automation_id)}
+
+
 def _read_table(name: Optional[str] = None, role: Optional[str] = None,
                 app_name: Optional[str] = None,
                 automation_id: Optional[str] = None) -> List[List[str]]:
@@ -6092,6 +6119,9 @@ def __init__(self):
             "AC_control_range": _control_range,
             "AC_set_control_range": _set_control_range,
             "AC_scroll_control_into_view": _scroll_control_into_view,
+            "AC_get_control_text": _get_control_text,
+            "AC_get_selected_text": _get_selected_text,
+            "AC_get_visible_text": _get_visible_text,
             "AC_read_table": _read_table,
             "AC_watchdog_add": _watchdog_add,
             "AC_watchdog_start": _watchdog_start,