diff --git a/WHATS_NEW.md b/WHATS_NEW.md index bb61e333..9b6390ec 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -1,5 +1,29 @@ # What's New — AutoControl +## What's new (2026-06-25) — Container Selection + View Switching (Selection / MultipleView) + +Read what's selected in a listbox/grid, and switch Explorer-style views. Full reference: [`docs/source/Eng/doc/new_features/v201_features_doc.rst`](docs/source/Eng/doc/new_features/v201_features_doc.rst). + +- **`get_selection` / `list_views` / `set_view`** (`AC_get_selection`, `AC_list_views`, `AC_set_view`): `select_control_item` selects *one* item, but the container-level `SelectionPattern` answers "what is currently selected, and may it select multiple?" — the assertion target after selecting. `MultipleViewPattern` switches a control between its views (Explorer's list / details / tile / thumbnail), a precondition that otherwise needs fragile menu clicking. `get_selection` returns `{items, can_select_multiple, is_required}`, `list_views` returns `{current, views}`, and `set_view` switches by view name. Dispatched through the injectable accessibility backend seam (headless-testable via a fake backend; real UIA in the Windows backend). No `PySide6`. + +## What's new (2026-06-25) — Advanced TextPattern (find / select / read attributes) + +Search a control's text, select a match to replace it, and read font/colour formatting. Full reference: [`docs/source/Eng/doc/new_features/v200_features_doc.rst`](docs/source/Eng/doc/new_features/v200_features_doc.rst). + +- **`find_control_text` / `select_control_text` / `control_text_attributes`** (`AC_find_control_text`, `AC_select_control_text`, `AC_control_text_attributes`): `ax_text` shipped the three whole-range *reads*, but couldn't search for a substring, select a found range, or read text formatting — needed to assert "the error word is red and bold" or to place the selection at matched text before typing. This rounds out TextPattern: `find_control_text` searches the real content (not OCR) via `FindText`, `select_control_text` finds + selects a range so the next keystrokes replace it, and `control_text_attributes` reads `{font_name, font_size, bold, italic, foreground_color}`. Dispatched through the injectable accessibility backend seam (headless-testable via a fake backend; real UIA in the Windows backend). No `PySide6`. + +## What's new (2026-06-25) — MSAA Bridge for Legacy Controls (LegacyIAccessible) + +Automate the long tail of old Win32 controls that expose nothing via modern UIA. Full reference: [`docs/source/Eng/doc/new_features/v199_features_doc.rst`](docs/source/Eng/doc/new_features/v199_features_doc.rst). + +- **`legacy_info` / `legacy_default_action`** (`AC_legacy_info`, `AC_legacy_default_action`): many legacy Win32 / MFC / Delphi controls expose nothing useful via modern UIA patterns (`control_get_value` / `control_invoke` / `control_toggle` all return None), yet they're fully described through the MSAA `IAccessible` bridge — Name, Value, Description, Role, State and a **DefaultAction**. This reads that info and fires the default action via `LegacyIAccessiblePattern` — the last-resort fallback that makes old apps automatable. Dispatched through the injectable accessibility backend seam (headless-testable via a fake backend; real UIA in the Windows backend). No `PySide6`. + +## What's new (2026-06-25) — Move / Resize Elements + Window State (UIA Transform + Window) + +Move a floating panel, resize a control, and know if a window is modal-blocked. Full reference: [`docs/source/Eng/doc/new_features/v198_features_doc.rst`](docs/source/Eng/doc/new_features/v198_features_doc.rst). + +- **`move_element` / `resize_element` / `set_window_state` / `window_interaction_state`** (`AC_move_element`, `AC_resize_element`, `AC_set_window_state`, `AC_window_interaction_state`): this is UIA-**element-level**, not the HWND/title-level geometry in `window_layout`. `TransformPattern` moves/resizes a specific control or floating panel (dockable toolbars, MDI children, splitters) with no top-level window of its own; `WindowPattern` minimizes/maximizes a window and reports its **interaction state** (`ready` / `blocked_by_modal` / `not_responding`) — a reliable "is this window ready or modal-blocked?" signal pixel/title polling can't give. Dispatched through the injectable accessibility backend seam (headless-testable via a fake backend; real UIA in the Windows backend). No `PySide6`. + ## What's new (2026-06-25) — Table Headers + Cell Addressing (UIA TablePattern) Assert "the Status column of row 5 says Shipped" — by header, not by guessing indices. Full reference: [`docs/source/Eng/doc/new_features/v197_features_doc.rst`](docs/source/Eng/doc/new_features/v197_features_doc.rst). diff --git a/docs/source/Eng/doc/new_features/v198_features_doc.rst b/docs/source/Eng/doc/new_features/v198_features_doc.rst new file mode 100644 index 00000000..9743c2bd --- /dev/null +++ b/docs/source/Eng/doc/new_features/v198_features_doc.rst @@ -0,0 +1,46 @@ +Move / Resize Elements + Window State (UIA Transform + Window) +============================================================== + +This is **UIA-element-level**, not the HWND / title-level geometry in +``window_layout`` / ``window_geometry``. ``TransformPattern`` moves and resizes a +specific control or floating panel (dockable toolbars, MDI children, splitters) +that has no top-level window of its own; ``WindowPattern`` minimizes / maximizes a +window and — most usefully — reports its **interaction state**, a reliable "is +this window ready or modal-blocked?" signal that pixel or title polling can't give. + +* :func:`move_element` / :func:`resize_element` — TransformPattern, +* :func:`set_window_state` — minimize / maximize / restore, +* :func:`window_interaction_state` — the readiness / modal-blocked signal. + +Each is a thin dispatch onto the injectable ``accessibility.backends.get_backend()`` +seam — headless-testable via a fake backend; the real UIA calls live in the +Windows backend. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import (move_element, resize_element, + set_window_state, window_interaction_state) + + move_element(100, 200, name="Tool Palette") # reposition a floating panel + resize_element(640, 480, name="Preview") # resize a control + set_window_state("maximized", name="Editor") # normal / maximized / minimized + + if window_interaction_state(name="Editor") == "ready": + ... # not "blocked_by_modal" / "not_responding" — safe to drive + +The element / window is located by ``name`` / ``role`` / ``app_name`` / +``automation_id`` (same as the other native-control actions). The actions return +``bool``; ``window_interaction_state`` returns ``ready`` / ``blocked_by_modal`` / +``not_responding`` / ``running`` / ``closing`` (or ``None`` if not found). + +Executor commands +----------------- + +``AC_move_element`` (``x`` / ``y``), ``AC_resize_element`` (``width`` / +``height``), ``AC_set_window_state`` (``state``) and +``AC_window_interaction_state`` (``{state}``). They are exposed as the matching +``ac_*`` MCP tools (the actions destructive, the read read-only) and as Script +Builder commands under **Native UI**. diff --git a/docs/source/Eng/doc/new_features/v199_features_doc.rst b/docs/source/Eng/doc/new_features/v199_features_doc.rst new file mode 100644 index 00000000..15f7de18 --- /dev/null +++ b/docs/source/Eng/doc/new_features/v199_features_doc.rst @@ -0,0 +1,43 @@ +MSAA Bridge for Legacy Controls (LegacyIAccessible) +=================================================== + +Many legacy Win32 / MFC / Delphi controls expose **nothing useful** via the modern +UI Automation patterns — ``control_get_value`` / ``control_invoke`` / +``control_toggle`` all return None or do nothing — yet they are fully described +through the MSAA ``IAccessible`` bridge: a Name, Value, Description, Role, State +and a **DefaultAction**. ``legacy_accessible`` is the last-resort fallback that +still reads that info and fires the default action, making the long tail of old +apps automatable. + +* :func:`legacy_info` — the MSAA fields ``{name, value, description, + default_action, role, state}``, +* :func:`legacy_default_action` — fire the control's default action. + +Each is a thin dispatch onto the injectable ``accessibility.backends.get_backend()`` +seam — headless-testable via a fake backend; the real ``LegacyIAccessiblePattern`` +calls live in the Windows backend. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import (legacy_info, legacy_default_action, + control_invoke) + + # Modern patterns came up empty? Fall back to MSAA: + if not control_invoke(name="Apply"): + info = legacy_info(name="Apply") # {"default_action": "Press", ...} + legacy_default_action(name="Apply") # fires the MSAA default action + +The control is located by ``name`` / ``role`` / ``app_name`` / ``automation_id`` +(same as the other native-control actions). ``legacy_info`` returns the MSAA info +dict (``role`` / ``state`` are the raw MSAA numbers) or ``None`` if the control or +pattern isn't found; ``legacy_default_action`` returns ``bool``. + +Executor commands +----------------- + +``AC_legacy_info`` (``{found, info}``) and ``AC_legacy_default_action``. They are +exposed as the matching ``ac_*`` MCP tools (info read-only, the action +destructive) and as Script Builder commands under **Native UI**. diff --git a/docs/source/Eng/doc/new_features/v200_features_doc.rst b/docs/source/Eng/doc/new_features/v200_features_doc.rst new file mode 100644 index 00000000..0054a6cd --- /dev/null +++ b/docs/source/Eng/doc/new_features/v200_features_doc.rst @@ -0,0 +1,48 @@ +Advanced TextPattern — Find / Select / Read Attributes +====================================================== + +``ax_text`` shipped the three whole-range *reads* (document / selection / visible +text). It could not **search** for a substring, **select** a found range, or read +text **formatting attributes** — needed to assert "the error word is red and +bold" or to place the caret / selection at matched text before typing. This rounds +TextPattern out from "dump the text" to "interrogate and manipulate" it. + +* :func:`find_control_text` — whether ``text`` occurs in the control + (TextPattern.FindText, searches the real content, not OCR), +* :func:`select_control_text` — find ``text`` and select its range, so the next + keystrokes replace it (FindText + Select), +* :func:`control_text_attributes` — the selection's font / colour formatting. + +Each is a thin dispatch onto the injectable ``accessibility.backends.get_backend()`` +seam — headless-testable via a fake backend; the real UIA calls live in the +Windows backend. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import (find_control_text, select_control_text, + control_text_attributes, type_text) + + if find_control_text("TODO", name="Editor"): + select_control_text("TODO", name="Editor") # selection now spans "TODO" + type_text("DONE") # replaces it + + control_text_attributes(name="Editor") + # {"font_name": "Consolas", "font_size": 11.0, "bold": True, + # "italic": False, "foreground_color": 16711680} + +``ignore_case`` (default ``True``) controls the search. The control is located by +``name`` / ``role`` / ``app_name`` / ``automation_id`` (same as the other +TextPattern reads). ``find_control_text`` / ``select_control_text`` return +``bool``; ``control_text_attributes`` returns the formatting dict (values may be +``None`` where the range spans mixed formatting) or ``None`` if not found. + +Executor commands +----------------- + +``AC_find_control_text`` / ``AC_select_control_text`` (``text`` / ``ignore_case``) +and ``AC_control_text_attributes`` (``{found, attributes}``). They are exposed as +the matching ``ac_*`` MCP tools (find / attributes read-only, select destructive) +and as Script Builder commands under **Native UI**. diff --git a/docs/source/Eng/doc/new_features/v201_features_doc.rst b/docs/source/Eng/doc/new_features/v201_features_doc.rst new file mode 100644 index 00000000..c4817940 --- /dev/null +++ b/docs/source/Eng/doc/new_features/v201_features_doc.rst @@ -0,0 +1,45 @@ +Container Selection + View Switching (Selection / MultipleView) +=============================================================== + +``select_control_item`` (SelectionItemPattern) selects *one* item; the +container-level ``SelectionPattern`` answers the natural follow-up — **what is +currently selected** in a listbox / grid / tab, and **may it select multiple?** — +the assertion target after selecting. ``MultipleViewPattern`` switches a control +between its views (Explorer's list / details / tile / thumbnail), a common +precondition that otherwise needs fragile menu clicking. + +* :func:`get_selection` — ``{items, can_select_multiple, is_required}``, +* :func:`list_views` — ``{current, views: [...]}``, +* :func:`set_view` — switch to a named view. + +Each is a thin dispatch onto the injectable ``accessibility.backends.get_backend()`` +seam — headless-testable via a fake backend; the real UIA calls live in the +Windows backend. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import get_selection, list_views, set_view + + get_selection(name="File List") + # {"items": ["report.pdf", "notes.txt"], "can_select_multiple": True, + # "is_required": False} + + list_views(name="File List") + # {"current": "Details", "views": ["List", "Details", "Tiles"]} + set_view("Tiles", name="File List") # switch the view + +The control is located by ``name`` / ``role`` / ``app_name`` / ``automation_id`` +(same as the other native-control actions). ``get_selection`` / ``list_views`` +return their dict (or ``None`` if the control or pattern isn't found); +``set_view`` returns ``bool`` (False when the named view isn't supported). + +Executor commands +----------------- + +``AC_get_selection`` (``{found, selection}``), ``AC_list_views`` (``{found, +views}``) and ``AC_set_view`` (``view``). They are exposed as the matching +``ac_*`` MCP tools (the reads read-only, ``set_view`` destructive) and as Script +Builder commands under **Native UI**. diff --git a/docs/source/Zh/doc/new_features/v198_features_doc.rst b/docs/source/Zh/doc/new_features/v198_features_doc.rst new file mode 100644 index 00000000..265715de --- /dev/null +++ b/docs/source/Zh/doc/new_features/v198_features_doc.rst @@ -0,0 +1,41 @@ +移動 / 縮放元素 + 視窗狀態(UIA Transform + Window) +================================================== + +這是 **UIA 元素層級**,而非 ``window_layout`` / ``window_geometry`` 的 HWND / title 層級幾何。 +``TransformPattern`` 移動與縮放某個沒有自己頂層視窗的特定控制項或浮動面板(可停靠工具列、MDI +子視窗、分隔器);``WindowPattern`` 最小化 / 最大化視窗,並——最有用地——回報其 **interaction +state(互動狀態)**,這是像素或標題輪詢給不出的可靠「這視窗是否就緒 / 被 modal 擋住?」訊號。 + +* :func:`move_element` / :func:`resize_element` ——TransformPattern, +* :func:`set_window_state` ——最小化 / 最大化 / 還原, +* :func:`window_interaction_state` ——就緒 / 被 modal 擋住的訊號。 + +每個都是對可注入的 ``accessibility.backends.get_backend()`` 接縫的薄分派——可透過注入 fake +backend 進行無頭測試;真正的 UIA 呼叫位於 Windows 後端。不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import (move_element, resize_element, + set_window_state, window_interaction_state) + + move_element(100, 200, name="Tool Palette") # 重新定位浮動面板 + resize_element(640, 480, name="Preview") # 縮放控制項 + set_window_state("maximized", name="Editor") # normal / maximized / minimized + + if window_interaction_state(name="Editor") == "ready": + ... # 不是 "blocked_by_modal" / "not_responding"——可安全驅動 + +元素 / 視窗以 ``name`` / ``role`` / ``app_name`` / ``automation_id`` 定位(與其他原生控制動作 +相同)。動作回傳 ``bool``;``window_interaction_state`` 回傳 ``ready`` / ``blocked_by_modal`` / +``not_responding`` / ``running`` / ``closing``(找不到則為 ``None``)。 + +執行器指令 +---------- + +``AC_move_element``(``x`` / ``y``)、``AC_resize_element``(``width`` / ``height``)、 +``AC_set_window_state``(``state``)與 ``AC_window_interaction_state``(``{state}``)。皆以對應的 +``ac_*`` MCP 工具(動作類為破壞性、讀取類為唯讀)及 Script Builder 指令(位於 **Native UI** +分類下)形式提供。 diff --git a/docs/source/Zh/doc/new_features/v199_features_doc.rst b/docs/source/Zh/doc/new_features/v199_features_doc.rst new file mode 100644 index 00000000..ea5d746d --- /dev/null +++ b/docs/source/Zh/doc/new_features/v199_features_doc.rst @@ -0,0 +1,38 @@ +舊式控制項的 MSAA 橋接(LegacyIAccessible) +========================================== + +許多舊式 Win32 / MFC / Delphi 控制項透過現代 UI Automation 模式**完全不提供有用資訊**—— +``control_get_value`` / ``control_invoke`` / ``control_toggle`` 都回 None 或毫無作用——但它們透過 +MSAA ``IAccessible`` 橋接卻有完整描述:Name、Value、Description、Role、State,以及一個 +**DefaultAction**。``legacy_accessible`` 就是那個最後手段的後備:仍能讀取這些資訊並觸發預設動作, +讓大量舊應用程式得以自動化。 + +* :func:`legacy_info` ——MSAA 欄位 ``{name, value, description, default_action, role, state}``, +* :func:`legacy_default_action` ——觸發控制項的預設動作。 + +每個都是對可注入的 ``accessibility.backends.get_backend()`` 接縫的薄分派——可透過注入 fake +backend 進行無頭測試;真正的 ``LegacyIAccessiblePattern`` 呼叫位於 Windows 後端。不匯入 +``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import (legacy_info, legacy_default_action, + control_invoke) + + # 現代模式撲空?退回 MSAA: + if not control_invoke(name="Apply"): + info = legacy_info(name="Apply") # {"default_action": "Press", ...} + legacy_default_action(name="Apply") # 觸發 MSAA 預設動作 + +控制項以 ``name`` / ``role`` / ``app_name`` / ``automation_id`` 定位(與其他原生控制動作相同)。 +``legacy_info`` 回傳 MSAA 資訊字典(``role`` / ``state`` 為原始 MSAA 數字),找不到控制項或模式 +則回傳 ``None``;``legacy_default_action`` 回傳 ``bool``。 + +執行器指令 +---------- + +``AC_legacy_info``(``{found, info}``)與 ``AC_legacy_default_action``。皆以對應的 ``ac_*`` MCP +工具(info 為唯讀、動作為破壞性)及 Script Builder 指令(位於 **Native UI** 分類下)形式提供。 diff --git a/docs/source/Zh/doc/new_features/v200_features_doc.rst b/docs/source/Zh/doc/new_features/v200_features_doc.rst new file mode 100644 index 00000000..fea1107b --- /dev/null +++ b/docs/source/Zh/doc/new_features/v200_features_doc.rst @@ -0,0 +1,44 @@ +進階 TextPattern——搜尋 / 選取 / 讀取屬性 +======================================== + +``ax_text`` 先前提供三種整段*讀取*(document / selection / visible 文字)。它無法**搜尋**子字串、 +**選取**找到的範圍,或讀取文字的**格式屬性**——而這些正是斷言「錯誤字是紅色且粗體」或在輸入前 +把游標 / 選取定位到匹配文字所需。本次把 TextPattern 從「傾印文字」擴充為「查詢與操作」文字。 + +* :func:`find_control_text` ——``text`` 是否出現在控制項中(TextPattern.FindText,搜尋真正的 + 內容,而非 OCR), +* :func:`select_control_text` ——找到 ``text`` 並選取其範圍,讓接下來的按鍵取代它 + (FindText + Select), +* :func:`control_text_attributes` ——選取範圍的字型 / 顏色格式。 + +每個都是對可注入的 ``accessibility.backends.get_backend()`` 接縫的薄分派——可透過注入 fake +backend 進行無頭測試;真正的 UIA 呼叫位於 Windows 後端。不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import (find_control_text, select_control_text, + control_text_attributes, type_text) + + if find_control_text("TODO", name="Editor"): + select_control_text("TODO", name="Editor") # 選取範圍現在涵蓋 "TODO" + type_text("DONE") # 取代它 + + control_text_attributes(name="Editor") + # {"font_name": "Consolas", "font_size": 11.0, "bold": True, + # "italic": False, "foreground_color": 16711680} + +``ignore_case``(預設 ``True``)控制搜尋。控制項以 ``name`` / ``role`` / ``app_name`` / +``automation_id`` 定位(與其他 TextPattern 讀取相同)。``find_control_text`` / +``select_control_text`` 回傳 ``bool``;``control_text_attributes`` 回傳格式字典(範圍跨越混合 +格式時某些值可能為 ``None``),找不到則回傳 ``None``。 + +執行器指令 +---------- + +``AC_find_control_text`` / ``AC_select_control_text``(``text`` / ``ignore_case``)與 +``AC_control_text_attributes``(``{found, attributes}``)。皆以對應的 ``ac_*`` MCP 工具 +(find / attributes 為唯讀、select 為破壞性)及 Script Builder 指令(位於 **Native UI** 分類下) +形式提供。 diff --git a/docs/source/Zh/doc/new_features/v201_features_doc.rst b/docs/source/Zh/doc/new_features/v201_features_doc.rst new file mode 100644 index 00000000..193be128 --- /dev/null +++ b/docs/source/Zh/doc/new_features/v201_features_doc.rst @@ -0,0 +1,40 @@ +容器選取 + 檢視切換(Selection / MultipleView) +============================================== + +``select_control_item``(SelectionItemPattern)選取*單一*項目;容器層級的 ``SelectionPattern`` +回答自然的後續問題——listbox / grid / tab 中**目前選了什麼**,以及**是否可多選?**——這正是選取 +之後的斷言目標。``MultipleViewPattern`` 在控制項的各檢視之間切換(檔案總管的清單 / 詳細資料 / +並排 / 縮圖),這是個常見前置條件,否則就得靠脆弱的選單點擊。 + +* :func:`get_selection` ——``{items, can_select_multiple, is_required}``, +* :func:`list_views` ——``{current, views: [...]}``, +* :func:`set_view` ——切換到具名的檢視。 + +每個都是對可注入的 ``accessibility.backends.get_backend()`` 接縫的薄分派——可透過注入 fake +backend 進行無頭測試;真正的 UIA 呼叫位於 Windows 後端。不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import get_selection, list_views, set_view + + get_selection(name="File List") + # {"items": ["report.pdf", "notes.txt"], "can_select_multiple": True, + # "is_required": False} + + list_views(name="File List") + # {"current": "Details", "views": ["List", "Details", "Tiles"]} + set_view("Tiles", name="File List") # 切換檢視 + +控制項以 ``name`` / ``role`` / ``app_name`` / ``automation_id`` 定位(與其他原生控制動作相同)。 +``get_selection`` / ``list_views`` 回傳其字典(找不到控制項或模式則為 ``None``);``set_view`` +回傳 ``bool``(具名檢視不支援時為 False)。 + +執行器指令 +---------- + +``AC_get_selection``(``{found, selection}``)、``AC_list_views``(``{found, views}``)與 +``AC_set_view``(``view``)。皆以對應的 ``ac_*`` MCP 工具(讀取類為唯讀、``set_view`` 為破壞性) +及 Script Builder 指令(位於 **Native UI** 分類下)形式提供。 diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index 6ae62724..f514a937 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -53,9 +53,10 @@ collapse_control, control_expand_state, control_range, expand_control, scroll_control_into_view, select_control_item, set_control_range, ) -# Native text reads via UIA TextPattern (document / selection / visible) +# Native text via UIA TextPattern (read / find / select / attributes) from je_auto_control.utils.ax_text import ( - get_control_text, get_selected_text, get_visible_text, + control_text_attributes, find_control_text, get_control_text, + get_selected_text, get_visible_text, select_control_text, ) # Readable / addressable a11y-tree post-processing (role names + node paths) from je_auto_control.utils.ax_tree_walk import ( @@ -76,6 +77,18 @@ from je_auto_control.utils.table_pattern import ( cell_by_header, table_cell, table_headers, ) +# Move/resize UIA elements + window state (Transform + Window patterns) +from je_auto_control.utils.transform_window import ( + move_element, resize_element, set_window_state, window_interaction_state, +) +# MSAA bridge for old controls UIA can't model (LegacyIAccessiblePattern) +from je_auto_control.utils.legacy_accessible import ( + legacy_default_action, legacy_info, +) +# Container selection state + view switching (Selection / MultipleView patterns) +from je_auto_control.utils.selection_view import ( + get_selection, list_views, set_view, +) # Rich clipboard formats — RTF + CSV/TSV codecs and Windows get / set from je_auto_control.utils.clipboard_rich_formats import ( build_rtf, csv_to_rows, get_clipboard_csv, get_clipboard_rtf, rows_to_csv, @@ -1673,12 +1686,17 @@ def start_autocontrol_gui(*args, **kwargs): "select_control_item", "control_range", "set_control_range", "scroll_control_into_view", "get_control_text", "get_selected_text", "get_visible_text", + "find_control_text", "select_control_text", "control_text_attributes", "control_type_name", "humanize_role", "humanize_tree", "assign_node_paths", "find_by_path", "is_interactive_role", "tab_order", "audit_focus_order", "focus_control", "realize_item", "get_element_properties", "is_element_enabled", "table_headers", "table_cell", "cell_by_header", + "move_element", "resize_element", "set_window_state", + "window_interaction_state", + "legacy_info", "legacy_default_action", + "get_selection", "list_views", "set_view", "build_rtf", "rtf_to_text", "rows_to_csv", "csv_to_rows", "set_clipboard_rtf", "get_clipboard_rtf", "set_clipboard_csv", "get_clipboard_csv", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index ebbaf161..c7aabaed 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -1643,11 +1643,78 @@ def _add_native_control_specs(specs: List[CommandSpec]) -> None: FieldSpec("column_header", FieldType.STRING)) + fields, description="Read the cell at (row, named column) — assert by header.", )) + specs.append(CommandSpec( + "AC_move_element", "Native UI", "Move Element (Transform)", + fields=(FieldSpec("x", FieldType.FLOAT), + FieldSpec("y", FieldType.FLOAT)) + fields, + description="Move a UIA element to (x, y) (TransformPattern).", + )) + specs.append(CommandSpec( + "AC_resize_element", "Native UI", "Resize Element (Transform)", + fields=(FieldSpec("width", FieldType.FLOAT), + FieldSpec("height", FieldType.FLOAT)) + fields, + description="Resize a UIA element (TransformPattern).", + )) + specs.append(CommandSpec( + "AC_set_window_state", "Native UI", "Set Window State", + fields=(FieldSpec("state", FieldType.ENUM, default="normal", + choices=("normal", "maximized", "minimized")),) + fields, + description="Minimize / maximize / restore a window (WindowPattern).", + )) + specs.append(CommandSpec( + "AC_window_interaction_state", "Native UI", "Window Interaction State", + fields=fields, + description="Read window readiness (ready/blocked_by_modal/...).", + )) + specs.append(CommandSpec( + "AC_legacy_info", "Native UI", "Legacy (MSAA) Info", + fields=fields, + description="Read an old control's MSAA info (LegacyIAccessible).", + )) + specs.append(CommandSpec( + "AC_legacy_default_action", "Native UI", "Legacy (MSAA) Default Action", + fields=fields, + description="Fire an old control's MSAA default action (fallback).", + )) + specs.append(CommandSpec( + "AC_get_selection", "Native UI", "Get Container Selection", + fields=fields, + description="Read a container's selection (SelectionPattern).", + )) + specs.append(CommandSpec( + "AC_list_views", "Native UI", "List Control Views", + fields=fields, + description="List a control's selectable views (MultipleViewPattern).", + )) + specs.append(CommandSpec( + "AC_set_view", "Native UI", "Set Control View", + fields=(FieldSpec("view", FieldType.STRING),) + fields, + description="Switch a control to the named view (MultipleViewPattern).", + )) specs.append(CommandSpec( "AC_get_control_text", "Native UI", "Get Control Text", fields=fields, description="Read full text via TextPattern (multiline / document safe).", )) + specs.append(CommandSpec( + "AC_find_control_text", "Native UI", "Find Text in Control", + fields=(FieldSpec("text", FieldType.STRING), + FieldSpec("ignore_case", FieldType.BOOL, optional=True, + default=True)) + fields, + description="Whether text occurs in a control (TextPattern.FindText).", + )) + specs.append(CommandSpec( + "AC_select_control_text", "Native UI", "Select Text in Control", + fields=(FieldSpec("text", FieldType.STRING), + FieldSpec("ignore_case", FieldType.BOOL, optional=True, + default=True)) + fields, + description="Find + select text in a control (FindText + Select).", + )) + specs.append(CommandSpec( + "AC_control_text_attributes", "Native UI", "Get Text Attributes", + fields=fields, + description="Read selection formatting (font/size/bold/italic/colour).", + )) specs.append(CommandSpec( "AC_get_selected_text", "Native UI", "Get Selected Text", fields=fields, diff --git a/je_auto_control/utils/accessibility/backends/base.py b/je_auto_control/utils/accessibility/backends/base.py index 2624d5d3..de085ab9 100644 --- a/je_auto_control/utils/accessibility/backends/base.py +++ b/je_auto_control/utils/accessibility/backends/base.py @@ -123,6 +123,28 @@ def visible_text(self, name: Optional[str] = None, role: Optional[str] = None, """Return only the on-screen text of the control (TextPattern), or None.""" self._unsupported("visible_text") + def find_text(self, text: str = "", ignore_case: bool = True, + name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Return whether ``text`` occurs in the control (TextPattern.FindText).""" + self._unsupported("find_text") + + def select_text(self, text: str = "", ignore_case: bool = True, + name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Find ``text`` and select its range (TextPattern.FindText + Select).""" + self._unsupported("select_text") + + def text_attributes(self, name: Optional[str] = None, + role: Optional[str] = None, app_name: Optional[str] = None, + automation_id: Optional[str] = None, + ) -> Optional[Dict[str, Any]]: + """Return formatting of the control's selection — ``{font_name, font_size, + bold, italic, foreground_color}`` (TextPattern attributes), or None.""" + self._unsupported("text_attributes") + # --- keyboard focus ---------------------------------------------------- def set_focus(self, name: Optional[str] = None, role: Optional[str] = None, @@ -181,6 +203,84 @@ def get_grid_cell(self, row: int = 0, column: int = 0, row_span, column_span}`` (GridPattern.GetItem + GridItemPattern).""" self._unsupported("get_grid_cell") + # --- transform + window patterns (UIA-element-level) -------------------- + + def move_element(self, x: float = 0.0, y: float = 0.0, + name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Move the matched element to ``(x, y)`` (TransformPattern); True on success.""" + self._unsupported("move_element") + + def resize_element(self, width: float = 0.0, height: float = 0.0, + name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Resize the matched element (TransformPattern); True on success.""" + self._unsupported("resize_element") + + def set_window_state(self, state: str = "normal", + name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Set a window's visual state ``normal`` / ``maximized`` / ``minimized``.""" + self._unsupported("set_window_state") + + def window_interaction_state(self, name: Optional[str] = None, + role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None, + ) -> Optional[str]: + """Return a window's interaction state — ``ready`` / ``blocked_by_modal`` / + ``not_responding`` / ``running`` / ``closing`` (WindowPattern), or None.""" + self._unsupported("window_interaction_state") + + # --- MSAA bridge (LegacyIAccessiblePattern) ---------------------------- + + def legacy_info(self, name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None, + ) -> Optional[Dict[str, Any]]: + """Return the MSAA ``IAccessible`` info of an old control, or None. + + ``{name, value, description, default_action, role, state}`` — the + last-resort read for legacy Win32 controls that expose nothing useful via + the modern UIA patterns. + """ + self._unsupported("legacy_info") + + def legacy_default_action(self, name: Optional[str] = None, + role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Fire an old control's MSAA default action (DoDefaultAction); True on + success — the fallback when Value / Invoke / Toggle all do nothing.""" + self._unsupported("legacy_default_action") + + # --- container selection + views (Selection / MultipleView patterns) ---- + + def get_selection(self, name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None, + ) -> Optional[Dict[str, Any]]: + """Return a container's selection state — ``{items, can_select_multiple, + is_required}`` (SelectionPattern), or None.""" + self._unsupported("get_selection") + + def list_views(self, name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None, + ) -> Optional[Dict[str, Any]]: + """Return a control's selectable views — ``{current, views: [...]}`` + (MultipleViewPattern: list / details / tile / …), or None.""" + self._unsupported("list_views") + + def set_view(self, view: str = "", name: Optional[str] = None, + role: Optional[str] = None, app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Switch a control to the named view (MultipleViewPattern); True on success.""" + self._unsupported("set_view") + def _unsupported(self, operation: str): """Raise a clear error for an action this backend can't perform.""" raise AccessibilityNotAvailableError( diff --git a/je_auto_control/utils/accessibility/backends/windows_backend.py b/je_auto_control/utils/accessibility/backends/windows_backend.py index e46c69e9..8775bc5d 100644 --- a/je_auto_control/utils/accessibility/backends/windows_backend.py +++ b/je_auto_control/utils/accessibility/backends/windows_backend.py @@ -34,8 +34,18 @@ _UIA_VIRTUALIZEDITEM_PATTERN_ID = 10020 _UIA_TABLE_PATTERN_ID = 10012 _UIA_GRIDITEM_PATTERN_ID = 10007 +_UIA_TRANSFORM_PATTERN_ID = 10016 +_UIA_WINDOW_PATTERN_ID = 10009 +_UIA_LEGACYIACCESSIBLE_PATTERN_ID = 10018 +_UIA_SELECTION_PATTERN_ID = 10001 +_UIA_MULTIPLEVIEW_PATTERN_ID = 10008 _UIA_AUTOMATIONID_PROPERTY = 30011 _EXPAND_STATES = {0: "collapsed", 1: "expanded", 2: "partial", 3: "leaf"} +_WINDOW_VISUAL_STATES = {"normal": 0, "maximized": 1, "minimized": 2} +_WINDOW_INTERACTION_STATES = { + 0: "running", 1: "closing", 2: "ready", 3: "blocked_by_modal", + 4: "not_responding", +} def _is_available() -> bool: @@ -307,6 +317,109 @@ def get_properties(self, name=None, role=None, app_name=None, return None return _read_properties(raw) + def move_element(self, x=0.0, y=0.0, name=None, role=None, app_name=None, + automation_id=None): + return self._invoke_pattern_method( + name, role, app_name, automation_id, _UIA_TRANSFORM_PATTERN_ID, + "IUIAutomationTransformPattern", + lambda pattern: pattern.Move(float(x), float(y))) + + def resize_element(self, width=0.0, height=0.0, name=None, role=None, + app_name=None, automation_id=None): + return self._invoke_pattern_method( + name, role, app_name, automation_id, _UIA_TRANSFORM_PATTERN_ID, + "IUIAutomationTransformPattern", + lambda pattern: pattern.Resize(float(width), float(height))) + + def set_window_state(self, state="normal", name=None, role=None, + app_name=None, automation_id=None): + visual = _WINDOW_VISUAL_STATES.get(str(state).lower()) + if visual is None: + return False + return self._invoke_pattern_method( + name, role, app_name, automation_id, _UIA_WINDOW_PATTERN_ID, + "IUIAutomationWindowPattern", + lambda pattern: pattern.SetWindowVisualState(visual)) + + def window_interaction_state(self, name=None, role=None, app_name=None, + automation_id=None) -> Optional[str]: + raw = self._find_raw(name, role, app_name, automation_id) + pattern = self._pattern(raw, _UIA_WINDOW_PATTERN_ID, + "IUIAutomationWindowPattern") if raw else None + if pattern is None: + return None + try: + return _WINDOW_INTERACTION_STATES.get( + int(pattern.CurrentWindowInteractionState)) + except (OSError, AttributeError, ValueError, TypeError): + return None + + def legacy_info(self, name=None, role=None, app_name=None, + automation_id=None) -> Optional[Dict[str, Any]]: + raw = self._find_raw(name, role, app_name, automation_id) + pattern = self._pattern(raw, _UIA_LEGACYIACCESSIBLE_PATTERN_ID, + "IUIAutomationLegacyIAccessiblePattern" + ) if raw else None + if pattern is None: + return None + return _read_legacy(pattern) + + def legacy_default_action(self, name=None, role=None, app_name=None, + automation_id=None): + return self._invoke_pattern_method( + name, role, app_name, automation_id, + _UIA_LEGACYIACCESSIBLE_PATTERN_ID, + "IUIAutomationLegacyIAccessiblePattern", + lambda pattern: pattern.DoDefaultAction()) + + def get_selection(self, name=None, role=None, app_name=None, + automation_id=None) -> Optional[Dict[str, Any]]: + raw = self._find_raw(name, role, app_name, automation_id) + pattern = self._pattern(raw, _UIA_SELECTION_PATTERN_ID, + "IUIAutomationSelectionPattern") if raw else None + if pattern is None: + return None + try: + items = _header_names(pattern.GetCurrentSelection()) + can_multiple = bool(pattern.CurrentCanSelectMultiple) + required = bool(pattern.CurrentIsSelectionRequired) + except (OSError, AttributeError): + return None + return {"items": items, "can_select_multiple": can_multiple, + "is_required": required} + + def _multiple_view(self, name, role, app_name, automation_id): + raw = self._find_raw(name, role, app_name, automation_id) + return self._pattern(raw, _UIA_MULTIPLEVIEW_PATTERN_ID, + "IUIAutomationMultipleViewPattern") if raw else None + + def list_views(self, name=None, role=None, app_name=None, + automation_id=None) -> Optional[Dict[str, Any]]: + pattern = self._multiple_view(name, role, app_name, automation_id) + if pattern is None: + return None + try: + view_ids = list(pattern.GetCurrentSupportedViews()) + current = int(pattern.CurrentCurrentView) + except (OSError, AttributeError, ValueError, TypeError): + return None + return {"current": _view_name(pattern, current), + "views": [_view_name(pattern, view_id) for view_id in view_ids]} + + def set_view(self, view="", name=None, role=None, app_name=None, + automation_id=None): + pattern = self._multiple_view(name, role, app_name, automation_id) + if pattern is None: + return False + try: + for view_id in pattern.GetCurrentSupportedViews(): + if _view_name(pattern, view_id) == str(view): + pattern.SetCurrentView(int(view_id)) + return True + except (OSError, AttributeError, ValueError, TypeError): + return False + return False + def get_table_headers(self, name=None, role=None, app_name=None, automation_id=None) -> Optional[Dict[str, Any]]: raw = self._find_raw(name, role, app_name, automation_id) @@ -382,6 +495,48 @@ def visible_text(self, name=None, role=None, app_name=None, except (OSError, AttributeError): return None + def _find_range(self, text, ignore_case, name, role, app_name, automation_id): + """Find ``text`` in the control's document range (TextPattern.FindText).""" + pattern = self._text_pattern(name, role, app_name, automation_id) + if pattern is None: + return None + try: + return pattern.DocumentRange.FindText(str(text), False, + bool(ignore_case)) + except (OSError, AttributeError): + return None + + def find_text(self, text="", ignore_case=True, name=None, role=None, + app_name=None, automation_id=None) -> bool: + return self._find_range(text, ignore_case, name, role, app_name, + automation_id) is not None + + def select_text(self, text="", ignore_case=True, name=None, role=None, + app_name=None, automation_id=None) -> bool: + found = self._find_range(text, ignore_case, name, role, app_name, + automation_id) + if not found: + return False + try: + found.Select() + return True + except (OSError, AttributeError): + return False + + def text_attributes(self, name=None, role=None, app_name=None, + automation_id=None) -> Optional[Dict[str, Any]]: + pattern = self._text_pattern(name, role, app_name, automation_id) + if pattern is None: + return None + try: + selection = pattern.GetSelection() + text_range = (selection.GetElement(0) + if selection and int(selection.Length or 0) > 0 + else pattern.DocumentRange) + except (OSError, AttributeError): + return None + return _read_text_attributes(text_range) + def set_focus(self, name=None, role=None, app_name=None, automation_id=None) -> bool: raw = self._find_raw(name, role, app_name, automation_id) @@ -406,6 +561,14 @@ def _read_row(pattern, row: int, cols: int): return cells +def _view_name(pattern, view_id) -> str: + """Return a MultipleViewPattern view's name, or '' on failure.""" + try: + return str(pattern.GetViewName(int(view_id)) or "") + except (OSError, AttributeError, ValueError, TypeError): + return "" + + def _header_names(array) -> List[str]: """Read an IUIAutomationElementArray of header elements into name strings.""" names: List[str] = [] @@ -449,6 +612,55 @@ def _as_text(value) -> str: return str(value or "") +# UIA TextPattern attribute ids (UIAutomationClient AttributeId range). +_TEXT_ATTR_FONT_NAME = 40005 +_TEXT_ATTR_FONT_SIZE = 40006 +_TEXT_ATTR_FONT_WEIGHT = 40007 +_TEXT_ATTR_FOREGROUND = 40008 +_TEXT_ATTR_IS_ITALIC = 40014 + + +def _attr(text_range, attribute_id, cast): + try: + return cast(text_range.GetAttributeValue(attribute_id)) + except (OSError, AttributeError, ValueError, TypeError): + return None + + +def _read_text_attributes(text_range) -> Dict[str, Any]: + """Read font / colour formatting of a TextRange into a plain dict.""" + weight = _attr(text_range, _TEXT_ATTR_FONT_WEIGHT, int) + return { + "font_name": _attr(text_range, _TEXT_ATTR_FONT_NAME, _as_text), + "font_size": _attr(text_range, _TEXT_ATTR_FONT_SIZE, float), + "bold": (weight >= 700) if isinstance(weight, int) else None, + "italic": _attr(text_range, _TEXT_ATTR_IS_ITALIC, bool), + "foreground_color": _attr(text_range, _TEXT_ATTR_FOREGROUND, int), + } + + +# (key, LegacyIAccessiblePattern attribute, cast) for the MSAA bridge read. +_LEGACY_READS = ( + ("name", "CurrentName", _as_text), + ("value", "CurrentValue", _as_text), + ("description", "CurrentDescription", _as_text), + ("default_action", "CurrentDefaultAction", _as_text), + ("role", "CurrentRole", int), + ("state", "CurrentState", int), +) + + +def _read_legacy(pattern) -> Dict[str, Any]: + """Read a LegacyIAccessiblePattern's MSAA fields into a plain dict.""" + info: Dict[str, Any] = {} + for key, attribute, cast in _LEGACY_READS: + try: + info[key] = cast(getattr(pattern, attribute)) + except (OSError, AttributeError, ValueError, TypeError): + info[key] = None + return info + + # (key, UIA element attribute, cast) for the rich properties the flat list omits. _PROPERTY_READS = ( ("enabled", "CurrentIsEnabled", bool), diff --git a/je_auto_control/utils/ax_text/__init__.py b/je_auto_control/utils/ax_text/__init__.py index 66ed749b..d18d48c3 100644 --- a/je_auto_control/utils/ax_text/__init__.py +++ b/je_auto_control/utils/ax_text/__init__.py @@ -1,8 +1,10 @@ -"""Native text reading via the UI Automation TextPattern (document/selection/visible).""" +"""Native text via the UI Automation TextPattern (read / find / select / attributes).""" from je_auto_control.utils.ax_text.ax_text import ( - get_control_text, get_selected_text, get_visible_text, + control_text_attributes, find_control_text, get_control_text, + get_selected_text, get_visible_text, select_control_text, ) __all__ = [ "get_control_text", "get_selected_text", "get_visible_text", + "find_control_text", "select_control_text", "control_text_attributes", ] diff --git a/je_auto_control/utils/ax_text/ax_text.py b/je_auto_control/utils/ax_text/ax_text.py index 7f04f1dc..d61ccc2b 100644 --- a/je_auto_control/utils/ax_text/ax_text.py +++ b/je_auto_control/utils/ax_text/ax_text.py @@ -17,7 +17,7 @@ platform by injecting a fake backend; the real UIA calls live in the Windows backend. Imports no ``PySide6``. """ -from typing import Optional +from typing import Any, Dict, Optional def _backend(): @@ -59,3 +59,34 @@ def get_visible_text(name: Optional[str] = None, role: Optional[str] = None, """ return _backend().visible_text(name=name, role=role, app_name=app_name, automation_id=automation_id) + + +def find_control_text(text: str, *, ignore_case: bool = True, + name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Return whether ``text`` occurs in the control (TextPattern.FindText).""" + return _backend().find_text(str(text), bool(ignore_case), name=name, + role=role, app_name=app_name, + automation_id=automation_id) + + +def select_control_text(text: str, *, ignore_case: bool = True, + name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Find ``text`` and select its range — position the caret / selection before + typing to replace it (TextPattern.FindText + Select); True on success.""" + return _backend().select_text(str(text), bool(ignore_case), name=name, + role=role, app_name=app_name, + automation_id=automation_id) + + +def control_text_attributes(name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None, + ) -> Optional[Dict[str, Any]]: + """Return the control selection's formatting — ``{font_name, font_size, bold, + italic, foreground_color}`` (TextPattern attributes), or None if not found.""" + return _backend().text_attributes(name=name, role=role, app_name=app_name, + automation_id=automation_id) diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index 14e5265b..9f55c7eb 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -2516,6 +2516,92 @@ def _cell_by_header(row: Any, column_header: str, name: Optional[str] = None, return {"found": value is not None, "value": value} +def _move_element(x: Any, y: Any, name: Optional[str] = None, + role: Optional[str] = None, app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Adapter: move a UIA element to (x, y) (TransformPattern).""" + from je_auto_control.utils.transform_window import move_element + return move_element(float(x), float(y), name=name, role=role, + app_name=app_name, automation_id=automation_id) + + +def _resize_element(width: Any, height: Any, name: Optional[str] = None, + role: Optional[str] = None, app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Adapter: resize a UIA element (TransformPattern).""" + from je_auto_control.utils.transform_window import resize_element + return resize_element(float(width), float(height), name=name, role=role, + app_name=app_name, automation_id=automation_id) + + +def _set_window_state(state: str, name: Optional[str] = None, + role: Optional[str] = None, app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Adapter: set a window's visual state normal/maximized/minimized.""" + from je_auto_control.utils.transform_window import set_window_state + return set_window_state(str(state), name=name, role=role, app_name=app_name, + automation_id=automation_id) + + +def _window_interaction_state(name: Optional[str] = None, + role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> Dict[str, Any]: + """Adapter: a window's interaction state (ready/blocked/not_responding).""" + from je_auto_control.utils.transform_window import window_interaction_state + return {"state": window_interaction_state(name=name, role=role, + app_name=app_name, + automation_id=automation_id)} + + +def _legacy_info(name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> Dict[str, Any]: + """Adapter: MSAA IAccessible info of an old control (LegacyIAccessible).""" + from je_auto_control.utils.legacy_accessible import legacy_info + info = legacy_info(name=name, role=role, app_name=app_name, + automation_id=automation_id) + return {"found": info is not None, "info": info} + + +def _legacy_default_action(name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Adapter: fire an old control's MSAA default action (Value/Invoke fallback).""" + from je_auto_control.utils.legacy_accessible import legacy_default_action + return legacy_default_action(name=name, role=role, app_name=app_name, + automation_id=automation_id) + + +def _get_selection(name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> Dict[str, Any]: + """Adapter: a container's selection state (SelectionPattern).""" + from je_auto_control.utils.selection_view import get_selection + selection = get_selection(name=name, role=role, app_name=app_name, + automation_id=automation_id) + return {"found": selection is not None, "selection": selection} + + +def _list_views(name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> Dict[str, Any]: + """Adapter: a control's selectable views (MultipleViewPattern).""" + from je_auto_control.utils.selection_view import list_views + views = list_views(name=name, role=role, app_name=app_name, + automation_id=automation_id) + return {"found": views is not None, "views": views} + + +def _set_view(view: str, name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Adapter: switch a control to a named view (MultipleViewPattern).""" + from je_auto_control.utils.selection_view import set_view + return set_view(str(view), name=name, role=role, app_name=app_name, + automation_id=automation_id) + + def _get_control_text(name: Optional[str] = None, role: Optional[str] = None, app_name: Optional[str] = None, automation_id: Optional[str] = None) -> Dict[str, Any]: @@ -2525,6 +2611,38 @@ def _get_control_text(name: Optional[str] = None, role: Optional[str] = None, automation_id=automation_id)} +def _find_control_text(text: str, ignore_case: Any = True, + name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Adapter: whether text occurs in a control (TextPattern.FindText).""" + from je_auto_control.utils.ax_text import find_control_text + return find_control_text(str(text), ignore_case=bool(ignore_case), name=name, + role=role, app_name=app_name, + automation_id=automation_id) + + +def _select_control_text(text: str, ignore_case: Any = True, + name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Adapter: find + select text in a control (TextPattern.FindText + Select).""" + from je_auto_control.utils.ax_text import select_control_text + return select_control_text(str(text), ignore_case=bool(ignore_case), + name=name, role=role, app_name=app_name, + automation_id=automation_id) + + +def _control_text_attributes(name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> Dict[str, Any]: + """Adapter: read a control selection's font/colour formatting (TextPattern).""" + from je_auto_control.utils.ax_text import control_text_attributes + attrs = control_text_attributes(name=name, role=role, app_name=app_name, + automation_id=automation_id) + return {"found": attrs is not None, "attributes": attrs} + + def _get_selected_text(name: Optional[str] = None, role: Optional[str] = None, app_name: Optional[str] = None, automation_id: Optional[str] = None) -> Dict[str, Any]: @@ -6464,7 +6582,19 @@ def __init__(self): "AC_table_headers": _table_headers, "AC_table_cell": _table_cell, "AC_cell_by_header": _cell_by_header, + "AC_move_element": _move_element, + "AC_resize_element": _resize_element, + "AC_set_window_state": _set_window_state, + "AC_window_interaction_state": _window_interaction_state, + "AC_legacy_info": _legacy_info, + "AC_legacy_default_action": _legacy_default_action, + "AC_get_selection": _get_selection, + "AC_list_views": _list_views, + "AC_set_view": _set_view, "AC_get_control_text": _get_control_text, + "AC_find_control_text": _find_control_text, + "AC_select_control_text": _select_control_text, + "AC_control_text_attributes": _control_text_attributes, "AC_get_selected_text": _get_selected_text, "AC_get_visible_text": _get_visible_text, "AC_read_table": _read_table, diff --git a/je_auto_control/utils/legacy_accessible/__init__.py b/je_auto_control/utils/legacy_accessible/__init__.py new file mode 100644 index 00000000..bb7b5a5a --- /dev/null +++ b/je_auto_control/utils/legacy_accessible/__init__.py @@ -0,0 +1,6 @@ +"""MSAA bridge for old controls UIA can't model (LegacyIAccessiblePattern).""" +from je_auto_control.utils.legacy_accessible.legacy_accessible import ( + legacy_default_action, legacy_info, +) + +__all__ = ["legacy_info", "legacy_default_action"] diff --git a/je_auto_control/utils/legacy_accessible/legacy_accessible.py b/je_auto_control/utils/legacy_accessible/legacy_accessible.py new file mode 100644 index 00000000..16c4ccfd --- /dev/null +++ b/je_auto_control/utils/legacy_accessible/legacy_accessible.py @@ -0,0 +1,39 @@ +"""MSAA bridge for old controls UIA can't model (LegacyIAccessiblePattern). + +Many legacy Win32 / MFC / Delphi controls expose **nothing useful** via the modern +UIA patterns — ``control_get_value`` / ``control_invoke`` / ``control_toggle`` all +return None / do nothing — yet they are fully described through the MSAA +``IAccessible`` bridge: a Name, Value, Description, Role, State and a +**DefaultAction**. ``legacy_accessible`` is the last-resort fallback that still +reads that info and fires the default action, making the long tail of old apps +automatable. + +* :func:`legacy_info` — the MSAA fields ``{name, value, description, + default_action, role, state}``, +* :func:`legacy_default_action` — fire the control's default action. + +Each is a thin dispatch onto the injectable ``accessibility.backends.get_backend()`` +seam — headless-testable via a fake backend; the real ``LegacyIAccessiblePattern`` +calls live in the Windows backend. Imports no ``PySide6``. +""" +from typing import Any, Dict, Optional + + +def legacy_info(name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None, + ) -> Optional[Dict[str, Any]]: + """Return the MSAA ``IAccessible`` info of an old control, or None if absent.""" + from je_auto_control.utils.accessibility.backends import get_backend + return get_backend().legacy_info(name=name, role=role, app_name=app_name, + automation_id=automation_id) + + +def legacy_default_action(name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Fire an old control's MSAA default action (the Value/Invoke/Toggle fallback).""" + from je_auto_control.utils.accessibility.backends import get_backend + return get_backend().legacy_default_action(name=name, role=role, + app_name=app_name, + automation_id=automation_id) diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index 9ddc7998..7b91585c 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -1219,6 +1219,97 @@ def a11y_control_tools() -> List[MCPTool]: handler=h.cell_by_header, annotations=READ_ONLY, ), + MCPTool( + name="ac_move_element", + description=("Move a UIA element to ('x','y') via TransformPattern — " + "element-level, for panels / toolbars / MDI children " + "without their own window. Returns True on success."), + input_schema=schema({"x": {"type": "number"}, + "y": {"type": "number"}, **_M}, + required=["x", "y"]), + handler=h.move_element, + annotations=DESTRUCTIVE, + ), + MCPTool( + name="ac_resize_element", + description=("Resize a UIA element to 'width' x 'height' via " + "TransformPattern. Returns True on success."), + input_schema=schema({"width": {"type": "number"}, + "height": {"type": "number"}, **_M}, + required=["width", "height"]), + handler=h.resize_element, + annotations=DESTRUCTIVE, + ), + MCPTool( + name="ac_set_window_state", + description=("Set a window's visual state via WindowPattern: 'state' " + "= normal / maximized / minimized. Returns True on " + "success."), + input_schema=schema({ + "state": {"type": "string", + "enum": ["normal", "maximized", "minimized"]}, **_M}, + required=["state"]), + handler=h.set_window_state, + annotations=DESTRUCTIVE, + ), + MCPTool( + name="ac_window_interaction_state", + description=("Read a window's interaction state via WindowPattern: " + "{state: ready|blocked_by_modal|not_responding|running|" + "closing|null}. A reliable 'is this window ready / " + "modal-blocked?' signal."), + input_schema=schema(dict(_M)), + handler=h.window_interaction_state, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_legacy_info", + description=("Read an old control's MSAA IAccessible info via " + "LegacyIAccessiblePattern: {found, info:{name, value, " + "description, default_action, role, state}}. The " + "last-resort read for legacy Win32 controls UIA can't " + "model."), + input_schema=schema(dict(_M)), + handler=h.legacy_info, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_legacy_default_action", + description=("Fire an old control's MSAA default action " + "(LegacyIAccessible.DoDefaultAction) — the fallback when " + "Value/Invoke/Toggle all do nothing. Returns True on " + "success."), + input_schema=schema(dict(_M)), + handler=h.legacy_default_action, + annotations=DESTRUCTIVE, + ), + MCPTool( + name="ac_get_selection", + description=("Read a container's selection via SelectionPattern: " + "{found, selection:{items:[names], can_select_multiple, " + "is_required}} — what's selected in a listbox/grid/tab."), + input_schema=schema(dict(_M)), + handler=h.get_selection, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_list_views", + description=("List a control's selectable views via " + "MultipleViewPattern: {found, views:{current, views:" + "[names]}} — e.g. Explorer list / details / tile."), + input_schema=schema(dict(_M)), + handler=h.list_views, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_set_view", + description=("Switch a control to the named 'view' via " + "MultipleViewPattern. Returns True on success."), + input_schema=schema({"view": {"type": "string"}, **_M}, + required=["view"]), + handler=h.set_view, + annotations=DESTRUCTIVE, + ), MCPTool( name="ac_get_control_text", description=("Read a control's full text via TextPattern: " @@ -1228,6 +1319,39 @@ def a11y_control_tools() -> List[MCPTool]: handler=h.get_control_text, annotations=READ_ONLY, ), + MCPTool( + name="ac_find_control_text", + description=("Whether 'text' occurs in a control via " + "TextPattern.FindText (searches the real text content, " + "not OCR). Returns True/False. 'ignore_case' default " + "true."), + input_schema=schema({"text": {"type": "string"}, + "ignore_case": {"type": "boolean"}, **_M}, + required=["text"]), + handler=h.find_control_text, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_select_control_text", + description=("Find 'text' in a control and SELECT its range " + "(TextPattern.FindText + Select) — position the caret / " + "selection before typing to replace it. Returns True on " + "success."), + input_schema=schema({"text": {"type": "string"}, + "ignore_case": {"type": "boolean"}, **_M}, + required=["text"]), + handler=h.select_control_text, + annotations=DESTRUCTIVE, + ), + MCPTool( + name="ac_control_text_attributes", + description=("Read the control selection's formatting via TextPattern " + "attributes: {found, attributes:{font_name, font_size, " + "bold, italic, foreground_color}}."), + input_schema=schema(dict(_M)), + handler=h.control_text_attributes, + annotations=READ_ONLY, + ), MCPTool( name="ac_get_selected_text", description=("Read a control's currently selected text via " diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py index 069f90eb..2604c23a 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers.py @@ -795,6 +795,28 @@ def get_control_text(name=None, role=None, app_name=None, automation_id=None): return _get_control_text(name, role, app_name, automation_id) +def find_control_text(text, ignore_case=True, name=None, role=None, + app_name=None, automation_id=None): + from je_auto_control.utils.executor.action_executor import _find_control_text + return _find_control_text(text, ignore_case, name, role, app_name, + automation_id) + + +def select_control_text(text, ignore_case=True, name=None, role=None, + app_name=None, automation_id=None): + from je_auto_control.utils.executor.action_executor import ( + _select_control_text) + return _select_control_text(text, ignore_case, name, role, app_name, + automation_id) + + +def control_text_attributes(name=None, role=None, app_name=None, + automation_id=None): + from je_auto_control.utils.executor.action_executor import ( + _control_text_attributes) + return _control_text_attributes(name, role, app_name, automation_id) + + def realize_item(item_name, by="name", container_name=None, container_role=None, app_name=None, automation_id=None): from je_auto_control.utils.executor.action_executor import _realize_item @@ -827,6 +849,57 @@ def cell_by_header(row, column_header, name=None, role=None, app_name=None, automation_id) +def move_element(x, y, name=None, role=None, app_name=None, automation_id=None): + from je_auto_control.utils.executor.action_executor import _move_element + return _move_element(x, y, name, role, app_name, automation_id) + + +def resize_element(width, height, name=None, role=None, app_name=None, + automation_id=None): + from je_auto_control.utils.executor.action_executor import _resize_element + return _resize_element(width, height, name, role, app_name, automation_id) + + +def set_window_state(state, name=None, role=None, app_name=None, + automation_id=None): + from je_auto_control.utils.executor.action_executor import _set_window_state + return _set_window_state(state, name, role, app_name, automation_id) + + +def window_interaction_state(name=None, role=None, app_name=None, + automation_id=None): + from je_auto_control.utils.executor.action_executor import ( + _window_interaction_state) + return _window_interaction_state(name, role, app_name, automation_id) + + +def legacy_info(name=None, role=None, app_name=None, automation_id=None): + from je_auto_control.utils.executor.action_executor import _legacy_info + return _legacy_info(name, role, app_name, automation_id) + + +def legacy_default_action(name=None, role=None, app_name=None, + automation_id=None): + from je_auto_control.utils.executor.action_executor import ( + _legacy_default_action) + return _legacy_default_action(name, role, app_name, automation_id) + + +def get_selection(name=None, role=None, app_name=None, automation_id=None): + from je_auto_control.utils.executor.action_executor import _get_selection + return _get_selection(name, role, app_name, automation_id) + + +def list_views(name=None, role=None, app_name=None, automation_id=None): + from je_auto_control.utils.executor.action_executor import _list_views + return _list_views(name, role, app_name, automation_id) + + +def set_view(view, name=None, role=None, app_name=None, automation_id=None): + from je_auto_control.utils.executor.action_executor import _set_view + return _set_view(view, name, role, app_name, automation_id) + + def get_selected_text(name=None, role=None, app_name=None, automation_id=None): from je_auto_control.utils.executor.action_executor import _get_selected_text return _get_selected_text(name, role, app_name, automation_id) diff --git a/je_auto_control/utils/selection_view/__init__.py b/je_auto_control/utils/selection_view/__init__.py new file mode 100644 index 00000000..1efccec3 --- /dev/null +++ b/je_auto_control/utils/selection_view/__init__.py @@ -0,0 +1,6 @@ +"""Container selection state + view switching (Selection / MultipleView patterns).""" +from je_auto_control.utils.selection_view.selection_view import ( + get_selection, list_views, set_view, +) + +__all__ = ["get_selection", "list_views", "set_view"] diff --git a/je_auto_control/utils/selection_view/selection_view.py b/je_auto_control/utils/selection_view/selection_view.py new file mode 100644 index 00000000..64c38540 --- /dev/null +++ b/je_auto_control/utils/selection_view/selection_view.py @@ -0,0 +1,51 @@ +"""Container selection state + view switching (Selection / MultipleView patterns). + +``select_control_item`` (SelectionItemPattern) selects *one* item; the +container-level ``SelectionPattern`` answers the natural follow-up — **what is +currently selected** in a listbox / grid / tab, and **may it select multiple?** — +the assertion target after selecting. ``MultipleViewPattern`` switches a control +between its views (Explorer's list / details / tile / thumbnail), a common +precondition that otherwise needs fragile menu clicking. + +* :func:`get_selection` — ``{items, can_select_multiple, is_required}``, +* :func:`list_views` — ``{current, views: [...]}``, +* :func:`set_view` — switch to a named view. + +Each is a thin dispatch onto the injectable ``accessibility.backends.get_backend()`` +seam — headless-testable via a fake backend; the real UIA calls live in the +Windows backend. Imports no ``PySide6``. +""" +from typing import Any, Dict, Optional + + +def _backend(): + from je_auto_control.utils.accessibility.backends import get_backend + return get_backend() + + +def get_selection(name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None, + ) -> Optional[Dict[str, Any]]: + """Return a container's selection state — ``{items, can_select_multiple, + is_required}`` (SelectionPattern), or None if not found.""" + return _backend().get_selection(name=name, role=role, app_name=app_name, + automation_id=automation_id) + + +def list_views(name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None, + ) -> Optional[Dict[str, Any]]: + """Return a control's selectable views — ``{current, views: [...]}`` + (MultipleViewPattern), or None if not found.""" + return _backend().list_views(name=name, role=role, app_name=app_name, + automation_id=automation_id) + + +def set_view(view: str, name: Optional[str] = None, role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Switch a control to the named view (MultipleViewPattern); True on success.""" + return _backend().set_view(str(view), name=name, role=role, + app_name=app_name, automation_id=automation_id) diff --git a/je_auto_control/utils/transform_window/__init__.py b/je_auto_control/utils/transform_window/__init__.py new file mode 100644 index 00000000..e486f378 --- /dev/null +++ b/je_auto_control/utils/transform_window/__init__.py @@ -0,0 +1,9 @@ +"""Move/resize UIA elements + window state (Transform + Window patterns).""" +from je_auto_control.utils.transform_window.transform_window import ( + move_element, resize_element, set_window_state, window_interaction_state, +) + +__all__ = [ + "move_element", "resize_element", "set_window_state", + "window_interaction_state", +] diff --git a/je_auto_control/utils/transform_window/transform_window.py b/je_auto_control/utils/transform_window/transform_window.py new file mode 100644 index 00000000..6302a2c6 --- /dev/null +++ b/je_auto_control/utils/transform_window/transform_window.py @@ -0,0 +1,61 @@ +"""Move / resize UIA elements and drive window state (Transform + Window patterns). + +This is **UIA-element-level**, not the HWND/title-level geometry in +``window_layout`` / ``window_geometry``. ``TransformPattern`` moves and resizes a +specific control or floating panel (dockable toolbars, MDI children, splitters) +that has no top-level window of its own; ``WindowPattern`` minimizes / maximizes a +window and — most usefully — reports its **interaction state** (``ready`` / +``blocked_by_modal`` / ``not_responding``), a reliable "is this window ready or +modal-blocked?" signal that pixel or title polling can't give. + +* :func:`move_element` / :func:`resize_element` — TransformPattern, +* :func:`set_window_state` — minimize / maximize / restore, +* :func:`window_interaction_state` — the readiness / modal-blocked signal. + +Each is a thin dispatch onto the injectable ``accessibility.backends.get_backend()`` +seam — headless-testable via a fake backend; the real UIA calls live in the +Windows backend. Imports no ``PySide6``. +""" +from typing import Optional + + +def _backend(): + from je_auto_control.utils.accessibility.backends import get_backend + return get_backend() + + +def move_element(x: float, y: float, name: Optional[str] = None, + role: Optional[str] = None, app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Move the matched element to ``(x, y)`` (TransformPattern); True on success.""" + return _backend().move_element(float(x), float(y), name=name, role=role, + app_name=app_name, automation_id=automation_id) + + +def resize_element(width: float, height: float, name: Optional[str] = None, + role: Optional[str] = None, app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Resize the matched element to ``width`` x ``height`` (TransformPattern).""" + return _backend().resize_element(float(width), float(height), name=name, + role=role, app_name=app_name, + automation_id=automation_id) + + +def set_window_state(state: str, name: Optional[str] = None, + role: Optional[str] = None, app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> bool: + """Set a window's visual state — ``normal`` / ``maximized`` / ``minimized``.""" + return _backend().set_window_state(str(state), name=name, role=role, + app_name=app_name, + automation_id=automation_id) + + +def window_interaction_state(name: Optional[str] = None, + role: Optional[str] = None, + app_name: Optional[str] = None, + automation_id: Optional[str] = None) -> Optional[str]: + """Return a window's interaction state — ``ready`` / ``blocked_by_modal`` / + ``not_responding`` / ``running`` / ``closing`` (WindowPattern), or None.""" + return _backend().window_interaction_state(name=name, role=role, + app_name=app_name, + automation_id=automation_id) diff --git a/test/unit_test/headless/test_ax_text_batch.py b/test/unit_test/headless/test_ax_text_batch.py index 617ec9a3..127613fa 100644 --- a/test/unit_test/headless/test_ax_text_batch.py +++ b/test/unit_test/headless/test_ax_text_batch.py @@ -1,10 +1,14 @@ -"""Headless tests for native TextPattern reads (fake backend via the seam).""" +"""Headless tests for native TextPattern reads + find/select/attrs (fake seam).""" import je_auto_control as ac from je_auto_control.utils.accessibility.backends import base as backend_base from je_auto_control.utils.ax_text import ( - get_control_text, get_selected_text, get_visible_text, + control_text_attributes, find_control_text, get_control_text, + get_selected_text, get_visible_text, select_control_text, ) +_ATTRS = {"font_name": "Consolas", "font_size": 11.0, "bold": True, + "italic": False, "foreground_color": 16711680} + class _FakeBackend(backend_base.AccessibilityBackend): name = "fake" @@ -30,6 +34,20 @@ def visible_text(self, name=None, role=None, app_name=None, self.calls.append(("visible", name)) return "line 1\nline 2" + def find_text(self, text="", ignore_case=True, name=None, role=None, + app_name=None, automation_id=None): + self.calls.append(("find", text, ignore_case)) + return "line 2" in str(text) + + def select_text(self, text="", ignore_case=True, name=None, role=None, + app_name=None, automation_id=None): + self.calls.append(("select", text)) + return "line 2" in str(text) + + def text_attributes(self, name=None, role=None, app_name=None, + automation_id=None): + return dict(_ATTRS) + def _inject(monkeypatch, backend): import je_auto_control.utils.accessibility.backends as backends @@ -58,6 +76,26 @@ def test_visible_text(monkeypatch): assert fake.calls[0][0] == "visible" +def test_find_control_text(monkeypatch): + fake = _FakeBackend() + _inject(monkeypatch, fake) + assert find_control_text("line 2", name="Editor") is True + assert find_control_text("absent", name="Editor") is False + assert ("find", "line 2", True) in fake.calls + + +def test_select_control_text(monkeypatch): + fake = _FakeBackend() + _inject(monkeypatch, fake) + assert select_control_text("line 2", ignore_case=False, name="Editor") is True + assert ("select", "line 2") in fake.calls + + +def test_control_text_attributes(monkeypatch): + _inject(monkeypatch, _FakeBackend()) + assert control_text_attributes(name="Editor") == _ATTRS + + def test_unsupported_backend_raises(monkeypatch): from je_auto_control.utils.accessibility.element import ( AccessibilityNotAvailableError) @@ -80,18 +118,21 @@ def test_executor_adapter_wraps_text(monkeypatch): def test_wiring(): known = set(ac.executor.known_commands()) - assert {"AC_get_control_text", "AC_get_selected_text", - "AC_get_visible_text"} <= known + assert {"AC_get_control_text", "AC_get_selected_text", "AC_get_visible_text", + "AC_find_control_text", "AC_select_control_text", + "AC_control_text_attributes"} <= known from je_auto_control.utils.mcp_server.tools import build_default_tool_registry names = {t.name for t in build_default_tool_registry()} - assert {"ac_get_control_text", "ac_get_selected_text", - "ac_get_visible_text"} <= names + assert {"ac_get_control_text", "ac_find_control_text", + "ac_select_control_text", "ac_control_text_attributes"} <= names from je_auto_control.gui.script_builder.command_schema import _build_specs specs = {s.command for s in _build_specs()} - assert {"AC_get_control_text", "AC_get_selected_text", - "AC_get_visible_text"} <= specs + assert {"AC_get_control_text", "AC_find_control_text", + "AC_select_control_text", "AC_control_text_attributes"} <= specs def test_facade_exports(): - for name in ("get_control_text", "get_selected_text", "get_visible_text"): + for name in ("get_control_text", "get_selected_text", "get_visible_text", + "find_control_text", "select_control_text", + "control_text_attributes"): assert hasattr(ac, name) and name in ac.__all__ diff --git a/test/unit_test/headless/test_legacy_accessible_batch.py b/test/unit_test/headless/test_legacy_accessible_batch.py new file mode 100644 index 00000000..e0b59c9b --- /dev/null +++ b/test/unit_test/headless/test_legacy_accessible_batch.py @@ -0,0 +1,87 @@ +"""Headless tests for the MSAA bridge (LegacyIAccessiblePattern, fake backend).""" +import je_auto_control as ac +from je_auto_control.utils.accessibility.backends import base as backend_base +from je_auto_control.utils.legacy_accessible import ( + legacy_default_action, legacy_info, +) + +_INFO = {"name": "OK", "value": "", "description": "Accept the dialog", + "default_action": "Press", "role": 43, "state": 1048576} + + +class _FakeBackend(backend_base.AccessibilityBackend): + name = "fake" + available = True + + def __init__(self, info=None): + self.info = info + self.actions = [] + + def legacy_info(self, name=None, role=None, app_name=None, automation_id=None): + return self.info + + def legacy_default_action(self, name=None, role=None, app_name=None, + automation_id=None): + self.actions.append(name) + return True + + +def _inject(monkeypatch, backend): + import je_auto_control.utils.accessibility.backends as backends + monkeypatch.setattr(backends, "_cached_backend", backend, raising=False) + + +def test_legacy_info_dispatch(monkeypatch): + _inject(monkeypatch, _FakeBackend(dict(_INFO))) + assert legacy_info(name="OK", role="button") == _INFO + + +def test_legacy_info_not_found(monkeypatch): + _inject(monkeypatch, _FakeBackend(None)) + assert legacy_info(name="missing") is None + + +def test_legacy_default_action(monkeypatch): + fake = _FakeBackend() + _inject(monkeypatch, fake) + assert legacy_default_action(name="OK") is True + assert fake.actions == ["OK"] + + +def test_unsupported_backend_raises(monkeypatch): + from je_auto_control.utils.accessibility.element import ( + AccessibilityNotAvailableError) + _inject(monkeypatch, backend_base.AccessibilityBackend()) + try: + legacy_info(name="x") + raised = False + except AccessibilityNotAvailableError: + raised = True + assert raised is True + + +# --- wiring --------------------------------------------------------------- + +def test_executor_adapters(monkeypatch): + _inject(monkeypatch, _FakeBackend(dict(_INFO))) + from je_auto_control.utils.executor.action_executor import ( + _legacy_default_action, _legacy_info) + out = _legacy_info(name="OK") + assert out["found"] is True and out["info"]["default_action"] == "Press" + assert _legacy_default_action(name="OK") is True + + +def test_wiring(): + known = set(ac.executor.known_commands()) + assert {"AC_legacy_info", "AC_legacy_default_action"} <= known + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert {"ac_legacy_info", "ac_legacy_default_action"} <= names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert {"AC_legacy_info", "AC_legacy_default_action"} <= specs + + +def test_facade_exports(): + for name in ("legacy_info", "legacy_default_action"): + assert hasattr(ac, name) and name in ac.__all__ diff --git a/test/unit_test/headless/test_selection_view_batch.py b/test/unit_test/headless/test_selection_view_batch.py new file mode 100644 index 00000000..ee65ad4b --- /dev/null +++ b/test/unit_test/headless/test_selection_view_batch.py @@ -0,0 +1,92 @@ +"""Headless tests for container selection + views (fake backend seam).""" +import je_auto_control as ac +from je_auto_control.utils.accessibility.backends import base as backend_base +from je_auto_control.utils.selection_view import ( + get_selection, list_views, set_view, +) + +_SELECTION = {"items": ["Row 2", "Row 4"], "can_select_multiple": True, + "is_required": False} +_VIEWS = {"current": "Details", "views": ["List", "Details", "Tiles"]} + + +class _FakeBackend(backend_base.AccessibilityBackend): + name = "fake" + available = True + + def __init__(self): + self.applied_view = None + + def get_selection(self, name=None, role=None, app_name=None, + automation_id=None): + return dict(_SELECTION) + + def list_views(self, name=None, role=None, app_name=None, automation_id=None): + return dict(_VIEWS) + + def set_view(self, view="", name=None, role=None, app_name=None, + automation_id=None): + self.applied_view = view + return view in _VIEWS["views"] + + +def _inject(monkeypatch, backend): + import je_auto_control.utils.accessibility.backends as backends + monkeypatch.setattr(backends, "_cached_backend", backend, raising=False) + + +def test_get_selection(monkeypatch): + _inject(monkeypatch, _FakeBackend()) + assert get_selection(name="List") == _SELECTION + + +def test_list_views(monkeypatch): + _inject(monkeypatch, _FakeBackend()) + assert list_views(name="Folder") == _VIEWS + + +def test_set_view_known_and_unknown(monkeypatch): + fake = _FakeBackend() + _inject(monkeypatch, fake) + assert set_view("Tiles", name="Folder") is True + assert fake.applied_view == "Tiles" + assert set_view("Nonexistent", name="Folder") is False + + +def test_unsupported_backend_raises(monkeypatch): + from je_auto_control.utils.accessibility.element import ( + AccessibilityNotAvailableError) + _inject(monkeypatch, backend_base.AccessibilityBackend()) + try: + get_selection(name="x") + raised = False + except AccessibilityNotAvailableError: + raised = True + assert raised is True + + +# --- wiring --------------------------------------------------------------- + +def test_executor_adapters(monkeypatch): + _inject(monkeypatch, _FakeBackend()) + from je_auto_control.utils.executor.action_executor import ( + _get_selection, _list_views, _set_view) + assert _get_selection(name="L")["selection"]["can_select_multiple"] is True + assert _list_views(name="F")["views"]["current"] == "Details" + assert _set_view("List", name="F") is True + + +def test_wiring(): + known = set(ac.executor.known_commands()) + assert {"AC_get_selection", "AC_list_views", "AC_set_view"} <= known + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert {"ac_get_selection", "ac_list_views", "ac_set_view"} <= names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert {"AC_get_selection", "AC_list_views", "AC_set_view"} <= specs + + +def test_facade_exports(): + for name in ("get_selection", "list_views", "set_view"): + assert hasattr(ac, name) and name in ac.__all__ diff --git a/test/unit_test/headless/test_transform_window_batch.py b/test/unit_test/headless/test_transform_window_batch.py new file mode 100644 index 00000000..a7a8d4ce --- /dev/null +++ b/test/unit_test/headless/test_transform_window_batch.py @@ -0,0 +1,112 @@ +"""Headless tests for Transform + Window patterns (fake backend seam).""" +import je_auto_control as ac +from je_auto_control.utils.accessibility.backends import base as backend_base +from je_auto_control.utils.transform_window import ( + move_element, resize_element, set_window_state, window_interaction_state, +) + + +class _FakeBackend(backend_base.AccessibilityBackend): + name = "fake" + available = True + + def __init__(self, interaction="ready"): + self.interaction = interaction + self.calls = [] + + def move_element(self, x=0.0, y=0.0, name=None, role=None, app_name=None, + automation_id=None): + self.calls.append(("move", x, y, name)) + return True + + def resize_element(self, width=0.0, height=0.0, name=None, role=None, + app_name=None, automation_id=None): + self.calls.append(("resize", width, height, name)) + return True + + def set_window_state(self, state="normal", name=None, role=None, + app_name=None, automation_id=None): + self.calls.append(("state", state, name)) + return state in ("normal", "maximized", "minimized") + + def window_interaction_state(self, name=None, role=None, app_name=None, + automation_id=None): + return self.interaction + + +def _inject(monkeypatch, backend): + import je_auto_control.utils.accessibility.backends as backends + monkeypatch.setattr(backends, "_cached_backend", backend, raising=False) + + +def test_move_and_resize_dispatch(monkeypatch): + fake = _FakeBackend() + _inject(monkeypatch, fake) + assert move_element(100, 200, name="Panel") is True + assert resize_element(640, 480, name="Panel") is True + assert ("move", 100.0, 200.0, "Panel") in fake.calls + assert ("resize", 640.0, 480.0, "Panel") in fake.calls + + +def test_set_window_state(monkeypatch): + fake = _FakeBackend() + _inject(monkeypatch, fake) + assert set_window_state("maximized", name="Editor") is True + assert ("state", "maximized", "Editor") in fake.calls + + +def test_window_interaction_state(monkeypatch): + _inject(monkeypatch, _FakeBackend(interaction="blocked_by_modal")) + assert window_interaction_state(name="Editor") == "blocked_by_modal" + + +def test_unsupported_backend_raises(monkeypatch): + from je_auto_control.utils.accessibility.element import ( + AccessibilityNotAvailableError) + _inject(monkeypatch, backend_base.AccessibilityBackend()) + try: + move_element(1, 2, name="x") + raised = False + except AccessibilityNotAvailableError: + raised = True + assert raised is True + + +def test_base_set_window_state_rejects_unknown(): + # the real Windows backend returns False for an unknown state; the base + # ABC default still routes through _unsupported, so verify the mapping guard + from je_auto_control.utils.accessibility.backends import windows_backend + backend = windows_backend.WindowsAccessibilityBackend.__new__( + windows_backend.WindowsAccessibilityBackend) + assert backend.set_window_state("not-a-state", name="x") is False + + +# --- wiring --------------------------------------------------------------- + +def test_executor_adapters(monkeypatch): + _inject(monkeypatch, _FakeBackend(interaction="not_responding")) + from je_auto_control.utils.executor.action_executor import ( + _move_element, _set_window_state, _window_interaction_state) + assert _move_element(1, 2, name="P") is True + assert _set_window_state("minimized", name="W") is True + assert _window_interaction_state(name="W") == {"state": "not_responding"} + + +def test_wiring(): + known = set(ac.executor.known_commands()) + assert {"AC_move_element", "AC_resize_element", "AC_set_window_state", + "AC_window_interaction_state"} <= known + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert {"ac_move_element", "ac_resize_element", "ac_set_window_state", + "ac_window_interaction_state"} <= names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert {"AC_move_element", "AC_resize_element", "AC_set_window_state", + "AC_window_interaction_state"} <= specs + + +def test_facade_exports(): + for name in ("move_element", "resize_element", "set_window_state", + "window_interaction_state"): + assert hasattr(ac, name) and name in ac.__all__