Add transform_window: move/resize UIA elements + window state

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-25) — Move / Resize Elements + Window State (UIA Transform + Window)
+
+Move a floating panel, resize a control, and know if a window is modal-blocked. Full reference: [`docs/source/Eng/doc/new_features/v198_features_doc.rst`](docs/source/Eng/doc/new_features/v198_features_doc.rst).
+
+- **`move_element` / `resize_element` / `set_window_state` / `window_interaction_state`** (`AC_move_element`, `AC_resize_element`, `AC_set_window_state`, `AC_window_interaction_state`): this is UIA-**element-level**, not the HWND/title-level geometry in `window_layout`. `TransformPattern` moves/resizes a specific control or floating panel (dockable toolbars, MDI children, splitters) with no top-level window of its own; `WindowPattern` minimizes/maximizes a window and reports its **interaction state** (`ready` / `blocked_by_modal` / `not_responding`) — a reliable "is this window ready or modal-blocked?" signal pixel/title polling can't give. Dispatched through the injectable accessibility backend seam (headless-testable via a fake backend; real UIA in the Windows backend). No `PySide6`.
+
 ## What's new (2026-06-25) — Table Headers + Cell Addressing (UIA TablePattern)
 
 Assert "the Status column of row 5 says Shipped" — by header, not by guessing indices. Full reference: [`docs/source/Eng/doc/new_features/v197_features_doc.rst`](docs/source/Eng/doc/new_features/v197_features_doc.rst).

docs/source/Eng/doc/new_features/v198_features_doc.rst

@@ -0,0 +1,46 @@
+Move / Resize Elements + Window State (UIA Transform + Window)
+==============================================================
+
+This is **UIA-element-level**, not the HWND / title-level geometry in
+``window_layout`` / ``window_geometry``. ``TransformPattern`` moves and resizes a
+specific control or floating panel (dockable toolbars, MDI children, splitters)
+that has no top-level window of its own; ``WindowPattern`` minimizes / maximizes a
+window and — most usefully — reports its **interaction state**, a reliable "is
+this window ready or modal-blocked?" signal that pixel or title polling can't give.
+
+* :func:`move_element` / :func:`resize_element` — TransformPattern,
+* :func:`set_window_state` — minimize / maximize / restore,
+* :func:`window_interaction_state` — the readiness / modal-blocked signal.
+
+Each is a thin dispatch onto the injectable ``accessibility.backends.get_backend()``
+seam — headless-testable via a fake backend; the real UIA calls live in the
+Windows backend. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import (move_element, resize_element,
+                                 set_window_state, window_interaction_state)
+
+    move_element(100, 200, name="Tool Palette")     # reposition a floating panel
+    resize_element(640, 480, name="Preview")        # resize a control
+    set_window_state("maximized", name="Editor")    # normal / maximized / minimized
+
+    if window_interaction_state(name="Editor") == "ready":
+        ...   # not "blocked_by_modal" / "not_responding" — safe to drive
+
+The element / window is located by ``name`` / ``role`` / ``app_name`` /
+``automation_id`` (same as the other native-control actions). The actions return
+``bool``; ``window_interaction_state`` returns ``ready`` / ``blocked_by_modal`` /
+``not_responding`` / ``running`` / ``closing`` (or ``None`` if not found).
+
+Executor commands
+-----------------
+
+``AC_move_element`` (``x`` / ``y``), ``AC_resize_element`` (``width`` /
+``height``), ``AC_set_window_state`` (``state``) and
+``AC_window_interaction_state`` (``{state}``). They are exposed as the matching
+``ac_*`` MCP tools (the actions destructive, the read read-only) and as Script
+Builder commands under **Native UI**.

docs/source/Zh/doc/new_features/v198_features_doc.rst

@@ -0,0 +1,41 @@
+移動 / 縮放元素 + 視窗狀態(UIA Transform + Window)
+==================================================
+
+這是 **UIA 元素層級**,而非 ``window_layout`` / ``window_geometry`` 的 HWND / title 層級幾何。
+``TransformPattern`` 移動與縮放某個沒有自己頂層視窗的特定控制項或浮動面板(可停靠工具列、MDI
+子視窗、分隔器);``WindowPattern`` 最小化 / 最大化視窗,並——最有用地——回報其 **interaction
+state(互動狀態)**,這是像素或標題輪詢給不出的可靠「這視窗是否就緒 / 被 modal 擋住?」訊號。
+
+* :func:`move_element` / :func:`resize_element` ——TransformPattern,
+* :func:`set_window_state` ——最小化 / 最大化 / 還原,
+* :func:`window_interaction_state` ——就緒 / 被 modal 擋住的訊號。
+
+每個都是對可注入的 ``accessibility.backends.get_backend()`` 接縫的薄分派——可透過注入 fake
+backend 進行無頭測試;真正的 UIA 呼叫位於 Windows 後端。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import (move_element, resize_element,
+                                 set_window_state, window_interaction_state)
+
+    move_element(100, 200, name="Tool Palette")     # 重新定位浮動面板
+    resize_element(640, 480, name="Preview")        # 縮放控制項
+    set_window_state("maximized", name="Editor")    # normal / maximized / minimized
+
+    if window_interaction_state(name="Editor") == "ready":
+        ...   # 不是 "blocked_by_modal" / "not_responding"——可安全驅動
+
+元素 / 視窗以 ``name`` / ``role`` / ``app_name`` / ``automation_id`` 定位(與其他原生控制動作
+相同)。動作回傳 ``bool``;``window_interaction_state`` 回傳 ``ready`` / ``blocked_by_modal`` /
+``not_responding`` / ``running`` / ``closing``(找不到則為 ``None``)。
+
+執行器指令
+----------
+
+``AC_move_element``(``x`` / ``y``)、``AC_resize_element``(``width`` / ``height``)、
+``AC_set_window_state``(``state``)與 ``AC_window_interaction_state``(``{state}``)。皆以對應的
+``ac_*`` MCP 工具(動作類為破壞性、讀取類為唯讀)及 Script Builder 指令(位於 **Native UI**
+分類下)形式提供。

je_auto_control/__init__.py

@@ -76,6 +76,10 @@
 from je_auto_control.utils.table_pattern import (
     cell_by_header, table_cell, table_headers,
 )
+# Move/resize UIA elements + window state (Transform + Window patterns)
+from je_auto_control.utils.transform_window import (
+    move_element, resize_element, set_window_state, window_interaction_state,
+)
 # Rich clipboard formats — RTF + CSV/TSV codecs and Windows get / set
 from je_auto_control.utils.clipboard_rich_formats import (
     build_rtf, csv_to_rows, get_clipboard_csv, get_clipboard_rtf, rows_to_csv,
@@ -1679,6 +1683,8 @@ def start_autocontrol_gui(*args, **kwargs):
     "realize_item",
     "get_element_properties", "is_element_enabled",
     "table_headers", "table_cell", "cell_by_header",
+    "move_element", "resize_element", "set_window_state",
+    "window_interaction_state",
     "build_rtf", "rtf_to_text", "rows_to_csv", "csv_to_rows",
     "set_clipboard_rtf", "get_clipboard_rtf",
     "set_clipboard_csv", "get_clipboard_csv",

je_auto_control/gui/script_builder/command_schema.py

@@ -1643,6 +1643,29 @@ def _add_native_control_specs(specs: List[CommandSpec]) -> None:
                 FieldSpec("column_header", FieldType.STRING)) + fields,
         description="Read the cell at (row, named column) — assert by header.",
     ))
+    specs.append(CommandSpec(
+        "AC_move_element", "Native UI", "Move Element (Transform)",
+        fields=(FieldSpec("x", FieldType.FLOAT),
+                FieldSpec("y", FieldType.FLOAT)) + fields,
+        description="Move a UIA element to (x, y) (TransformPattern).",
+    ))
+    specs.append(CommandSpec(
+        "AC_resize_element", "Native UI", "Resize Element (Transform)",
+        fields=(FieldSpec("width", FieldType.FLOAT),
+                FieldSpec("height", FieldType.FLOAT)) + fields,
+        description="Resize a UIA element (TransformPattern).",
+    ))
+    specs.append(CommandSpec(
+        "AC_set_window_state", "Native UI", "Set Window State",
+        fields=(FieldSpec("state", FieldType.ENUM, default="normal",
+                          choices=("normal", "maximized", "minimized")),) + fields,
+        description="Minimize / maximize / restore a window (WindowPattern).",
+    ))
+    specs.append(CommandSpec(
+        "AC_window_interaction_state", "Native UI", "Window Interaction State",
+        fields=fields,
+        description="Read window readiness (ready/blocked_by_modal/...).",
+    ))
     specs.append(CommandSpec(
         "AC_get_control_text", "Native UI", "Get Control Text",
         fields=fields,

je_auto_control/utils/accessibility/backends/base.py

@@ -181,6 +181,38 @@
         row_span, column_span}`` (GridPattern.GetItem + GridItemPattern)."""
         self._unsupported("get_grid_cell")
 
+    # --- transform + window patterns (UIA-element-level) --------------------
+
+    def move_element(self, x: float = 0.0, y: float = 0.0,
+                     name: Optional[str] = None, role: Optional[str] = None,
+                     app_name: Optional[str] = None,
+                     automation_id: Optional[str] = None) -> bool:
+        """Move the matched element to ``(x, y)`` (TransformPattern); True on success."""
+        self._unsupported("move_element")
+
+    def resize_element(self, width: float = 0.0, height: float = 0.0,
+                       name: Optional[str] = None, role: Optional[str] = None,
+                       app_name: Optional[str] = None,
+                       automation_id: Optional[str] = None) -> bool:
+        """Resize the matched element (TransformPattern); True on success."""
+        self._unsupported("resize_element")
+
+    def set_window_state(self, state: str = "normal",
+                         name: Optional[str] = None, role: Optional[str] = None,
+                         app_name: Optional[str] = None,
+                         automation_id: Optional[str] = None) -> bool:
+        """Set a window's visual state ``normal`` / ``maximized`` / ``minimized``."""
+        self._unsupported("set_window_state")
+
+    def window_interaction_state(self, name: Optional[str] = None,
+                                 role: Optional[str] = None,
+                                 app_name: Optional[str] = None,
+                                 automation_id: Optional[str] = None,
+                                 ) -> Optional[str]:
+        """Return a window's interaction state — ``ready`` / ``blocked_by_modal`` /
+        ``not_responding`` / ``running`` / ``closing`` (WindowPattern), or None."""
+        self._unsupported("window_interaction_state")
+
     def _unsupported(self, operation: str):
         """Raise a clear error for an action this backend can't perform."""
         raise AccessibilityNotAvailableError(

je_auto_control/utils/accessibility/backends/windows_backend.py

@@ -34,8 +34,15 @@
 _UIA_VIRTUALIZEDITEM_PATTERN_ID = 10020
 _UIA_TABLE_PATTERN_ID = 10012
 _UIA_GRIDITEM_PATTERN_ID = 10007
+_UIA_TRANSFORM_PATTERN_ID = 10016
+_UIA_WINDOW_PATTERN_ID = 10009
 _UIA_AUTOMATIONID_PROPERTY = 30011
 _EXPAND_STATES = {0: "collapsed", 1: "expanded", 2: "partial", 3: "leaf"}
+_WINDOW_VISUAL_STATES = {"normal": 0, "maximized": 1, "minimized": 2}
+_WINDOW_INTERACTION_STATES = {
+    0: "running", 1: "closing", 2: "ready", 3: "blocked_by_modal",
+    4: "not_responding",
+}
 
 
 def _is_available() -> bool:
@@ -307,6 +314,43 @@ def get_properties(self, name=None, role=None, app_name=None,
             return None
         return _read_properties(raw)
 
+    def move_element(self, x=0.0, y=0.0, name=None, role=None, app_name=None,
+                     automation_id=None):
+        return self._invoke_pattern_method(
+            name, role, app_name, automation_id, _UIA_TRANSFORM_PATTERN_ID,
+            "IUIAutomationTransformPattern",
+            lambda pattern: pattern.Move(float(x), float(y)))
+
+    def resize_element(self, width=0.0, height=0.0, name=None, role=None,
+                       app_name=None, automation_id=None):
+        return self._invoke_pattern_method(
+            name, role, app_name, automation_id, _UIA_TRANSFORM_PATTERN_ID,
+            "IUIAutomationTransformPattern",
+            lambda pattern: pattern.Resize(float(width), float(height)))
+
+    def set_window_state(self, state="normal", name=None, role=None,
+                         app_name=None, automation_id=None):
+        visual = _WINDOW_VISUAL_STATES.get(str(state).lower())
+        if visual is None:
+            return False
+        return self._invoke_pattern_method(
+            name, role, app_name, automation_id, _UIA_WINDOW_PATTERN_ID,
+            "IUIAutomationWindowPattern",
+            lambda pattern: pattern.SetWindowVisualState(visual))
+
+    def window_interaction_state(self, name=None, role=None, app_name=None,
+                                 automation_id=None) -> Optional[str]:
+        raw = self._find_raw(name, role, app_name, automation_id)
+        pattern = self._pattern(raw, _UIA_WINDOW_PATTERN_ID,
+                                "IUIAutomationWindowPattern") if raw else None
+        if pattern is None:
+            return None
+        try:
+            return _WINDOW_INTERACTION_STATES.get(
+                int(pattern.CurrentWindowInteractionState))
+        except (OSError, AttributeError, ValueError, TypeError):
+            return None
+
     def get_table_headers(self, name=None, role=None, app_name=None,
                           automation_id=None) -> Optional[Dict[str, Any]]:
         raw = self._find_raw(name, role, app_name, automation_id)

je_auto_control/utils/executor/action_executor.py

@@ -2516,6 +2516,44 @@ def _cell_by_header(row: Any, column_header: str, name: Optional[str] = None,
     return {"found": value is not None, "value": value}
 
 
+def _move_element(x: Any, y: Any, name: Optional[str] = None,
+                  role: Optional[str] = None, app_name: Optional[str] = None,
+                  automation_id: Optional[str] = None) -> bool:
+    """Adapter: move a UIA element to (x, y) (TransformPattern)."""
+    from je_auto_control.utils.transform_window import move_element
+    return move_element(float(x), float(y), name=name, role=role,
+                        app_name=app_name, automation_id=automation_id)
+
+
+def _resize_element(width: Any, height: Any, name: Optional[str] = None,
+                    role: Optional[str] = None, app_name: Optional[str] = None,
+                    automation_id: Optional[str] = None) -> bool:
+    """Adapter: resize a UIA element (TransformPattern)."""
+    from je_auto_control.utils.transform_window import resize_element
+    return resize_element(float(width), float(height), name=name, role=role,
+                          app_name=app_name, automation_id=automation_id)
+
+
+def _set_window_state(state: str, name: Optional[str] = None,
+                      role: Optional[str] = None, app_name: Optional[str] = None,
+                      automation_id: Optional[str] = None) -> bool:
+    """Adapter: set a window's visual state normal/maximized/minimized."""
+    from je_auto_control.utils.transform_window import set_window_state
+    return set_window_state(str(state), name=name, role=role, app_name=app_name,
+                            automation_id=automation_id)
+
+
+def _window_interaction_state(name: Optional[str] = None,
+                              role: Optional[str] = None,
+                              app_name: Optional[str] = None,
+                              automation_id: Optional[str] = None) -> Dict[str, Any]:
+    """Adapter: a window's interaction state (ready/blocked/not_responding)."""
+    from je_auto_control.utils.transform_window import window_interaction_state
+    return {"state": window_interaction_state(name=name, role=role,
+                                              app_name=app_name,
+                                              automation_id=automation_id)}
+
+
 def _get_control_text(name: Optional[str] = None, role: Optional[str] = None,
                       app_name: Optional[str] = None,
                       automation_id: Optional[str] = None) -> Dict[str, Any]:
@@ -6464,6 +6502,10 @@ def __init__(self):
             "AC_table_headers": _table_headers,
             "AC_table_cell": _table_cell,
             "AC_cell_by_header": _cell_by_header,
+            "AC_move_element": _move_element,
+            "AC_resize_element": _resize_element,
+            "AC_set_window_state": _set_window_state,
+            "AC_window_interaction_state": _window_interaction_state,
             "AC_get_control_text": _get_control_text,
             "AC_get_selected_text": _get_selected_text,
             "AC_get_visible_text": _get_visible_text,

je_auto_control/utils/mcp_server/tools/_factories.py

@@ -1219,6 +1219,49 @@ def a11y_control_tools() -> List[MCPTool]:
             handler=h.cell_by_header,
             annotations=READ_ONLY,
         ),
+        MCPTool(
+            name="ac_move_element",
+            description=("Move a UIA element to ('x','y') via TransformPattern — "
+                         "element-level, for panels / toolbars / MDI children "
+                         "without their own window. Returns True on success."),
+            input_schema=schema({"x": {"type": "number"},
+                                 "y": {"type": "number"}, **_M},
+                                required=["x", "y"]),
+            handler=h.move_element,
+            annotations=DESTRUCTIVE,
+        ),
+        MCPTool(
+            name="ac_resize_element",
+            description=("Resize a UIA element to 'width' x 'height' via "
+                         "TransformPattern. Returns True on success."),
+            input_schema=schema({"width": {"type": "number"},
+                                 "height": {"type": "number"}, **_M},
+                                required=["width", "height"]),
+            handler=h.resize_element,
+            annotations=DESTRUCTIVE,
+        ),
+        MCPTool(
+            name="ac_set_window_state",
+            description=("Set a window's visual state via WindowPattern: 'state' "
+                         "= normal / maximized / minimized. Returns True on "
+                         "success."),
+            input_schema=schema({
+                "state": {"type": "string",
+                          "enum": ["normal", "maximized", "minimized"]}, **_M},
+                required=["state"]),
+            handler=h.set_window_state,
+            annotations=DESTRUCTIVE,
+        ),
+        MCPTool(
+            name="ac_window_interaction_state",
+            description=("Read a window's interaction state via WindowPattern: "
+                         "{state: ready|blocked_by_modal|not_responding|running|"
+                         "closing|null}. A reliable 'is this window ready / "
+                         "modal-blocked?' signal."),
+            input_schema=schema(dict(_M)),
+            handler=h.window_interaction_state,
+            annotations=READ_ONLY,
+        ),
         MCPTool(
             name="ac_get_control_text",
             description=("Read a control's full text via TextPattern: "