Add ORB feature matching (rotation/scale/theme-robust template location)

README/WHATS_NEW_zh-CN.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — ORB 特征匹配(对旋转 / 缩放 / 主题稳健)
+
+即使目标旋转、缩放或换主题也能找到。完整参考:[`docs/source/Zh/doc/new_features/v131_features_doc.rst`](../docs/source/Zh/doc/new_features/v131_features_doc.rst)。
+
+- **`feature_match`**(`AC_feature_match`):像素模板匹配(`match_template` / `match_masked`)是做像素相关运算,因此目标一旦旋转、以未列出的倍率缩放或重新上色(亮 / 暗主题、hover)就会失效。本功能匹配 ORB *关键点*并拟合 RANSAC 单应矩阵,返回四个投影 `corners`、`center`、`inliers` 内点数与内点比例 `score`。ORB 边界 / patch 尺寸会针对图标大小的模板自动缩小(OpenCV 预设会将其舍弃)。仅用 OpenCV 核心(不需 contrib);可注入 haystack → 无头可测。
+
 ## 本次更新 (2026-06-23) — 结构相似度(SSIM)比较
 
 会告诉你*哪里*变了的感知式画面比较。完整参考:[`docs/source/Zh/doc/new_features/v130_features_doc.rst`](../docs/source/Zh/doc/new_features/v130_features_doc.rst)。

README/WHATS_NEW_zh-TW.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — ORB 特徵比對(對旋轉 / 縮放 / 主題穩健)
+
+即使目標旋轉、縮放或換主題也能找到。完整參考:[`docs/source/Zh/doc/new_features/v131_features_doc.rst`](../docs/source/Zh/doc/new_features/v131_features_doc.rst)。
+
+- **`feature_match`**(`AC_feature_match`):像素模板比對(`match_template` / `match_masked`)是做像素相關運算,因此目標一旦旋轉、以未列出的倍率縮放或重新上色(亮 / 暗主題、hover)就會失效。本功能比對 ORB *關鍵點*並擬合 RANSAC 單應矩陣,回傳四個投影 `corners`、`center`、`inliers` 內點數與內點比例 `score`。ORB 邊界 / patch 尺寸會針對圖示大小的模板自動縮小(OpenCV 預設會將其捨棄)。僅用 OpenCV 核心(不需 contrib);可注入 haystack → 無頭可測。
+
 ## 本次更新 (2026-06-23) — 結構相似度(SSIM)比較
 
 會告訴你*哪裡*變了的感知式畫面比較。完整參考:[`docs/source/Zh/doc/new_features/v130_features_doc.rst`](../docs/source/Zh/doc/new_features/v130_features_doc.rst)。

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-23) — ORB Feature Matching (Rotation / Scale / Theme Robust)
+
+Find a target even when it is rotated, rescaled or re-themed. Full reference: [`docs/source/Eng/doc/new_features/v131_features_doc.rst`](docs/source/Eng/doc/new_features/v131_features_doc.rst).
+
+- **`feature_match`** (`AC_feature_match`): pixel template matching (`match_template` / `match_masked`) correlates pixels, so it breaks the moment the target is rotated, scaled by an unlisted factor, or re-coloured (light/dark theme, hover). This matches ORB *keypoints* and fits a RANSAC homography, returning the four projected `corners`, the `center`, the `inliers` count and an inlier-fraction `score`. ORB border/patch sizes auto-scale down for icon-sized templates (OpenCV's defaults reject them). Core OpenCV only (no contrib); injectable haystack → headless-testable.
+
 ## What's new (2026-06-23) — Structural-Similarity (SSIM) Comparison
 
 Perceptual screen comparison that tells you *what* changed. Full reference: [`docs/source/Eng/doc/new_features/v130_features_doc.rst`](docs/source/Eng/doc/new_features/v130_features_doc.rst).

docs/source/Eng/doc/new_features/v131_features_doc.rst

@@ -0,0 +1,43 @@
+ORB Feature Matching (Rotation / Scale / Theme Robust)
+======================================================
+
+Pixel template matching — ``match_template``, ``match_masked`` — correlates the
+template's pixels against the screen, so it breaks the moment the target is
+*rotated*, scaled by a factor you did not list, or re-coloured (a light-vs-dark
+theme, a hover state, a different skin). ``feature_match`` instead matches
+*keypoints*: distinctive corners described by orientation-invariant binary
+descriptors (ORB), then fits a RANSAC homography through the consistent ones. It
+locates the element under rotation, scale and appearance change, and reports the
+four projected corners plus the inlier count as a built-in confidence signal.
+
+It runs on an injectable ``haystack`` image (ndarray / path / PIL), so it is
+unit-testable on synthetic arrays without a real screen. ORB, the brute-force
+matcher and ``findHomography`` are all in core OpenCV (no contrib modules);
+OpenCV + NumPy come in via ``je_open_cv``. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import feature_match
+
+    hit = feature_match("logo.png", min_inliers=12)
+    if hit:
+        click(*hit.center)            # centre of the located quad
+        print(hit.corners)            # 4 [x, y] points, in template order
+        print(hit.inliers, hit.score) # geometric inliers and inlier fraction
+
+``feature_match`` returns a ``FeatureMatch`` (``corners``, ``center``,
+``inliers``, ``matches``, ``score``) or ``None`` when fewer than ``min_inliers``
+geometrically consistent matches survive. ``ratio`` is Lowe's ratio-test cutoff
+(lower = stricter); ``max_features`` caps the ORB keypoint budget. The ORB border
+and patch sizes are scaled down automatically for icon-sized templates, which
+OpenCV's defaults would otherwise reject outright.
+
+Executor command
+----------------
+
+``AC_feature_match`` takes ``template`` plus ``region`` / ``max_features`` /
+``ratio`` / ``min_inliers`` and returns ``{found, match}``. It is exposed as the
+MCP tool ``ac_feature_match`` and as a Script Builder command under **Image**.

docs/source/Eng/eng_index.rst

@@ -153,6 +153,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v128_features_doc
    doc/new_features/v129_features_doc
    doc/new_features/v130_features_doc
+   doc/new_features/v131_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v131_features_doc.rst

@@ -0,0 +1,35 @@
+ORB 特徵比對(對旋轉 / 縮放 / 主題穩健)
+========================================
+
+像素模板比對——``match_template``、``match_masked``——是把模板像素與螢幕做相關運算,因此目標一旦*旋轉*、
+以未列出的倍率縮放、或重新上色(亮 / 暗主題、hover 狀態、不同外觀),就會失效。``feature_match`` 改為比對
+*關鍵點*:以方向不變的二進位描述子(ORB)描述的特徵角點,再用 RANSAC 對一致的點擬合單應矩陣。它能在
+旋轉、縮放與外觀變化下定位元素,並回傳四個投影角點以及做為內建信心訊號的內點數。
+
+它在可注入的 ``haystack`` 影像(ndarray / 路徑 / PIL)上執行,因此可在無真實螢幕下對合成陣列做單元測試。
+ORB、暴力比對器與 ``findHomography`` 皆屬於 OpenCV 核心(不需 contrib 模組);OpenCV + NumPy 透過
+``je_open_cv`` 引入。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import feature_match
+
+    hit = feature_match("logo.png", min_inliers=12)
+    if hit:
+        click(*hit.center)            # 定位四邊形的中心
+        print(hit.corners)            # 4 個 [x, y] 點,依模板順序
+        print(hit.inliers, hit.score) # 幾何內點數與內點比例
+
+``feature_match`` 回傳 ``FeatureMatch``(``corners``、``center``、``inliers``、``matches``、``score``),
+或在幾何一致的比對少於 ``min_inliers`` 時回傳 ``None``。``ratio`` 是 Lowe 比例測試的門檻(越低越嚴格);
+``max_features`` 限制 ORB 關鍵點預算。ORB 的邊界與 patch 尺寸會針對圖示大小的模板自動縮小,否則 OpenCV
+的預設值會將其全數捨棄。
+
+執行器命令
+----------
+
+``AC_feature_match`` 接受 ``template`` 以及 ``region`` / ``max_features`` / ``ratio`` / ``min_inliers``,
+回傳 ``{found, match}``。它以 MCP 工具 ``ac_feature_match`` 以及 Script Builder 中 **Image** 分類下的命令提供。

docs/source/Zh/zh_index.rst

@@ -153,6 +153,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v128_features_doc
    doc/new_features/v129_features_doc
    doc/new_features/v130_features_doc
+   doc/new_features/v131_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -287,6 +287,9 @@
 from je_auto_control.utils.ssim import (
     ssim_changed_regions, ssim_compare,
 )
+# ORB feature matching (rotation / scale / theme-robust template location)
+from je_auto_control.utils.feature_match import feature_match
+from je_auto_control.utils.feature_match import FeatureMatch
 # CI workflow annotations (GitHub Actions)
 from je_auto_control.utils.ci_annotations import (
     emit_annotations, format_annotation,
@@ -1086,6 +1089,8 @@ def start_autocontrol_gui(*args, **kwargs):
     "find_color_regions",
     "ssim_compare",
     "ssim_changed_regions",
+    "feature_match",
+    "FeatureMatch",
     "emit_annotations", "format_annotation",
     "ClipboardHistory", "default_clipboard_history",
     "analyze_heal_log", "heal_stats", "scan_secrets",

je_auto_control/gui/script_builder/command_schema.py

@@ -262,7 +262,7 @@
             FieldSpec("scales", FieldType.STRING, optional=True,
                       placeholder="[0.9, 1.0, 1.1]"),
             FieldSpec("region", FieldType.STRING, optional=True,
                       placeholder="[left, top, right, bottom]"),
         ),
         description="Locate a template and return its confidence score + scale.",
     ))
@@ -341,6 +341,19 @@
         ),
         description="Boxes of the regions that structurally changed.",
     ))
+    specs.append(CommandSpec(
+        "AC_feature_match", "Image", "Feature Match (ORB)",
+        fields=(
+            FieldSpec("template", FieldType.FILE_PATH),
+            FieldSpec("region", FieldType.STRING, optional=True,
+                      placeholder="[left, top, right, bottom]"),
+            FieldSpec("max_features", FieldType.INT, optional=True, default=500),
+            FieldSpec("ratio", FieldType.FLOAT, optional=True, default=0.75,
+                      min_value=0.0, max_value=1.0),
+            FieldSpec("min_inliers", FieldType.INT, optional=True, default=10),
+        ),
+        description="Locate a template under rotation / scale / theme change.",
+    ))
 
 
 def _add_ocr_specs(specs: List[CommandSpec]) -> None:

je_auto_control/utils/executor/action_executor.py

@@ -3302,6 +3302,19 @@ def _ssim_changed_regions(reference: str, current: Any = None, ignore: Any = Non
     return {"count": len(regions), "regions": regions}
 
 
+def _feature_match(template: str, region: Any = None, max_features: Any = 500,
+                   ratio: Any = 0.75, min_inliers: Any = 10) -> Dict[str, Any]:
+    """Adapter: locate a template by ORB keypoints (rotation/scale/theme robust)."""
+    import json
+    from je_auto_control.utils.feature_match import feature_match
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    match = feature_match(template, region=region, max_features=int(max_features),
+                          ratio=float(ratio), min_inliers=int(min_inliers))
+    return {"found": match is not None,
+            "match": match.to_dict() if match else None}
+
+
 def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]:
     """Adapter: run nested actions while modifier keys are held down."""
     import json
@@ -5025,6 +5038,7 @@ def __init__(self):
             "AC_match_masked_all": _match_masked_all,
             "AC_ssim_compare": _ssim_compare,
             "AC_ssim_changed_regions": _ssim_changed_regions,
+            "AC_feature_match": _feature_match,
             "AC_find_color_region": _find_color_region,
             "AC_detect_drift": _detect_drift,
             "AC_categorical_drift": _categorical_drift,

je_auto_control/utils/feature_match/__init__.py

@@ -0,0 +1,6 @@
+"""ORB feature matching: locate templates under rotation / scale / theme change."""
+from je_auto_control.utils.feature_match.feature_match import (
+    FeatureMatch, feature_match,
+)
+
+__all__ = ["FeatureMatch", "feature_match"]

je_auto_control/utils/feature_match/feature_match.py

@@ -0,0 +1,123 @@
+"""ORB feature matching: locate a template that is rotated, rescaled or re-themed.
+
+Template matching (``match_template`` / ``match_masked``) correlates pixels, so it
+fails the moment the target is rotated, scaled by a non-listed factor, or its
+colours change (light vs dark theme, hover state). ORB matches *keypoints* —
+distinctive corners described by orientation-invariant binary descriptors — and
+fits a homography through them, so it finds the element under rotation, scale and
+appearance change, and reports the four projected corners plus the inlier count
+as a confidence signal.
+
+It runs on an injectable ``haystack`` image (ndarray / path / PIL), so it is
+unit-testable on synthetic arrays without a real screen. OpenCV + NumPy come in
+via the project's ``je_open_cv`` dependency and are imported lazily. Imports no
+``PySide6``. ORB, the brute-force matcher and ``findHomography`` are all in core
+OpenCV (no contrib modules).
+"""
+from dataclasses import asdict, dataclass
+from typing import Any, Dict, List, Optional, Sequence
+
+from je_auto_control.utils.visual_match.visual_match import (
+    _haystack_gray, _to_gray,
+)
+
+ImageSource = Any
+
+
+@dataclass(frozen=True)
+class FeatureMatch:
+    """A homography-located match: projected corners, centre, inliers, score."""
+
+    corners: List[List[int]]
+    center: List[int]
+    inliers: int
+    matches: int
+    score: float
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the match as a plain dict."""
+        return asdict(self)
+
+
+def _ratio_filter(knn_matches, ratio: float) -> list:
+    """Lowe's ratio test: keep matches clearly better than their runner-up."""
+    good = []
+    for pair in knn_matches:
+        if len(pair) == 2 and pair[0].distance < ratio * pair[1].distance:
+            good.append(pair[0])
+    return good
+
+
+def _make_orb(template_gray, max_features: int):
+    """Build an ORB detector with border/patch sizes that suit small templates.
+
+    OpenCV's default ``edgeThreshold``/``patchSize`` (31) reject every keypoint
+    in an icon-sized template, so they are scaled down to the template.
+    """
+    import cv2
+    smaller = min(template_gray.shape[:2])
+    patch = max(7, min(31, smaller // 3))
+    edge = max(2, patch // 3)
+    return cv2.ORB_create(nfeatures=int(max_features), edgeThreshold=edge,
+                          patchSize=patch)
+
+
+def _keypoint_matches(template_gray, scene_gray, max_features: int, ratio: float):
+    """Return (kp_template, kp_scene, good_matches) or None if too few features."""
+    import cv2
+    orb = _make_orb(template_gray, max_features)
+    kp1, des1 = orb.detectAndCompute(template_gray, None)
+    kp2, des2 = orb.detectAndCompute(scene_gray, None)
+    if des1 is None or des2 is None or len(des1) < 2 or len(des2) < 2:
+        return None
+    matcher = cv2.BFMatcher(cv2.NORM_HAMMING)
+    good = _ratio_filter(matcher.knnMatch(des1, des2, k=2), float(ratio))
+    return kp1, kp2, good
+
+
+def _locate(template_shape, kp1, kp2, good, min_inliers: int
+            ) -> Optional[FeatureMatch]:
+    """Fit a RANSAC homography from ``good`` matches and project the template box."""
+    import cv2
+    import numpy as np
+    src = np.float32([kp1[m.queryIdx].pt for m in good]).reshape(-1, 1, 2)
+    dst = np.float32([kp2[m.trainIdx].pt for m in good]).reshape(-1, 1, 2)
+    homography, mask = cv2.findHomography(src, dst, cv2.RANSAC, 5.0)
+    if homography is None:
+        return None
+    inliers = int(mask.sum())
+    if inliers < int(min_inliers):
+        return None
+    height, width = template_shape[:2]
+    box = np.float32([[0, 0], [width, 0], [width, height],
+                      [0, height]]).reshape(-1, 1, 2)
+    projected = cv2.perspectiveTransform(box, homography).reshape(-1, 2)
+    corners = [[int(round(px)), int(round(py))] for px, py in projected]
+    center = [int(round(float(projected[:, 0].mean()))),
+              int(round(float(projected[:, 1].mean())))]
+    return FeatureMatch(corners, center, inliers, len(good),
+                        round(inliers / len(good), 4))
+
+
+def feature_match(template: ImageSource, *,
+                  haystack: Optional[ImageSource] = None,
+                  region: Optional[Sequence[int]] = None,
+                  max_features: int = 500, ratio: float = 0.75,
+                  min_inliers: int = 10) -> Optional[FeatureMatch]:
+    """Locate ``template`` in the scene by ORB keypoints + a RANSAC homography.
+
+    Robust to rotation, scale and appearance change. ``haystack`` is an ndarray /
+    path / PIL image (default: grab the screen / ``region``). ``ratio`` is Lowe's
+    ratio-test cutoff; at least ``min_inliers`` geometrically consistent matches
+    are required. Returns a :class:`FeatureMatch` (projected ``corners``,
+    ``center``, ``inliers``, ``score``) or ``None`` when the target is not found.
+    """
+    template_gray = _to_gray(template)
+    scene_gray = _haystack_gray(haystack, region)
+    found = _keypoint_matches(template_gray, scene_gray, max_features, ratio)
+    if found is None:
+        return None
+    kp1, kp2, good = found
+    if len(good) < int(min_inliers):
+        return None
+    return _locate(template_gray.shape, kp1, kp2, good, min_inliers)

je_auto_control/utils/mcp_server/tools/_factories.py

@@ -2648,6 +2648,29 @@ def color_region_tools() -> List[MCPTool]:
     ]
 
 
+def feature_match_tools() -> List[MCPTool]:
+    return [
+        MCPTool(
+            name="ac_feature_match",
+            description=("Locate 'template' (image path) on screen by ORB "
+                         "keypoints + a RANSAC homography — robust to ROTATION, "
+                         "SCALE and theme/colour change, where pixel template "
+                         "matching fails. Returns {found, match:{corners (4 "
+                         "points), center, inliers, matches, score}}. 'min_inliers' "
+                         "is the confidence floor; 'ratio' the match cutoff."),
+            input_schema=schema({
+                "template": {"type": "string"},
+                "region": {"type": "array", "items": {"type": "integer"}},
+                "max_features": {"type": "integer"},
+                "ratio": {"type": "number"},
+                "min_inliers": {"type": "integer"}},
+                required=["template"]),
+            handler=h.feature_match,
+            annotations=READ_ONLY,
+        ),
+    ]
+
+
 def ssim_tools() -> List[MCPTool]:
     return [
         MCPTool(
@@ -6149,7 +6172,8 @@ def media_assert_tools() -> List[MCPTool]:
     process_doc_tools, tween_drag_tools, mouse_path_tools, field_entry_tools,
     key_hold_tools, mouse_relative_tools, text_unicode_tools,
     modifier_state_tools, grid_locator_tools, visual_match_tools,
-    color_region_tools, ssim_tools, plugin_sdk_tools, governance_tools,
+    color_region_tools, ssim_tools, feature_match_tools, plugin_sdk_tools,
+    governance_tools,
     credential_lease_tools, egress_tools, approval_testing_tools,
     trajectory_eval_tools, compliance_tools, agent_trace_tools,
     video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools,

je_auto_control/utils/mcp_server/tools/_handlers.py

@@ -2113,6 +2113,12 @@ def ssim_changed_regions(reference, current=None, ignore=None, threshold=0.35,
                                  region)
 
 
+def feature_match(template, region=None, max_features=500, ratio=0.75,
+                  min_inliers=10):
+    from je_auto_control.utils.executor.action_executor import _feature_match
+    return _feature_match(template, region, max_features, ratio, min_inliers)
+
+
 def detect_drift(reference, current, threshold=0.25, bins=10):
     from je_auto_control.utils.executor.action_executor import _detect_drift
     return _detect_drift(reference, current, threshold, bins)