Add SSIM structural comparison (perceptual score + changed regions)

README/WHATS_NEW_zh-CN.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 结构相似度(SSIM)比较
+
+会告诉你*哪里*变了的感知式画面比较。完整参考:[`docs/source/Zh/doc/new_features/v130_features_doc.rst`](../docs/source/Zh/doc/new_features/v130_features_doc.rst)。
+
+- **`ssim_compare` / `ssim_changed_regions`**(`AC_ssim_compare`、`AC_ssim_changed_regions`):像素差(`diff_screenshots`)会因一像素位移而误报;直方图(`detect_drift`)对版面无感。SSIM 是标准视觉回归度量——容忍轻微光照变化、对结构变化敏感。`ssim_compare` 返回 0..1 分数(1.0 = 完全相同);`ssim_changed_regions` 返回哪里移动了的方框。`ignore=[[x,y,w,h]]` 可遮罩即时时钟 / 光标。纯 NumPy + OpenCV(不需 scikit-image);可注入影像配对 → 无头可测。
+
 ## 本次更新 (2026-06-23) — 遮罩模板匹配
 
 不论背景如何都能匹配图标。完整参考:[`docs/source/Zh/doc/new_features/v129_features_doc.rst`](../docs/source/Zh/doc/new_features/v129_features_doc.rst)。

README/WHATS_NEW_zh-TW.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 結構相似度(SSIM)比較
+
+會告訴你*哪裡*變了的感知式畫面比較。完整參考:[`docs/source/Zh/doc/new_features/v130_features_doc.rst`](../docs/source/Zh/doc/new_features/v130_features_doc.rst)。
+
+- **`ssim_compare` / `ssim_changed_regions`**(`AC_ssim_compare`、`AC_ssim_changed_regions`):像素差(`diff_screenshots`)會因一像素位移而誤報;直方圖(`detect_drift`)對版面無感。SSIM 是標準視覺回歸度量——容忍輕微光照變化、對結構變化敏感。`ssim_compare` 回傳 0..1 分數(1.0 = 完全相同);`ssim_changed_regions` 回傳哪裡移動了的方框。`ignore=[[x,y,w,h]]` 可遮罩即時時鐘 / 游標。純 NumPy + OpenCV(不需 scikit-image);可注入影像配對 → 無頭可測。
+
 ## 本次更新 (2026-06-23) — 遮罩模板比對
 
 不論背景如何都能比對圖示。完整參考:[`docs/source/Zh/doc/new_features/v129_features_doc.rst`](../docs/source/Zh/doc/new_features/v129_features_doc.rst)。

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-23) — Structural-Similarity (SSIM) Comparison
+
+Perceptual screen comparison that tells you *what* changed. Full reference: [`docs/source/Eng/doc/new_features/v130_features_doc.rst`](docs/source/Eng/doc/new_features/v130_features_doc.rst).
+
+- **`ssim_compare` / `ssim_changed_regions`** (`AC_ssim_compare`, `AC_ssim_changed_regions`): pixel diff (`diff_screenshots`) fires on a one-pixel shift; a histogram (`detect_drift`) is blind to layout. SSIM is the standard visual-regression metric — tolerant of small illumination changes, sensitive to structural change. `ssim_compare` returns a 0..1 score (1.0 = identical); `ssim_changed_regions` returns boxes of what moved. `ignore=[[x,y,w,h]]` masks live clocks / cursors. Pure NumPy + OpenCV (no scikit-image); injectable image pair → headless-testable.
+
 ## What's new (2026-06-23) — Masked Template Matching
 
 Match icons regardless of their background. Full reference: [`docs/source/Eng/doc/new_features/v129_features_doc.rst`](docs/source/Eng/doc/new_features/v129_features_doc.rst).

docs/source/Eng/doc/new_features/v130_features_doc.rst

@@ -0,0 +1,46 @@
+Structural-Similarity (SSIM) Comparison
+=======================================
+
+The framework already compares screens by raw pixel diff (``diff_screenshots``)
+and by colour histogram (``detect_drift``) — but neither is *structural*. A pixel
+diff fires on a one-pixel shift or a brightness change that a human would ignore;
+a histogram is blind to layout (swap two halves of the screen and it is
+unchanged). SSIM is the standard visual-regression metric: tolerant of small
+illumination changes, sensitive to structural change (edited text, moved or
+missing elements). ``ssim_compare`` returns a single 0..1 score, and
+``ssim_changed_regions`` returns the boxes of *what* actually changed.
+
+It is a pure NumPy + OpenCV implementation (no scikit-image dependency) over an
+injectable image pair, so it is unit-testable on synthetic arrays without a real
+screen. OpenCV + NumPy come in via ``je_open_cv``. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import ssim_compare, ssim_changed_regions
+
+    # Gate a visual regression test against a golden screenshot.
+    score = ssim_compare("golden.png")           # current = live screen
+    assert score > 0.98
+
+    # Ignore a live clock / blinking cursor, then show what moved.
+    for box in ssim_changed_regions("golden.png", ignore=[[0, 0, 120, 30]]):
+        print(box["x"], box["y"], box["width"], box["height"])
+
+``ssim_compare`` returns the mean SSIM over the image (``1.0`` = identical);
+``current`` defaults to a screen grab of the optional ``region``. ``ignore`` is a
+list of ``[x, y, w, h]`` boxes excluded from the score and from change detection.
+``ssim_changed_regions`` flags pixels where local dissimilarity ``1 - SSIM``
+exceeds ``threshold``, groups the connected ones (``min_area`` and up) and returns
+``{x, y, width, height, area, center}`` largest first. Comparing two
+different-sized images raises ``ValueError``.
+
+Executor commands
+-----------------
+
+``AC_ssim_compare`` (``reference`` / ``current`` / ``ignore`` / ``region`` →
+``{score}``) and ``AC_ssim_changed_regions`` (also ``threshold`` / ``min_area`` →
+``{count, regions}``). They are exposed as the MCP tools ``ac_ssim_compare`` /
+``ac_ssim_changed_regions`` and as Script Builder commands under **Image**.

docs/source/Eng/eng_index.rst

@@ -152,6 +152,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v127_features_doc
    doc/new_features/v128_features_doc
    doc/new_features/v129_features_doc
+   doc/new_features/v130_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v130_features_doc.rst

@@ -0,0 +1,38 @@
+結構相似度(SSIM)比較
+========================
+
+框架已能以原始像素差(``diff_screenshots``)與顏色直方圖(``detect_drift``)比較畫面——但兩者都不是*結構性*的。
+像素差會因為一像素位移或人眼會忽略的亮度變化而誤報;直方圖則對版面無感(把畫面左右兩半互換,它毫無變化)。
+SSIM 是標準的視覺回歸度量:容忍輕微光照變化,對結構變化(文字被編輯、元素移動或消失)敏感。``ssim_compare``
+回傳單一 0..1 分數,``ssim_changed_regions`` 則回傳*哪裡*真正變了的方框。
+
+它是純 NumPy + OpenCV 實作(不需 scikit-image),在可注入的影像配對上執行,因此可在無真實螢幕下對合成陣列做
+單元測試。OpenCV + NumPy 透過 ``je_open_cv`` 引入。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import ssim_compare, ssim_changed_regions
+
+    # 以黃金截圖把關視覺回歸測試。
+    score = ssim_compare("golden.png")           # current = 實際螢幕
+    assert score > 0.98
+
+    # 忽略即時時鐘 / 閃爍游標,再列出哪裡移動了。
+    for box in ssim_changed_regions("golden.png", ignore=[[0, 0, 120, 30]]):
+        print(box["x"], box["y"], box["width"], box["height"])
+
+``ssim_compare`` 回傳整張影像的平均 SSIM(``1.0`` = 完全相同);``current`` 預設為對選用 ``region`` 的螢幕擷取。
+``ignore`` 是一組從分數與變化偵測中排除的 ``[x, y, w, h]`` 方框。``ssim_changed_regions`` 標記局部不相似度
+``1 - SSIM`` 超過 ``threshold`` 的像素,將相連者(``min_area`` 以上)分群,回傳 ``{x, y, width, height, area,
+center}``,由大到小。比較兩張不同尺寸的影像會丟出 ``ValueError``。
+
+執行器命令
+----------
+
+``AC_ssim_compare``(``reference`` / ``current`` / ``ignore`` / ``region`` →
+``{score}``)與 ``AC_ssim_changed_regions``(另有 ``threshold`` / ``min_area`` →
+``{count, regions}``)。它們以 MCP 工具 ``ac_ssim_compare`` / ``ac_ssim_changed_regions`` 以及 Script Builder 中
+**Image** 分類下的命令提供。

docs/source/Zh/zh_index.rst

@@ -152,6 +152,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v127_features_doc
    doc/new_features/v128_features_doc
    doc/new_features/v129_features_doc
+   doc/new_features/v130_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -283,6 +283,10 @@
 from je_auto_control.utils.color_region import (
     find_color_region, find_color_regions,
 )
+# Structural-similarity comparison (perceptual score + changed regions)
+from je_auto_control.utils.ssim import (
+    ssim_changed_regions, ssim_compare,
+)
 # CI workflow annotations (GitHub Actions)
 from je_auto_control.utils.ci_annotations import (
     emit_annotations, format_annotation,
@@ -1080,6 +1084,8 @@ def start_autocontrol_gui(*args, **kwargs):
     "best_matches",
     "find_color_region",
     "find_color_regions",
+    "ssim_compare",
+    "ssim_changed_regions",
     "emit_annotations", "format_annotation",
     "ClipboardHistory", "default_clipboard_history",
     "analyze_heal_log", "heal_stats", "scan_secrets",

je_auto_control/gui/script_builder/command_schema.py

@@ -262,7 +262,7 @@
             FieldSpec("scales", FieldType.STRING, optional=True,
                       placeholder="[0.9, 1.0, 1.1]"),
             FieldSpec("region", FieldType.STRING, optional=True,
                       placeholder="[left, top, right, bottom]"),
         ),
         description="Locate a template and return its confidence score + scale.",
     ))
@@ -314,6 +314,33 @@
         ),
         description="Locate regions by colour (status light / banner / fill).",
     ))
+    specs.append(CommandSpec(
+        "AC_ssim_compare", "Image", "SSIM Compare",
+        fields=(
+            FieldSpec("reference", FieldType.FILE_PATH),
+            FieldSpec("current", FieldType.FILE_PATH, optional=True),
+            FieldSpec("ignore", FieldType.STRING, optional=True,
+                      placeholder="[[x, y, w, h], ...]"),
+            FieldSpec("region", FieldType.STRING, optional=True,
+                      placeholder="[left, top, right, bottom]"),
+        ),
+        description="Structural-similarity score (0..1) vs reference / screen.",
+    ))
+    specs.append(CommandSpec(
+        "AC_ssim_changed_regions", "Image", "SSIM Changed Regions",
+        fields=(
+            FieldSpec("reference", FieldType.FILE_PATH),
+            FieldSpec("current", FieldType.FILE_PATH, optional=True),
+            FieldSpec("ignore", FieldType.STRING, optional=True,
+                      placeholder="[[x, y, w, h], ...]"),
+            FieldSpec("threshold", FieldType.FLOAT, optional=True, default=0.35,
+                      min_value=0.0, max_value=1.0),
+            FieldSpec("min_area", FieldType.INT, optional=True, default=50),
+            FieldSpec("region", FieldType.STRING, optional=True,
+                      placeholder="[left, top, right, bottom]"),
+        ),
+        description="Boxes of the regions that structurally changed.",
+    ))
 
 
 def _add_ocr_specs(specs: List[CommandSpec]) -> None:

je_auto_control/utils/color_region/color_region.py

@@ -50,6 +50,7 @@ def find_color_regions(rgb: Sequence[int], *,
     """
     import cv2
     import numpy as np
+    from je_auto_control.utils.cv2_utils.blobs import connected_boxes
     image = _to_rgb(haystack) if haystack is not None else _grab_rgb(region)
     red, green, blue = (int(channel) for channel in rgb[:3])
     tol = int(tolerance)
@@ -58,17 +59,7 @@ def find_color_regions(rgb: Sequence[int], *,
     upper = np.array([min(255, red + tol), min(255, green + tol),
                       min(255, blue + tol)], dtype=np.uint8)
     mask = cv2.inRange(image, lower, upper)
-    count, _labels, stats, centroids = cv2.connectedComponentsWithStats(
-        mask, connectivity=8)
-    regions: List[Dict[str, Any]] = []
-    for index in range(1, count):                     # 0 is the background
-        x, y, width, height, area = (int(v) for v in stats[index])
-        if area >= int(min_area):
-            cx, cy = centroids[index]
-            regions.append({"x": x, "y": y, "width": width, "height": height,
-                            "area": area, "center": [int(cx), int(cy)]})
-    regions.sort(key=lambda item: item["area"], reverse=True)
-    return regions
+    return connected_boxes(mask, int(min_area))
 
 
 def find_color_region(rgb: Sequence[int], *,

je_auto_control/utils/cv2_utils/blobs.py

@@ -0,0 +1,29 @@
+"""Connected-component bounding boxes from a binary mask.
+
+Shared by the colour-region locator and the SSIM change detector: both threshold
+an image into a binary mask and then need the bounding boxes of the connected
+blobs. OpenCV + NumPy come in via ``je_open_cv`` and are imported lazily. Imports
+no ``PySide6``.
+"""
+from typing import Any, Dict, List
+
+
+def connected_boxes(mask, min_area: int = 1) -> List[Dict[str, Any]]:
+    """Return ``{x, y, width, height, area, center}`` per blob, largest first.
+
+    ``mask`` is a uint8 binary image (non-zero = foreground). Components whose
+    area is below ``min_area`` are dropped; ``center`` is the component centroid
+    (truncated to int). Uses 8-connectivity; the background label (0) is skipped.
+    """
+    import cv2
+    count, _labels, stats, centroids = cv2.connectedComponentsWithStats(
+        mask, connectivity=8)
+    boxes: List[Dict[str, Any]] = []
+    for index in range(1, count):                     # 0 is the background
+        x, y, width, height, area = (int(v) for v in stats[index])
+        if area >= int(min_area):
+            cx, cy = centroids[index]
+            boxes.append({"x": x, "y": y, "width": width, "height": height,
+                          "area": area, "center": [int(cx), int(cy)]})
+    boxes.sort(key=lambda item: item["area"], reverse=True)
+    return boxes

je_auto_control/utils/executor/action_executor.py

@@ -3273,6 +3273,35 @@ def _find_color_region(rgb: Any, tolerance: Any = 20, min_area: Any = 50,
             "best": regions[0] if regions else None}
 
 
+def _ssim_compare(reference: str, current: Any = None, ignore: Any = None,
+                  region: Any = None) -> Dict[str, Any]:
+    """Adapter: structural-similarity score between reference and current/screen."""
+    import json
+    from je_auto_control.utils.ssim import ssim_compare
+    if isinstance(ignore, str):
+        ignore = json.loads(ignore) if ignore.strip() else None
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    score = ssim_compare(reference, current, ignore=ignore, region=region)
+    return {"score": score}
+
+
+def _ssim_changed_regions(reference: str, current: Any = None, ignore: Any = None,
+                          threshold: Any = 0.35, min_area: Any = 50,
+                          region: Any = None) -> Dict[str, Any]:
+    """Adapter: boxes of the regions that structurally changed, largest first."""
+    import json
+    from je_auto_control.utils.ssim import ssim_changed_regions
+    if isinstance(ignore, str):
+        ignore = json.loads(ignore) if ignore.strip() else None
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    regions = ssim_changed_regions(reference, current, ignore=ignore,
+                                   threshold=float(threshold),
+                                   min_area=int(min_area), region=region)
+    return {"count": len(regions), "regions": regions}
+
+
 def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]:
     """Adapter: run nested actions while modifier keys are held down."""
     import json
@@ -4994,6 +5023,8 @@ def __init__(self):
             "AC_match_template_all": _match_template_all,
             "AC_match_masked": _match_masked,
             "AC_match_masked_all": _match_masked_all,
+            "AC_ssim_compare": _ssim_compare,
+            "AC_ssim_changed_regions": _ssim_changed_regions,
             "AC_find_color_region": _find_color_region,
             "AC_detect_drift": _detect_drift,
             "AC_categorical_drift": _categorical_drift,

je_auto_control/utils/mcp_server/tools/_factories.py

@@ -2648,6 +2648,49 @@ def color_region_tools() -> List[MCPTool]:
     ]
 
 
+def ssim_tools() -> List[MCPTool]:
+    return [
+        MCPTool(
+            name="ac_ssim_compare",
+            description=("Structural-similarity (SSIM) score 0..1 between "
+                         "'reference' (image path) and 'current' (path; default: "
+                         "screen grab of 'region'). 1.0 = identical. 'ignore' is "
+                         "a list of [x,y,w,h] boxes to exclude (clocks/cursors). "
+                         "Returns {score}. Perceptual, unlike pixel diff."),
+            input_schema=schema({
+                "reference": {"type": "string"},
+                "current": {"type": "string"},
+                "ignore": {"type": "array",
+                           "items": {"type": "array",
+                                     "items": {"type": "integer"}}},
+                "region": {"type": "array", "items": {"type": "integer"}}},
+                required=["reference"]),
+            handler=h.ssim_compare,
+            annotations=READ_ONLY,
+        ),
+        MCPTool(
+            name="ac_ssim_changed_regions",
+            description=("Boxes of the regions that STRUCTURALLY changed between "
+                         "'reference' and 'current' (default: screen). A pixel "
+                         "changed where 1-SSIM > 'threshold'; blobs >= 'min_area'. "
+                         "'ignore' [x,y,w,h] boxes suppressed. Returns "
+                         "{count, regions} (largest first)."),
+            input_schema=schema({
+                "reference": {"type": "string"},
+                "current": {"type": "string"},
+                "ignore": {"type": "array",
+                           "items": {"type": "array",
+                                     "items": {"type": "integer"}}},
+                "threshold": {"type": "number"},
+                "min_area": {"type": "integer"},
+                "region": {"type": "array", "items": {"type": "integer"}}},
+                required=["reference"]),
+            handler=h.ssim_changed_regions,
+            annotations=READ_ONLY,
+        ),
+    ]
+
+
 def visual_match_tools() -> List[MCPTool]:
     return [
         MCPTool(
@@ -6106,7 +6149,7 @@ def media_assert_tools() -> List[MCPTool]:
     process_doc_tools, tween_drag_tools, mouse_path_tools, field_entry_tools,
     key_hold_tools, mouse_relative_tools, text_unicode_tools,
     modifier_state_tools, grid_locator_tools, visual_match_tools,
-    color_region_tools, plugin_sdk_tools, governance_tools,
+    color_region_tools, ssim_tools, plugin_sdk_tools, governance_tools,
     credential_lease_tools, egress_tools, approval_testing_tools,
     trajectory_eval_tools, compliance_tools, agent_trace_tools,
     video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools,

je_auto_control/utils/mcp_server/tools/_handlers.py

@@ -2100,6 +2100,19 @@ def find_color_region(rgb, tolerance=20, min_area=50, region=None):
     return _find_color_region(rgb, tolerance, min_area, region)
 
 
+def ssim_compare(reference, current=None, ignore=None, region=None):
+    from je_auto_control.utils.executor.action_executor import _ssim_compare
+    return _ssim_compare(reference, current, ignore, region)
+
+
+def ssim_changed_regions(reference, current=None, ignore=None, threshold=0.35,
+                         min_area=50, region=None):
+    from je_auto_control.utils.executor.action_executor import (
+        _ssim_changed_regions)
+    return _ssim_changed_regions(reference, current, ignore, threshold, min_area,
+                                 region)
+
+
 def detect_drift(reference, current, threshold=0.25, bins=10):
     from je_auto_control.utils.executor.action_executor import _detect_drift
     return _detect_drift(reference, current, threshold, bins)

je_auto_control/utils/ssim/__init__.py

@@ -0,0 +1,4 @@
+"""Structural-similarity (SSIM) comparison: perceptual score + changed regions."""
+from je_auto_control.utils.ssim.ssim import ssim_changed_regions, ssim_compare
+
+__all__ = ["ssim_changed_regions", "ssim_compare"]