diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md index 0ce36fb8..160dfe5a 100644 --- a/README/WHATS_NEW_zh-CN.md +++ b/README/WHATS_NEW_zh-CN.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-23) — 结构相似度(SSIM)比较 + +会告诉你*哪里*变了的感知式画面比较。完整参考:[`docs/source/Zh/doc/new_features/v130_features_doc.rst`](../docs/source/Zh/doc/new_features/v130_features_doc.rst)。 + +- **`ssim_compare` / `ssim_changed_regions`**(`AC_ssim_compare`、`AC_ssim_changed_regions`):像素差(`diff_screenshots`)会因一像素位移而误报;直方图(`detect_drift`)对版面无感。SSIM 是标准视觉回归度量——容忍轻微光照变化、对结构变化敏感。`ssim_compare` 返回 0..1 分数(1.0 = 完全相同);`ssim_changed_regions` 返回哪里移动了的方框。`ignore=[[x,y,w,h]]` 可遮罩即时时钟 / 光标。纯 NumPy + OpenCV(不需 scikit-image);可注入影像配对 → 无头可测。 + ## 本次更新 (2026-06-23) — 遮罩模板匹配 不论背景如何都能匹配图标。完整参考:[`docs/source/Zh/doc/new_features/v129_features_doc.rst`](../docs/source/Zh/doc/new_features/v129_features_doc.rst)。 diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md index 23033dfc..e71e8a85 100644 --- a/README/WHATS_NEW_zh-TW.md +++ b/README/WHATS_NEW_zh-TW.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-23) — 結構相似度(SSIM)比較 + +會告訴你*哪裡*變了的感知式畫面比較。完整參考:[`docs/source/Zh/doc/new_features/v130_features_doc.rst`](../docs/source/Zh/doc/new_features/v130_features_doc.rst)。 + +- **`ssim_compare` / `ssim_changed_regions`**(`AC_ssim_compare`、`AC_ssim_changed_regions`):像素差(`diff_screenshots`)會因一像素位移而誤報;直方圖(`detect_drift`)對版面無感。SSIM 是標準視覺回歸度量——容忍輕微光照變化、對結構變化敏感。`ssim_compare` 回傳 0..1 分數(1.0 = 完全相同);`ssim_changed_regions` 回傳哪裡移動了的方框。`ignore=[[x,y,w,h]]` 可遮罩即時時鐘 / 游標。純 NumPy + OpenCV(不需 scikit-image);可注入影像配對 → 無頭可測。 + ## 本次更新 (2026-06-23) — 遮罩模板比對 不論背景如何都能比對圖示。完整參考:[`docs/source/Zh/doc/new_features/v129_features_doc.rst`](../docs/source/Zh/doc/new_features/v129_features_doc.rst)。 diff --git a/WHATS_NEW.md b/WHATS_NEW.md index c505a65b..2b9becf3 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -1,5 +1,11 @@ # What's New — AutoControl +## What's new (2026-06-23) — Structural-Similarity (SSIM) Comparison + +Perceptual screen comparison that tells you *what* changed. Full reference: [`docs/source/Eng/doc/new_features/v130_features_doc.rst`](docs/source/Eng/doc/new_features/v130_features_doc.rst). + +- **`ssim_compare` / `ssim_changed_regions`** (`AC_ssim_compare`, `AC_ssim_changed_regions`): pixel diff (`diff_screenshots`) fires on a one-pixel shift; a histogram (`detect_drift`) is blind to layout. SSIM is the standard visual-regression metric — tolerant of small illumination changes, sensitive to structural change. `ssim_compare` returns a 0..1 score (1.0 = identical); `ssim_changed_regions` returns boxes of what moved. `ignore=[[x,y,w,h]]` masks live clocks / cursors. Pure NumPy + OpenCV (no scikit-image); injectable image pair → headless-testable. + ## What's new (2026-06-23) — Masked Template Matching Match icons regardless of their background. Full reference: [`docs/source/Eng/doc/new_features/v129_features_doc.rst`](docs/source/Eng/doc/new_features/v129_features_doc.rst). diff --git a/docs/source/Eng/doc/new_features/v130_features_doc.rst b/docs/source/Eng/doc/new_features/v130_features_doc.rst new file mode 100644 index 00000000..2e43812a --- /dev/null +++ b/docs/source/Eng/doc/new_features/v130_features_doc.rst @@ -0,0 +1,46 @@ +Structural-Similarity (SSIM) Comparison +======================================= + +The framework already compares screens by raw pixel diff (``diff_screenshots``) +and by colour histogram (``detect_drift``) — but neither is *structural*. A pixel +diff fires on a one-pixel shift or a brightness change that a human would ignore; +a histogram is blind to layout (swap two halves of the screen and it is +unchanged). SSIM is the standard visual-regression metric: tolerant of small +illumination changes, sensitive to structural change (edited text, moved or +missing elements). ``ssim_compare`` returns a single 0..1 score, and +``ssim_changed_regions`` returns the boxes of *what* actually changed. + +It is a pure NumPy + OpenCV implementation (no scikit-image dependency) over an +injectable image pair, so it is unit-testable on synthetic arrays without a real +screen. OpenCV + NumPy come in via ``je_open_cv``. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import ssim_compare, ssim_changed_regions + + # Gate a visual regression test against a golden screenshot. + score = ssim_compare("golden.png") # current = live screen + assert score > 0.98 + + # Ignore a live clock / blinking cursor, then show what moved. + for box in ssim_changed_regions("golden.png", ignore=[[0, 0, 120, 30]]): + print(box["x"], box["y"], box["width"], box["height"]) + +``ssim_compare`` returns the mean SSIM over the image (``1.0`` = identical); +``current`` defaults to a screen grab of the optional ``region``. ``ignore`` is a +list of ``[x, y, w, h]`` boxes excluded from the score and from change detection. +``ssim_changed_regions`` flags pixels where local dissimilarity ``1 - SSIM`` +exceeds ``threshold``, groups the connected ones (``min_area`` and up) and returns +``{x, y, width, height, area, center}`` largest first. Comparing two +different-sized images raises ``ValueError``. + +Executor commands +----------------- + +``AC_ssim_compare`` (``reference`` / ``current`` / ``ignore`` / ``region`` → +``{score}``) and ``AC_ssim_changed_regions`` (also ``threshold`` / ``min_area`` → +``{count, regions}``). They are exposed as the MCP tools ``ac_ssim_compare`` / +``ac_ssim_changed_regions`` and as Script Builder commands under **Image**. diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index 2186e071..30654e94 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -152,6 +152,7 @@ Comprehensive guides for all AutoControl features. doc/new_features/v127_features_doc doc/new_features/v128_features_doc doc/new_features/v129_features_doc + doc/new_features/v130_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/docs/source/Zh/doc/new_features/v130_features_doc.rst b/docs/source/Zh/doc/new_features/v130_features_doc.rst new file mode 100644 index 00000000..d511ca50 --- /dev/null +++ b/docs/source/Zh/doc/new_features/v130_features_doc.rst @@ -0,0 +1,38 @@ +結構相似度(SSIM)比較 +======================== + +框架已能以原始像素差(``diff_screenshots``)與顏色直方圖(``detect_drift``)比較畫面——但兩者都不是*結構性*的。 +像素差會因為一像素位移或人眼會忽略的亮度變化而誤報;直方圖則對版面無感(把畫面左右兩半互換,它毫無變化)。 +SSIM 是標準的視覺回歸度量:容忍輕微光照變化,對結構變化(文字被編輯、元素移動或消失)敏感。``ssim_compare`` +回傳單一 0..1 分數,``ssim_changed_regions`` 則回傳*哪裡*真正變了的方框。 + +它是純 NumPy + OpenCV 實作(不需 scikit-image),在可注入的影像配對上執行,因此可在無真實螢幕下對合成陣列做 +單元測試。OpenCV + NumPy 透過 ``je_open_cv`` 引入。不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import ssim_compare, ssim_changed_regions + + # 以黃金截圖把關視覺回歸測試。 + score = ssim_compare("golden.png") # current = 實際螢幕 + assert score > 0.98 + + # 忽略即時時鐘 / 閃爍游標,再列出哪裡移動了。 + for box in ssim_changed_regions("golden.png", ignore=[[0, 0, 120, 30]]): + print(box["x"], box["y"], box["width"], box["height"]) + +``ssim_compare`` 回傳整張影像的平均 SSIM(``1.0`` = 完全相同);``current`` 預設為對選用 ``region`` 的螢幕擷取。 +``ignore`` 是一組從分數與變化偵測中排除的 ``[x, y, w, h]`` 方框。``ssim_changed_regions`` 標記局部不相似度 +``1 - SSIM`` 超過 ``threshold`` 的像素,將相連者(``min_area`` 以上)分群,回傳 ``{x, y, width, height, area, +center}``,由大到小。比較兩張不同尺寸的影像會丟出 ``ValueError``。 + +執行器命令 +---------- + +``AC_ssim_compare``(``reference`` / ``current`` / ``ignore`` / ``region`` → +``{score}``)與 ``AC_ssim_changed_regions``(另有 ``threshold`` / ``min_area`` → +``{count, regions}``)。它們以 MCP 工具 ``ac_ssim_compare`` / ``ac_ssim_changed_regions`` 以及 Script Builder 中 +**Image** 分類下的命令提供。 diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst index a2c3c270..2c4a5b10 100644 --- a/docs/source/Zh/zh_index.rst +++ b/docs/source/Zh/zh_index.rst @@ -152,6 +152,7 @@ AutoControl 所有功能的完整使用指南。 doc/new_features/v127_features_doc doc/new_features/v128_features_doc doc/new_features/v129_features_doc + doc/new_features/v130_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index ce071d32..67e4c4fb 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -283,6 +283,10 @@ from je_auto_control.utils.color_region import ( find_color_region, find_color_regions, ) +# Structural-similarity comparison (perceptual score + changed regions) +from je_auto_control.utils.ssim import ( + ssim_changed_regions, ssim_compare, +) # CI workflow annotations (GitHub Actions) from je_auto_control.utils.ci_annotations import ( emit_annotations, format_annotation, @@ -1080,6 +1084,8 @@ def start_autocontrol_gui(*args, **kwargs): "best_matches", "find_color_region", "find_color_regions", + "ssim_compare", + "ssim_changed_regions", "emit_annotations", "format_annotation", "ClipboardHistory", "default_clipboard_history", "analyze_heal_log", "heal_stats", "scan_secrets", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index 69f498c9..be1f948a 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -314,6 +314,33 @@ def _add_image_specs(specs: List[CommandSpec]) -> None: ), description="Locate regions by colour (status light / banner / fill).", )) + specs.append(CommandSpec( + "AC_ssim_compare", "Image", "SSIM Compare", + fields=( + FieldSpec("reference", FieldType.FILE_PATH), + FieldSpec("current", FieldType.FILE_PATH, optional=True), + FieldSpec("ignore", FieldType.STRING, optional=True, + placeholder="[[x, y, w, h], ...]"), + FieldSpec("region", FieldType.STRING, optional=True, + placeholder="[left, top, right, bottom]"), + ), + description="Structural-similarity score (0..1) vs reference / screen.", + )) + specs.append(CommandSpec( + "AC_ssim_changed_regions", "Image", "SSIM Changed Regions", + fields=( + FieldSpec("reference", FieldType.FILE_PATH), + FieldSpec("current", FieldType.FILE_PATH, optional=True), + FieldSpec("ignore", FieldType.STRING, optional=True, + placeholder="[[x, y, w, h], ...]"), + FieldSpec("threshold", FieldType.FLOAT, optional=True, default=0.35, + min_value=0.0, max_value=1.0), + FieldSpec("min_area", FieldType.INT, optional=True, default=50), + FieldSpec("region", FieldType.STRING, optional=True, + placeholder="[left, top, right, bottom]"), + ), + description="Boxes of the regions that structurally changed.", + )) def _add_ocr_specs(specs: List[CommandSpec]) -> None: diff --git a/je_auto_control/utils/color_region/color_region.py b/je_auto_control/utils/color_region/color_region.py index bf96f325..7adb3c1c 100644 --- a/je_auto_control/utils/color_region/color_region.py +++ b/je_auto_control/utils/color_region/color_region.py @@ -50,6 +50,7 @@ def find_color_regions(rgb: Sequence[int], *, """ import cv2 import numpy as np + from je_auto_control.utils.cv2_utils.blobs import connected_boxes image = _to_rgb(haystack) if haystack is not None else _grab_rgb(region) red, green, blue = (int(channel) for channel in rgb[:3]) tol = int(tolerance) @@ -58,17 +59,7 @@ def find_color_regions(rgb: Sequence[int], *, upper = np.array([min(255, red + tol), min(255, green + tol), min(255, blue + tol)], dtype=np.uint8) mask = cv2.inRange(image, lower, upper) - count, _labels, stats, centroids = cv2.connectedComponentsWithStats( - mask, connectivity=8) - regions: List[Dict[str, Any]] = [] - for index in range(1, count): # 0 is the background - x, y, width, height, area = (int(v) for v in stats[index]) - if area >= int(min_area): - cx, cy = centroids[index] - regions.append({"x": x, "y": y, "width": width, "height": height, - "area": area, "center": [int(cx), int(cy)]}) - regions.sort(key=lambda item: item["area"], reverse=True) - return regions + return connected_boxes(mask, int(min_area)) def find_color_region(rgb: Sequence[int], *, diff --git a/je_auto_control/utils/cv2_utils/blobs.py b/je_auto_control/utils/cv2_utils/blobs.py new file mode 100644 index 00000000..a9f262e5 --- /dev/null +++ b/je_auto_control/utils/cv2_utils/blobs.py @@ -0,0 +1,29 @@ +"""Connected-component bounding boxes from a binary mask. + +Shared by the colour-region locator and the SSIM change detector: both threshold +an image into a binary mask and then need the bounding boxes of the connected +blobs. OpenCV + NumPy come in via ``je_open_cv`` and are imported lazily. Imports +no ``PySide6``. +""" +from typing import Any, Dict, List + + +def connected_boxes(mask, min_area: int = 1) -> List[Dict[str, Any]]: + """Return ``{x, y, width, height, area, center}`` per blob, largest first. + + ``mask`` is a uint8 binary image (non-zero = foreground). Components whose + area is below ``min_area`` are dropped; ``center`` is the component centroid + (truncated to int). Uses 8-connectivity; the background label (0) is skipped. + """ + import cv2 + count, _labels, stats, centroids = cv2.connectedComponentsWithStats( + mask, connectivity=8) + boxes: List[Dict[str, Any]] = [] + for index in range(1, count): # 0 is the background + x, y, width, height, area = (int(v) for v in stats[index]) + if area >= int(min_area): + cx, cy = centroids[index] + boxes.append({"x": x, "y": y, "width": width, "height": height, + "area": area, "center": [int(cx), int(cy)]}) + boxes.sort(key=lambda item: item["area"], reverse=True) + return boxes diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index 9900cda1..3518f155 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -3273,6 +3273,35 @@ def _find_color_region(rgb: Any, tolerance: Any = 20, min_area: Any = 50, "best": regions[0] if regions else None} +def _ssim_compare(reference: str, current: Any = None, ignore: Any = None, + region: Any = None) -> Dict[str, Any]: + """Adapter: structural-similarity score between reference and current/screen.""" + import json + from je_auto_control.utils.ssim import ssim_compare + if isinstance(ignore, str): + ignore = json.loads(ignore) if ignore.strip() else None + if isinstance(region, str): + region = json.loads(region) if region.strip() else None + score = ssim_compare(reference, current, ignore=ignore, region=region) + return {"score": score} + + +def _ssim_changed_regions(reference: str, current: Any = None, ignore: Any = None, + threshold: Any = 0.35, min_area: Any = 50, + region: Any = None) -> Dict[str, Any]: + """Adapter: boxes of the regions that structurally changed, largest first.""" + import json + from je_auto_control.utils.ssim import ssim_changed_regions + if isinstance(ignore, str): + ignore = json.loads(ignore) if ignore.strip() else None + if isinstance(region, str): + region = json.loads(region) if region.strip() else None + regions = ssim_changed_regions(reference, current, ignore=ignore, + threshold=float(threshold), + min_area=int(min_area), region=region) + return {"count": len(regions), "regions": regions} + + def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]: """Adapter: run nested actions while modifier keys are held down.""" import json @@ -4994,6 +5023,8 @@ def __init__(self): "AC_match_template_all": _match_template_all, "AC_match_masked": _match_masked, "AC_match_masked_all": _match_masked_all, + "AC_ssim_compare": _ssim_compare, + "AC_ssim_changed_regions": _ssim_changed_regions, "AC_find_color_region": _find_color_region, "AC_detect_drift": _detect_drift, "AC_categorical_drift": _categorical_drift, diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index 2996a9dc..019685ac 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -2648,6 +2648,49 @@ def color_region_tools() -> List[MCPTool]: ] +def ssim_tools() -> List[MCPTool]: + return [ + MCPTool( + name="ac_ssim_compare", + description=("Structural-similarity (SSIM) score 0..1 between " + "'reference' (image path) and 'current' (path; default: " + "screen grab of 'region'). 1.0 = identical. 'ignore' is " + "a list of [x,y,w,h] boxes to exclude (clocks/cursors). " + "Returns {score}. Perceptual, unlike pixel diff."), + input_schema=schema({ + "reference": {"type": "string"}, + "current": {"type": "string"}, + "ignore": {"type": "array", + "items": {"type": "array", + "items": {"type": "integer"}}}, + "region": {"type": "array", "items": {"type": "integer"}}}, + required=["reference"]), + handler=h.ssim_compare, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_ssim_changed_regions", + description=("Boxes of the regions that STRUCTURALLY changed between " + "'reference' and 'current' (default: screen). A pixel " + "changed where 1-SSIM > 'threshold'; blobs >= 'min_area'. " + "'ignore' [x,y,w,h] boxes suppressed. Returns " + "{count, regions} (largest first)."), + input_schema=schema({ + "reference": {"type": "string"}, + "current": {"type": "string"}, + "ignore": {"type": "array", + "items": {"type": "array", + "items": {"type": "integer"}}}, + "threshold": {"type": "number"}, + "min_area": {"type": "integer"}, + "region": {"type": "array", "items": {"type": "integer"}}}, + required=["reference"]), + handler=h.ssim_changed_regions, + annotations=READ_ONLY, + ), + ] + + def visual_match_tools() -> List[MCPTool]: return [ MCPTool( @@ -6106,7 +6149,7 @@ def media_assert_tools() -> List[MCPTool]: process_doc_tools, tween_drag_tools, mouse_path_tools, field_entry_tools, key_hold_tools, mouse_relative_tools, text_unicode_tools, modifier_state_tools, grid_locator_tools, visual_match_tools, - color_region_tools, plugin_sdk_tools, governance_tools, + color_region_tools, ssim_tools, plugin_sdk_tools, governance_tools, credential_lease_tools, egress_tools, approval_testing_tools, trajectory_eval_tools, compliance_tools, agent_trace_tools, video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools, diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py index 6f9ee660..8932e4f8 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers.py @@ -2100,6 +2100,19 @@ def find_color_region(rgb, tolerance=20, min_area=50, region=None): return _find_color_region(rgb, tolerance, min_area, region) +def ssim_compare(reference, current=None, ignore=None, region=None): + from je_auto_control.utils.executor.action_executor import _ssim_compare + return _ssim_compare(reference, current, ignore, region) + + +def ssim_changed_regions(reference, current=None, ignore=None, threshold=0.35, + min_area=50, region=None): + from je_auto_control.utils.executor.action_executor import ( + _ssim_changed_regions) + return _ssim_changed_regions(reference, current, ignore, threshold, min_area, + region) + + def detect_drift(reference, current, threshold=0.25, bins=10): from je_auto_control.utils.executor.action_executor import _detect_drift return _detect_drift(reference, current, threshold, bins) diff --git a/je_auto_control/utils/ssim/__init__.py b/je_auto_control/utils/ssim/__init__.py new file mode 100644 index 00000000..4297fdf0 --- /dev/null +++ b/je_auto_control/utils/ssim/__init__.py @@ -0,0 +1,4 @@ +"""Structural-similarity (SSIM) comparison: perceptual score + changed regions.""" +from je_auto_control.utils.ssim.ssim import ssim_changed_regions, ssim_compare + +__all__ = ["ssim_changed_regions", "ssim_compare"] diff --git a/je_auto_control/utils/ssim/ssim.py b/je_auto_control/utils/ssim/ssim.py new file mode 100644 index 00000000..c882a941 --- /dev/null +++ b/je_auto_control/utils/ssim/ssim.py @@ -0,0 +1,120 @@ +"""Structural-similarity (SSIM) comparison: perceptual score + changed regions. + +The framework already has pixel diff (``diff_screenshots``) and histogram drift +(``detect_drift``); neither is *structural*. SSIM is the standard visual-regression +metric — tolerant of small illumination shifts, sensitive to structural change +(text edits, moved or missing elements) — and yields a 0..1 similarity plus the +boxes of the regions that actually changed, so a test can both gate on a score and +point at *what* moved. + +It is a pure NumPy + OpenCV implementation (no scikit-image, which is not a +dependency) over an injectable image pair, so it is unit-testable on synthetic +arrays without a real screen. OpenCV + NumPy come in via ``je_open_cv`` and are +imported lazily. Imports no ``PySide6``. +""" +from typing import Any, Dict, List, Optional, Sequence + +ImageSource = Any +IgnoreBoxes = Optional[Sequence[Sequence[int]]] +_WINDOW = (11, 11) +_SIGMA = 1.5 +_C1 = (0.01 * 255) ** 2 +_C2 = (0.03 * 255) ** 2 + + +def _to_gray_f(source: ImageSource): + """Load a path / ndarray / PIL image as a 2-D float64 grayscale image.""" + import cv2 + import numpy as np + if hasattr(source, "shape"): + array = np.asarray(source) + elif isinstance(source, (str, bytes)) or hasattr(source, "__fspath__"): + array = cv2.imread(str(source), cv2.IMREAD_COLOR) + if array is None: + raise ValueError(f"could not read image: {source!r}") + else: + array = np.asarray(source) + if array.ndim == 3: + code = cv2.COLOR_BGRA2GRAY if array.shape[2] == 4 else cv2.COLOR_BGR2GRAY + array = cv2.cvtColor(array, code) + return array.astype(np.float64) + + +def _grab_gray_f(region: Optional[Sequence[int]]): + from je_auto_control.utils.cv2_utils.screenshot import pil_screenshot + image = pil_screenshot(screen_region=list(region) if region else None) + return _to_gray_f(image) + + +def _resolve_pair(reference: ImageSource, current: Optional[ImageSource], + region: Optional[Sequence[int]]): + reference_gray = _to_gray_f(reference) + current_gray = (_to_gray_f(current) if current is not None + else _grab_gray_f(region)) + if reference_gray.shape != current_gray.shape: + raise ValueError(f"reference {reference_gray.shape} and current " + f"{current_gray.shape} must be the same size") + return reference_gray, current_gray + + +def _ssim_map(reference, current): + """Per-pixel SSIM map via an 11x11 Gaussian window (sigma 1.5).""" + import cv2 + mu_ref = cv2.GaussianBlur(reference, _WINDOW, _SIGMA) + mu_cur = cv2.GaussianBlur(current, _WINDOW, _SIGMA) + mu_ref2, mu_cur2, mu_cross = mu_ref * mu_ref, mu_cur * mu_cur, mu_ref * mu_cur + var_ref = cv2.GaussianBlur(reference * reference, _WINDOW, _SIGMA) - mu_ref2 + var_cur = cv2.GaussianBlur(current * current, _WINDOW, _SIGMA) - mu_cur2 + cov = cv2.GaussianBlur(reference * current, _WINDOW, _SIGMA) - mu_cross + numerator = (2 * mu_cross + _C1) * (2 * cov + _C2) + denominator = (mu_ref2 + mu_cur2 + _C1) * (var_ref + var_cur + _C2) + return numerator / denominator + + +def _keep_mask(shape, ignore: IgnoreBoxes): + """Boolean keep-mask (True = counted); ``ignore`` boxes [x,y,w,h] set False.""" + import numpy as np + keep = np.ones(shape, dtype=bool) + for box in ignore or (): + x, y, width, height = (int(value) for value in box[:4]) + keep[y:y + height, x:x + width] = False + return keep + + +def ssim_compare(reference: ImageSource, current: Optional[ImageSource] = None, + *, ignore: IgnoreBoxes = None, + region: Optional[Sequence[int]] = None) -> float: + """Return the mean SSIM (0..1) between ``reference`` and ``current``. + + ``current`` defaults to a screen grab of the optional ``region``. ``ignore`` + is a list of ``[x, y, w, h]`` boxes excluded from the score (dynamic clocks, + blinking cursors). ``1.0`` means structurally identical; lower means more + change. Raises ``ValueError`` if the two images differ in size. + """ + import numpy as np + reference_gray, current_gray = _resolve_pair(reference, current, region) + smap = _ssim_map(reference_gray, current_gray) + keep = _keep_mask(smap.shape, ignore) + return round(float(np.mean(smap[keep])), 4) if keep.any() else 1.0 + + +def ssim_changed_regions(reference: ImageSource, + current: Optional[ImageSource] = None, *, + ignore: IgnoreBoxes = None, threshold: float = 0.35, + min_area: int = 50, + region: Optional[Sequence[int]] = None + ) -> List[Dict[str, Any]]: + """Return boxes of the regions that structurally changed, largest first. + + A pixel counts as changed where local dissimilarity ``1 - SSIM`` exceeds + ``threshold``; connected changed pixels covering at least ``min_area`` are + returned as ``{x, y, width, height, area, center}``. ``ignore`` boxes are + suppressed before detection. + """ + import numpy as np + from je_auto_control.utils.cv2_utils.blobs import connected_boxes + reference_gray, current_gray = _resolve_pair(reference, current, region) + smap = _ssim_map(reference_gray, current_gray) + changed = (1.0 - smap) > float(threshold) + changed &= _keep_mask(smap.shape, ignore) + return connected_boxes(changed.astype(np.uint8), int(min_area)) diff --git a/test/unit_test/headless/test_ssim_compare_batch.py b/test/unit_test/headless/test_ssim_compare_batch.py new file mode 100644 index 00000000..6b81cca2 --- /dev/null +++ b/test/unit_test/headless/test_ssim_compare_batch.py @@ -0,0 +1,79 @@ +"""Headless tests for SSIM structural comparison. No Qt.""" +import pytest + +import je_auto_control as ac + +np = pytest.importorskip("numpy") +pytest.importorskip("cv2") + +from je_auto_control.utils.ssim import ( # noqa: E402 + ssim_changed_regions, ssim_compare, +) + + +def _base(): + """A textured 80x80 reference image (not flat — SSIM needs structure).""" + yy, xx = np.mgrid[0:80, 0:80] + return ((yy * 7 + xx * 11 + yy * xx) % 256).astype(np.uint8) + + +def _with_block(): + """The reference with a 20x20 block overwritten at (40, 30).""" + changed = _base() + changed[30:50, 40:60] = 0 + return changed + + +def test_identical_scores_one(): + base = _base() + assert ssim_compare(base, base.copy()) == pytest.approx(1.0) + + +def test_change_lowers_score(): + assert ssim_compare(_base(), _with_block()) < 0.99 + + +def test_ignore_region_restores_score(): + # masking out the changed block lifts the score back towards identical + without = ssim_compare(_base(), _with_block()) + with_ignore = ssim_compare(_base(), _with_block(), ignore=[[40, 30, 20, 20]]) + assert with_ignore > without + assert with_ignore > 0.98 + + +def test_changed_regions_locates_the_block(): + regions = ssim_changed_regions(_base(), _with_block(), min_area=20) + assert len(regions) == 1 + box = regions[0] + # the changed blob overlaps the (40,30)-(60,50) block + assert 30 <= box["x"] <= 50 and 20 <= box["y"] <= 40 + assert box["area"] >= 20 + + +def test_changed_regions_empty_when_identical(): + base = _base() + assert ssim_changed_regions(base, base.copy(), min_area=20) == [] + + +def test_size_mismatch_raises(): + small = np.zeros((40, 40), dtype=np.uint8) + with pytest.raises(ValueError): + ssim_compare(_base(), small) + + +# --- wiring --------------------------------------------------------------- + +def test_wiring(): + known = set(ac.executor.known_commands()) + assert {"AC_ssim_compare", "AC_ssim_changed_regions"} <= known + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert {"ac_ssim_compare", "ac_ssim_changed_regions"} <= names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert {"AC_ssim_compare", "AC_ssim_changed_regions"} <= specs + + +def test_facade_exports(): + for attr in ("ssim_compare", "ssim_changed_regions"): + assert hasattr(ac, attr) and attr in ac.__all__