Add image preprocessing pipeline for OCR / template matching

README/WHATS_NEW_zh-CN.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 图像预处理(供 OCR / 模板匹配)
+
+在识别或匹配前先清理画面。完整参考:[`docs/source/Zh/doc/new_features/v135_features_doc.rst`](../docs/source/Zh/doc/new_features/v135_features_doc.rst)。
+
+- **`preprocess_image` + `to_grayscale` / `binarize` / `upscale` / `denoise` / `deskew` / `enhance_contrast`**(`AC_preprocess_image`):`locate_text` 与 `match_template` 把*原始*截取直接喂给 OCR / 匹配器——小字、暗色主题、低对比与歪斜会严重影响两者,而框架毫无预处理接缝。本功能加入标准流程(灰阶 → 放大 → 二值化 → 去歪斜 → 去噪 → CLAHE),倍增其准确度。可注入 haystack → ndarray;`detect_skew_angle` 测量文字旋转;`binarize` 提供 otsu / adaptive。执行器命令把清理后图像写入路径。可对合成数组无头测试。
+
 ## 本次更新 (2026-06-23) — 排列多个窗口(网格 / 层叠)
 
 一次调用排好一整组窗口。完整参考:[`docs/source/Zh/doc/new_features/v134_features_doc.rst`](../docs/source/Zh/doc/new_features/v134_features_doc.rst)。

README/WHATS_NEW_zh-TW.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 影像前處理(供 OCR / 模板比對)
+
+在辨識或比對前先清理畫面。完整參考:[`docs/source/Zh/doc/new_features/v135_features_doc.rst`](../docs/source/Zh/doc/new_features/v135_features_doc.rst)。
+
+- **`preprocess_image` + `to_grayscale` / `binarize` / `upscale` / `denoise` / `deskew` / `enhance_contrast`**(`AC_preprocess_image`):`locate_text` 與 `match_template` 把*原始*擷取直接餵給 OCR / 比對器——小字、暗色主題、低對比與歪斜會嚴重影響兩者,而框架毫無前處理接縫。本功能加入標準流程(灰階 → 放大 → 二值化 → 去歪斜 → 去噪 → CLAHE),倍增其準確度。可注入 haystack → ndarray;`detect_skew_angle` 量測文字旋轉;`binarize` 提供 otsu / adaptive。執行器命令把清理後影像寫入路徑。可對合成陣列無頭測試。
+
 ## 本次更新 (2026-06-23) — 排列多個視窗(網格 / 層疊)
 
 一次呼叫排好一整組視窗。完整參考:[`docs/source/Zh/doc/new_features/v134_features_doc.rst`](../docs/source/Zh/doc/new_features/v134_features_doc.rst)。

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-23) — Image Pre-processing for OCR / Template Matching
+
+Clean up the screen before reading or matching it. Full reference: [`docs/source/Eng/doc/new_features/v135_features_doc.rst`](docs/source/Eng/doc/new_features/v135_features_doc.rst).
+
+- **`preprocess_image` + `to_grayscale` / `binarize` / `upscale` / `denoise` / `deskew` / `enhance_contrast`** (`AC_preprocess_image`): `locate_text` and `match_template` fed the *raw* capture to OCR / the matcher — small text, dark themes, low contrast and skew wrecked both, with no preprocessing seam anywhere. This adds the standard pipeline (grayscale → upscale → binarize → deskew → denoise → CLAHE) that multiplies their accuracy. Injectable haystack → ndarray; `detect_skew_angle` measures text rotation; `binarize` does otsu / adaptive. The executor command writes the cleaned image to a path. Headless-testable on synthetic arrays.
+
 ## What's new (2026-06-23) — Arrange Multiple Windows (Grid / Cascade)
 
 Lay out a whole set of windows in one call. Full reference: [`docs/source/Eng/doc/new_features/v134_features_doc.rst`](docs/source/Eng/doc/new_features/v134_features_doc.rst).

docs/source/Eng/doc/new_features/v135_features_doc.rst

@@ -0,0 +1,48 @@
+Image Pre-processing for OCR / Template Matching
+================================================
+
+``locate_text`` / ``ocr_read_structure`` and ``match_template`` feed the *raw*
+screen capture straight to the OCR engine or the matcher. Small UI text, dark
+themes, low contrast and a slightly rotated screenshot wreck both — and there was
+no preprocessing seam anywhere in the framework. This adds the standard pre-step
+pipeline — grayscale → upscale → binarize → deskew → denoise → CLAHE contrast —
+that multiplies the accuracy of the OCR and matching features you already use.
+
+Every function runs on an injectable ``haystack`` image (ndarray / path / PIL,
+default: grab the screen / ``region``) and returns a NumPy ndarray, so it is
+unit-testable on synthetic arrays. OpenCV + NumPy come in via ``je_open_cv``.
+Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import preprocess_image, binarize, deskew, upscale
+
+    # One-shot pipeline, then OCR the cleaned image.
+    clean = preprocess_image("receipt.png", steps=("grayscale", "upscale",
+                                                    "deskew", "binarize"), scale=2.0)
+
+    # Or compose the individual steps.
+    bw = binarize("panel.png", method="adaptive_gaussian", block_size=41)
+    straight = deskew("scan.png", max_angle=10.0)
+    big = upscale("tiny_label.png", scale=3.0, interp="lanczos")
+
+The building blocks are ``to_grayscale``, ``upscale`` (``scale`` / ``interp``),
+``binarize`` (``method`` = ``otsu`` / ``adaptive_mean`` / ``adaptive_gaussian``),
+``denoise``, ``enhance_contrast`` (CLAHE), ``deskew`` and ``detect_skew_angle``
+(returns the measured text-skew in degrees, clamped to ``±max_angle``).
+``preprocess_image`` chains any of the named ``steps`` — ``grayscale``,
+``upscale``, ``binarize``, ``denoise``, ``deskew``, ``contrast`` — in order;
+unknown step names raise ``ValueError``.
+
+Executor command
+----------------
+
+``AC_preprocess_image`` runs the pipeline and *writes* the result to
+``output_path`` (so it is usable from JSON / MCP / the builder): ``source`` is an
+image path (default: screen grab of ``region``), ``steps`` an ordered list (or
+comma string), plus ``scale`` / ``block_size`` / ``c``; it returns
+``{path, width, height}``. It is exposed as the MCP tool ``ac_preprocess_image``
+and as a Script Builder command under **Image**.

docs/source/Eng/eng_index.rst

@@ -157,6 +157,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v132_features_doc
    doc/new_features/v133_features_doc
    doc/new_features/v134_features_doc
+   doc/new_features/v135_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v135_features_doc.rst

@@ -0,0 +1,39 @@
+影像前處理(供 OCR / 模板比對)
+================================
+
+``locate_text`` / ``ocr_read_structure`` 與 ``match_template`` 是把*原始*螢幕擷取直接餵給 OCR 引擎或比對器。
+小字、暗色主題、低對比與略為旋轉的截圖會嚴重影響兩者——而框架先前完全沒有前處理接縫。本功能加入標準前處理
+流程——灰階 → 放大 → 二值化 → 去歪斜 → 去噪 → CLAHE 對比——以倍增你已在使用的 OCR 與比對功能的準確度。
+
+每個函式都在可注入的 ``haystack`` 影像(ndarray / 路徑 / PIL,預設為對 ``region`` 擷取螢幕)上執行並回傳
+NumPy ndarray,因此可對合成陣列做單元測試。OpenCV + NumPy 透過 ``je_open_cv`` 引入。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import preprocess_image, binarize, deskew, upscale
+
+    # 一次性流程,再對清理後的影像做 OCR。
+    clean = preprocess_image("receipt.png", steps=("grayscale", "upscale",
+                                                    "deskew", "binarize"), scale=2.0)
+
+    # 或個別組合各步驟。
+    bw = binarize("panel.png", method="adaptive_gaussian", block_size=41)
+    straight = deskew("scan.png", max_angle=10.0)
+    big = upscale("tiny_label.png", scale=3.0, interp="lanczos")
+
+基礎元件有 ``to_grayscale``、``upscale``(``scale`` / ``interp``)、``binarize``(``method`` =
+``otsu`` / ``adaptive_mean`` / ``adaptive_gaussian``)、``denoise``、``enhance_contrast``(CLAHE)、``deskew``
+以及 ``detect_skew_angle``(回傳量測到的文字歪斜角度,夾在 ``±max_angle``)。``preprocess_image`` 依序串接任意
+具名 ``steps``——``grayscale``、``upscale``、``binarize``、``denoise``、``deskew``、``contrast``;未知步驟名稱
+會丟出 ``ValueError``。
+
+執行器命令
+----------
+
+``AC_preprocess_image`` 執行流程並把結果*寫入* ``output_path``(因此可從 JSON / MCP / builder 使用):
+``source`` 為影像路徑(預設為對 ``region`` 擷取螢幕)、``steps`` 為有序清單(或逗號字串),另有 ``scale`` /
+``block_size`` / ``c``;回傳 ``{path, width, height}``。它以 MCP 工具 ``ac_preprocess_image`` 以及 Script
+Builder 中 **Image** 分類下的命令提供。

docs/source/Zh/zh_index.rst

@@ -157,6 +157,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v132_features_doc
    doc/new_features/v133_features_doc
    doc/new_features/v134_features_doc
+   doc/new_features/v135_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -298,6 +298,11 @@
 from je_auto_control.utils.window_layout import (
     WindowRect, available_slots, cascade_rects, grid_rects, tile_rect,
 )
+# Image pre-processing for OCR / template matching (grayscale, binarize, deskew, …)
+from je_auto_control.utils.preprocess import (
+    binarize, denoise, deskew, detect_skew_angle, enhance_contrast,
+    preprocess_image, to_grayscale, upscale,
+)
 # CI workflow annotations (GitHub Actions)
 from je_auto_control.utils.ci_annotations import (
     emit_annotations, format_annotation,
@@ -1106,6 +1111,14 @@ def start_autocontrol_gui(*args, **kwargs):
     "tile_rect",
     "grid_rects",
     "cascade_rects",
+    "preprocess_image",
+    "to_grayscale",
+    "binarize",
+    "upscale",
+    "denoise",
+    "deskew",
+    "detect_skew_angle",
+    "enhance_contrast",
     "emit_annotations", "format_annotation",
     "ClipboardHistory", "default_clipboard_history",
     "analyze_heal_log", "heal_stats", "scan_secrets",

je_auto_control/gui/script_builder/command_schema.py

@@ -262,7 +262,7 @@
             FieldSpec("scales", FieldType.STRING, optional=True,
                       placeholder="[0.9, 1.0, 1.1]"),
             FieldSpec("region", FieldType.STRING, optional=True,
                       placeholder="[left, top, right, bottom]"),
         ),
         description="Locate a template and return its confidence score + scale.",
     ))
@@ -378,6 +378,19 @@
         ),
         description="Locate rectangular regions (buttons / cards / input fields).",
     ))
+    specs.append(CommandSpec(
+        "AC_preprocess_image", "Image", "Preprocess Image (OCR/match)",
+        fields=(
+            FieldSpec("output_path", FieldType.FILE_PATH),
+            FieldSpec("source", FieldType.FILE_PATH, optional=True),
+            FieldSpec("steps", FieldType.STRING, optional=True,
+                      placeholder="grayscale,upscale,binarize"),
+            FieldSpec("scale", FieldType.FLOAT, optional=True, default=2.0),
+            FieldSpec("region", FieldType.STRING, optional=True,
+                      placeholder="[left, top, right, bottom]"),
+        ),
+        description="Clean up an image for OCR / matching (grayscale/binarize/…).",
+    ))
 
 
 def _add_ocr_specs(specs: List[CommandSpec]) -> None:

je_auto_control/utils/executor/action_executor.py

@@ -3409,6 +3409,28 @@ def _cascade_rects(count: Any, screen: Any = None, offset: Any = 30,
     return {"count": len(rects), "rects": [rect.to_dict() for rect in rects]}
 
 
+def _preprocess_image(output_path: str, source: Any = None, steps: Any = None,
+                      scale: Any = 2.0, region: Any = None, block_size: Any = 31,
+                      c: Any = 11) -> Dict[str, Any]:
+    """Adapter: run the preprocessing pipeline and write the result to a file."""
+    import json
+    import cv2
+    from je_auto_control.utils.preprocess import preprocess_image
+    if isinstance(steps, str):
+        steps = (json.loads(steps) if steps.strip().startswith("[")
+                 else [part.strip() for part in steps.split(",") if part.strip()])
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    result = preprocess_image(
+        source, region=region,
+        steps=tuple(steps) if steps else ("grayscale", "upscale", "binarize"),
+        scale=float(scale), block_size=int(block_size), c=int(c))
+    if not cv2.imwrite(str(output_path), result):
+        raise AutoControlActionException(f"could not write image: {output_path!r}")
+    return {"path": str(output_path), "width": int(result.shape[1]),
+            "height": int(result.shape[0])}
+
+
 def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]:
     """Adapter: run nested actions while modifier keys are held down."""
     import json
@@ -5135,6 +5157,7 @@ def __init__(self):
             "AC_feature_match": _feature_match,
             "AC_find_shapes": _find_shapes,
             "AC_find_rectangles": _find_rectangles,
+            "AC_preprocess_image": _preprocess_image,
             "AC_tile_rect": _tile_rect,
             "AC_grid_rects": _grid_rects,
             "AC_cascade_rects": _cascade_rects,

je_auto_control/utils/mcp_server/tools/_factories.py

@@ -2791,6 +2791,31 @@ def window_arrange_tools() -> List[MCPTool]:
     ]
 
 
+def preprocess_tools() -> List[MCPTool]:
+    return [
+        MCPTool(
+            name="ac_preprocess_image",
+            description=("Pre-process an image for OCR / template matching and "
+                         "WRITE the result to 'output_path'. 'source' is an image "
+                         "path (default: screen grab of 'region'). 'steps' is an "
+                         "ordered list from grayscale/upscale/binarize/denoise/"
+                         "deskew/contrast (default grayscale,upscale,binarize); "
+                         "'scale' for upscale. Returns {path, width, height}."),
+            input_schema=schema({
+                "output_path": {"type": "string"},
+                "source": {"type": "string"},
+                "steps": {"type": "array", "items": {"type": "string"}},
+                "scale": {"type": "number"},
+                "region": {"type": "array", "items": {"type": "integer"}},
+                "block_size": {"type": "integer"},
+                "c": {"type": "integer"}},
+                required=["output_path"]),
+            handler=h.preprocess_image,
+            annotations=SIDE_EFFECT_ONLY,
+        ),
+    ]
+
+
 def ssim_tools() -> List[MCPTool]:
     return [
         MCPTool(
@@ -6293,7 +6318,8 @@ def media_assert_tools() -> List[MCPTool]:
     key_hold_tools, mouse_relative_tools, text_unicode_tools,
     modifier_state_tools, grid_locator_tools, visual_match_tools,
     color_region_tools, ssim_tools, feature_match_tools, shape_locator_tools,
-    window_layout_tools, window_arrange_tools, plugin_sdk_tools, governance_tools,
+    window_layout_tools, window_arrange_tools, preprocess_tools, plugin_sdk_tools,
+    governance_tools,
     credential_lease_tools, egress_tools, approval_testing_tools,
     trajectory_eval_tools, compliance_tools, agent_trace_tools,
     video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools,

je_auto_control/utils/mcp_server/tools/_handlers.py

@@ -2155,6 +2155,13 @@ def arrange_cascade(titles, offset=30):
     return _arrange_cascade(titles, offset)
 
 
+def preprocess_image(output_path, source=None, steps=None, scale=2.0, region=None,
+                     block_size=31, c=11):
+    from je_auto_control.utils.executor.action_executor import _preprocess_image
+    return _preprocess_image(output_path, source, steps, scale, region,
+                             block_size, c)
+
+
 def detect_drift(reference, current, threshold=0.25, bins=10):
     from je_auto_control.utils.executor.action_executor import _detect_drift
     return _detect_drift(reference, current, threshold, bins)

je_auto_control/utils/preprocess/__init__.py

@@ -0,0 +1,8 @@
+"""Image pre-processing for OCR / template matching (grayscale, binarize, deskew, …)."""
+from je_auto_control.utils.preprocess.preprocess import (
+    binarize, denoise, deskew, detect_skew_angle, enhance_contrast,
+    preprocess_image, to_grayscale, upscale,
+)
+
+__all__ = ["binarize", "denoise", "deskew", "detect_skew_angle",
+           "enhance_contrast", "preprocess_image", "to_grayscale", "upscale"]