diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md index 1db86cc6..8b325307 100644 --- a/README/WHATS_NEW_zh-CN.md +++ b/README/WHATS_NEW_zh-CN.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-23) — 图像预处理(供 OCR / 模板匹配) + +在识别或匹配前先清理画面。完整参考:[`docs/source/Zh/doc/new_features/v135_features_doc.rst`](../docs/source/Zh/doc/new_features/v135_features_doc.rst)。 + +- **`preprocess_image` + `to_grayscale` / `binarize` / `upscale` / `denoise` / `deskew` / `enhance_contrast`**(`AC_preprocess_image`):`locate_text` 与 `match_template` 把*原始*截取直接喂给 OCR / 匹配器——小字、暗色主题、低对比与歪斜会严重影响两者,而框架毫无预处理接缝。本功能加入标准流程(灰阶 → 放大 → 二值化 → 去歪斜 → 去噪 → CLAHE),倍增其准确度。可注入 haystack → ndarray;`detect_skew_angle` 测量文字旋转;`binarize` 提供 otsu / adaptive。执行器命令把清理后图像写入路径。可对合成数组无头测试。 + ## 本次更新 (2026-06-23) — 排列多个窗口(网格 / 层叠) 一次调用排好一整组窗口。完整参考:[`docs/source/Zh/doc/new_features/v134_features_doc.rst`](../docs/source/Zh/doc/new_features/v134_features_doc.rst)。 diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md index 317b0e09..a0bc750d 100644 --- a/README/WHATS_NEW_zh-TW.md +++ b/README/WHATS_NEW_zh-TW.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-23) — 影像前處理(供 OCR / 模板比對) + +在辨識或比對前先清理畫面。完整參考:[`docs/source/Zh/doc/new_features/v135_features_doc.rst`](../docs/source/Zh/doc/new_features/v135_features_doc.rst)。 + +- **`preprocess_image` + `to_grayscale` / `binarize` / `upscale` / `denoise` / `deskew` / `enhance_contrast`**(`AC_preprocess_image`):`locate_text` 與 `match_template` 把*原始*擷取直接餵給 OCR / 比對器——小字、暗色主題、低對比與歪斜會嚴重影響兩者,而框架毫無前處理接縫。本功能加入標準流程(灰階 → 放大 → 二值化 → 去歪斜 → 去噪 → CLAHE),倍增其準確度。可注入 haystack → ndarray;`detect_skew_angle` 量測文字旋轉;`binarize` 提供 otsu / adaptive。執行器命令把清理後影像寫入路徑。可對合成陣列無頭測試。 + ## 本次更新 (2026-06-23) — 排列多個視窗(網格 / 層疊) 一次呼叫排好一整組視窗。完整參考:[`docs/source/Zh/doc/new_features/v134_features_doc.rst`](../docs/source/Zh/doc/new_features/v134_features_doc.rst)。 diff --git a/WHATS_NEW.md b/WHATS_NEW.md index 96518726..367975e9 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -1,5 +1,11 @@ # What's New — AutoControl +## What's new (2026-06-23) — Image Pre-processing for OCR / Template Matching + +Clean up the screen before reading or matching it. Full reference: [`docs/source/Eng/doc/new_features/v135_features_doc.rst`](docs/source/Eng/doc/new_features/v135_features_doc.rst). + +- **`preprocess_image` + `to_grayscale` / `binarize` / `upscale` / `denoise` / `deskew` / `enhance_contrast`** (`AC_preprocess_image`): `locate_text` and `match_template` fed the *raw* capture to OCR / the matcher — small text, dark themes, low contrast and skew wrecked both, with no preprocessing seam anywhere. This adds the standard pipeline (grayscale → upscale → binarize → deskew → denoise → CLAHE) that multiplies their accuracy. Injectable haystack → ndarray; `detect_skew_angle` measures text rotation; `binarize` does otsu / adaptive. The executor command writes the cleaned image to a path. Headless-testable on synthetic arrays. + ## What's new (2026-06-23) — Arrange Multiple Windows (Grid / Cascade) Lay out a whole set of windows in one call. Full reference: [`docs/source/Eng/doc/new_features/v134_features_doc.rst`](docs/source/Eng/doc/new_features/v134_features_doc.rst). diff --git a/docs/source/Eng/doc/new_features/v135_features_doc.rst b/docs/source/Eng/doc/new_features/v135_features_doc.rst new file mode 100644 index 00000000..5965407b --- /dev/null +++ b/docs/source/Eng/doc/new_features/v135_features_doc.rst @@ -0,0 +1,48 @@ +Image Pre-processing for OCR / Template Matching +================================================ + +``locate_text`` / ``ocr_read_structure`` and ``match_template`` feed the *raw* +screen capture straight to the OCR engine or the matcher. Small UI text, dark +themes, low contrast and a slightly rotated screenshot wreck both — and there was +no preprocessing seam anywhere in the framework. This adds the standard pre-step +pipeline — grayscale → upscale → binarize → deskew → denoise → CLAHE contrast — +that multiplies the accuracy of the OCR and matching features you already use. + +Every function runs on an injectable ``haystack`` image (ndarray / path / PIL, +default: grab the screen / ``region``) and returns a NumPy ndarray, so it is +unit-testable on synthetic arrays. OpenCV + NumPy come in via ``je_open_cv``. +Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import preprocess_image, binarize, deskew, upscale + + # One-shot pipeline, then OCR the cleaned image. + clean = preprocess_image("receipt.png", steps=("grayscale", "upscale", + "deskew", "binarize"), scale=2.0) + + # Or compose the individual steps. + bw = binarize("panel.png", method="adaptive_gaussian", block_size=41) + straight = deskew("scan.png", max_angle=10.0) + big = upscale("tiny_label.png", scale=3.0, interp="lanczos") + +The building blocks are ``to_grayscale``, ``upscale`` (``scale`` / ``interp``), +``binarize`` (``method`` = ``otsu`` / ``adaptive_mean`` / ``adaptive_gaussian``), +``denoise``, ``enhance_contrast`` (CLAHE), ``deskew`` and ``detect_skew_angle`` +(returns the measured text-skew in degrees, clamped to ``±max_angle``). +``preprocess_image`` chains any of the named ``steps`` — ``grayscale``, +``upscale``, ``binarize``, ``denoise``, ``deskew``, ``contrast`` — in order; +unknown step names raise ``ValueError``. + +Executor command +---------------- + +``AC_preprocess_image`` runs the pipeline and *writes* the result to +``output_path`` (so it is usable from JSON / MCP / the builder): ``source`` is an +image path (default: screen grab of ``region``), ``steps`` an ordered list (or +comma string), plus ``scale`` / ``block_size`` / ``c``; it returns +``{path, width, height}``. It is exposed as the MCP tool ``ac_preprocess_image`` +and as a Script Builder command under **Image**. diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index b0b3405c..c0811d76 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -157,6 +157,7 @@ Comprehensive guides for all AutoControl features. doc/new_features/v132_features_doc doc/new_features/v133_features_doc doc/new_features/v134_features_doc + doc/new_features/v135_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/docs/source/Zh/doc/new_features/v135_features_doc.rst b/docs/source/Zh/doc/new_features/v135_features_doc.rst new file mode 100644 index 00000000..cb5a7e1e --- /dev/null +++ b/docs/source/Zh/doc/new_features/v135_features_doc.rst @@ -0,0 +1,39 @@ +影像前處理(供 OCR / 模板比對) +================================ + +``locate_text`` / ``ocr_read_structure`` 與 ``match_template`` 是把*原始*螢幕擷取直接餵給 OCR 引擎或比對器。 +小字、暗色主題、低對比與略為旋轉的截圖會嚴重影響兩者——而框架先前完全沒有前處理接縫。本功能加入標準前處理 +流程——灰階 → 放大 → 二值化 → 去歪斜 → 去噪 → CLAHE 對比——以倍增你已在使用的 OCR 與比對功能的準確度。 + +每個函式都在可注入的 ``haystack`` 影像(ndarray / 路徑 / PIL,預設為對 ``region`` 擷取螢幕)上執行並回傳 +NumPy ndarray,因此可對合成陣列做單元測試。OpenCV + NumPy 透過 ``je_open_cv`` 引入。不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import preprocess_image, binarize, deskew, upscale + + # 一次性流程,再對清理後的影像做 OCR。 + clean = preprocess_image("receipt.png", steps=("grayscale", "upscale", + "deskew", "binarize"), scale=2.0) + + # 或個別組合各步驟。 + bw = binarize("panel.png", method="adaptive_gaussian", block_size=41) + straight = deskew("scan.png", max_angle=10.0) + big = upscale("tiny_label.png", scale=3.0, interp="lanczos") + +基礎元件有 ``to_grayscale``、``upscale``(``scale`` / ``interp``)、``binarize``(``method`` = +``otsu`` / ``adaptive_mean`` / ``adaptive_gaussian``)、``denoise``、``enhance_contrast``(CLAHE)、``deskew`` +以及 ``detect_skew_angle``(回傳量測到的文字歪斜角度,夾在 ``±max_angle``)。``preprocess_image`` 依序串接任意 +具名 ``steps``——``grayscale``、``upscale``、``binarize``、``denoise``、``deskew``、``contrast``;未知步驟名稱 +會丟出 ``ValueError``。 + +執行器命令 +---------- + +``AC_preprocess_image`` 執行流程並把結果*寫入* ``output_path``(因此可從 JSON / MCP / builder 使用): +``source`` 為影像路徑(預設為對 ``region`` 擷取螢幕)、``steps`` 為有序清單(或逗號字串),另有 ``scale`` / +``block_size`` / ``c``;回傳 ``{path, width, height}``。它以 MCP 工具 ``ac_preprocess_image`` 以及 Script +Builder 中 **Image** 分類下的命令提供。 diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst index 2dacab6f..9a4c394f 100644 --- a/docs/source/Zh/zh_index.rst +++ b/docs/source/Zh/zh_index.rst @@ -157,6 +157,7 @@ AutoControl 所有功能的完整使用指南。 doc/new_features/v132_features_doc doc/new_features/v133_features_doc doc/new_features/v134_features_doc + doc/new_features/v135_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index 4397ec25..55a063f2 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -298,6 +298,11 @@ from je_auto_control.utils.window_layout import ( WindowRect, available_slots, cascade_rects, grid_rects, tile_rect, ) +# Image pre-processing for OCR / template matching (grayscale, binarize, deskew, …) +from je_auto_control.utils.preprocess import ( + binarize, denoise, deskew, detect_skew_angle, enhance_contrast, + preprocess_image, to_grayscale, upscale, +) # CI workflow annotations (GitHub Actions) from je_auto_control.utils.ci_annotations import ( emit_annotations, format_annotation, @@ -1106,6 +1111,14 @@ def start_autocontrol_gui(*args, **kwargs): "tile_rect", "grid_rects", "cascade_rects", + "preprocess_image", + "to_grayscale", + "binarize", + "upscale", + "denoise", + "deskew", + "detect_skew_angle", + "enhance_contrast", "emit_annotations", "format_annotation", "ClipboardHistory", "default_clipboard_history", "analyze_heal_log", "heal_stats", "scan_secrets", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index bc2101bd..b24f5d9f 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -378,6 +378,19 @@ def _add_image_specs(specs: List[CommandSpec]) -> None: ), description="Locate rectangular regions (buttons / cards / input fields).", )) + specs.append(CommandSpec( + "AC_preprocess_image", "Image", "Preprocess Image (OCR/match)", + fields=( + FieldSpec("output_path", FieldType.FILE_PATH), + FieldSpec("source", FieldType.FILE_PATH, optional=True), + FieldSpec("steps", FieldType.STRING, optional=True, + placeholder="grayscale,upscale,binarize"), + FieldSpec("scale", FieldType.FLOAT, optional=True, default=2.0), + FieldSpec("region", FieldType.STRING, optional=True, + placeholder="[left, top, right, bottom]"), + ), + description="Clean up an image for OCR / matching (grayscale/binarize/…).", + )) def _add_ocr_specs(specs: List[CommandSpec]) -> None: diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index ee8c5d58..969fabc9 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -3409,6 +3409,28 @@ def _cascade_rects(count: Any, screen: Any = None, offset: Any = 30, return {"count": len(rects), "rects": [rect.to_dict() for rect in rects]} +def _preprocess_image(output_path: str, source: Any = None, steps: Any = None, + scale: Any = 2.0, region: Any = None, block_size: Any = 31, + c: Any = 11) -> Dict[str, Any]: + """Adapter: run the preprocessing pipeline and write the result to a file.""" + import json + import cv2 + from je_auto_control.utils.preprocess import preprocess_image + if isinstance(steps, str): + steps = (json.loads(steps) if steps.strip().startswith("[") + else [part.strip() for part in steps.split(",") if part.strip()]) + if isinstance(region, str): + region = json.loads(region) if region.strip() else None + result = preprocess_image( + source, region=region, + steps=tuple(steps) if steps else ("grayscale", "upscale", "binarize"), + scale=float(scale), block_size=int(block_size), c=int(c)) + if not cv2.imwrite(str(output_path), result): + raise AutoControlActionException(f"could not write image: {output_path!r}") + return {"path": str(output_path), "width": int(result.shape[1]), + "height": int(result.shape[0])} + + def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]: """Adapter: run nested actions while modifier keys are held down.""" import json @@ -5135,6 +5157,7 @@ def __init__(self): "AC_feature_match": _feature_match, "AC_find_shapes": _find_shapes, "AC_find_rectangles": _find_rectangles, + "AC_preprocess_image": _preprocess_image, "AC_tile_rect": _tile_rect, "AC_grid_rects": _grid_rects, "AC_cascade_rects": _cascade_rects, diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index 02c90dcb..eb7f8d45 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -2791,6 +2791,31 @@ def window_arrange_tools() -> List[MCPTool]: ] +def preprocess_tools() -> List[MCPTool]: + return [ + MCPTool( + name="ac_preprocess_image", + description=("Pre-process an image for OCR / template matching and " + "WRITE the result to 'output_path'. 'source' is an image " + "path (default: screen grab of 'region'). 'steps' is an " + "ordered list from grayscale/upscale/binarize/denoise/" + "deskew/contrast (default grayscale,upscale,binarize); " + "'scale' for upscale. Returns {path, width, height}."), + input_schema=schema({ + "output_path": {"type": "string"}, + "source": {"type": "string"}, + "steps": {"type": "array", "items": {"type": "string"}}, + "scale": {"type": "number"}, + "region": {"type": "array", "items": {"type": "integer"}}, + "block_size": {"type": "integer"}, + "c": {"type": "integer"}}, + required=["output_path"]), + handler=h.preprocess_image, + annotations=SIDE_EFFECT_ONLY, + ), + ] + + def ssim_tools() -> List[MCPTool]: return [ MCPTool( @@ -6293,7 +6318,8 @@ def media_assert_tools() -> List[MCPTool]: key_hold_tools, mouse_relative_tools, text_unicode_tools, modifier_state_tools, grid_locator_tools, visual_match_tools, color_region_tools, ssim_tools, feature_match_tools, shape_locator_tools, - window_layout_tools, window_arrange_tools, plugin_sdk_tools, governance_tools, + window_layout_tools, window_arrange_tools, preprocess_tools, plugin_sdk_tools, + governance_tools, credential_lease_tools, egress_tools, approval_testing_tools, trajectory_eval_tools, compliance_tools, agent_trace_tools, video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools, diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py index c9fa87a9..1f65c634 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers.py @@ -2155,6 +2155,13 @@ def arrange_cascade(titles, offset=30): return _arrange_cascade(titles, offset) +def preprocess_image(output_path, source=None, steps=None, scale=2.0, region=None, + block_size=31, c=11): + from je_auto_control.utils.executor.action_executor import _preprocess_image + return _preprocess_image(output_path, source, steps, scale, region, + block_size, c) + + def detect_drift(reference, current, threshold=0.25, bins=10): from je_auto_control.utils.executor.action_executor import _detect_drift return _detect_drift(reference, current, threshold, bins) diff --git a/je_auto_control/utils/preprocess/__init__.py b/je_auto_control/utils/preprocess/__init__.py new file mode 100644 index 00000000..ad8aaebb --- /dev/null +++ b/je_auto_control/utils/preprocess/__init__.py @@ -0,0 +1,8 @@ +"""Image pre-processing for OCR / template matching (grayscale, binarize, deskew, …).""" +from je_auto_control.utils.preprocess.preprocess import ( + binarize, denoise, deskew, detect_skew_angle, enhance_contrast, + preprocess_image, to_grayscale, upscale, +) + +__all__ = ["binarize", "denoise", "deskew", "detect_skew_angle", + "enhance_contrast", "preprocess_image", "to_grayscale", "upscale"] diff --git a/je_auto_control/utils/preprocess/preprocess.py b/je_auto_control/utils/preprocess/preprocess.py new file mode 100644 index 00000000..0f62e63e --- /dev/null +++ b/je_auto_control/utils/preprocess/preprocess.py @@ -0,0 +1,177 @@ +"""Image pre-processing for OCR / template matching — grayscale, binarize, deskew, upscale. + +``locate_text`` / ``ocr_read_structure`` and ``match_template`` feed the *raw* capture +to the OCR engine / matcher; small UI text, dark themes and low contrast wreck both. +This is the standard pre-step pipeline — grayscale → upscale → binarize → deskew → +denoise → CLAHE contrast — that multiplies their accuracy, with no preprocessing seam +anywhere in the framework today. + +Every function runs on an injectable ``haystack`` image (ndarray / path / PIL, default: +grab the screen / ``region``) and returns a NumPy ndarray you can pass straight to an +OCR / match call or save. OpenCV + NumPy come in via the project's ``je_open_cv`` +dependency and are imported lazily. Imports no ``PySide6``. +""" +from typing import Any, Optional, Sequence + +ImageSource = Any +_INTERP = ("nearest", "linear", "cubic", "lanczos") + + +def _to_array(source: ImageSource): + """Load a path / ndarray / PIL image as a uint8 ndarray (as stored).""" + import cv2 + import numpy as np + if hasattr(source, "shape"): + return np.asarray(source) + if isinstance(source, (str, bytes)) or hasattr(source, "__fspath__"): + array = cv2.imread(str(source), cv2.IMREAD_UNCHANGED) + if array is None: + raise ValueError(f"could not read image: {source!r}") + return array + return np.asarray(source) + + +def _resolve(haystack: Optional[ImageSource], region: Optional[Sequence[int]]): + import numpy as np + if haystack is not None: + return _to_array(haystack) + from je_auto_control.utils.cv2_utils.screenshot import pil_screenshot + image = pil_screenshot(screen_region=list(region) if region else None) + return np.asarray(image.convert("RGB")) + + +def _gray(array): + import cv2 + if array.ndim == 2: + return array + code = cv2.COLOR_BGRA2GRAY if array.shape[2] == 4 else cv2.COLOR_BGR2GRAY + return cv2.cvtColor(array, code) + + +def to_grayscale(haystack: Optional[ImageSource] = None, *, + region: Optional[Sequence[int]] = None): + """Return the image as a single-channel grayscale ndarray.""" + return _gray(_resolve(haystack, region)) + + +def upscale(haystack: Optional[ImageSource] = None, *, + region: Optional[Sequence[int]] = None, scale: float = 2.0, + interp: str = "cubic"): + """Return the image resized by ``scale`` — enlarge small UI text before OCR.""" + import cv2 + if interp not in _INTERP: + raise ValueError(f"unknown interp: {interp!r}") + table = {"nearest": cv2.INTER_NEAREST, "linear": cv2.INTER_LINEAR, + "cubic": cv2.INTER_CUBIC, "lanczos": cv2.INTER_LANCZOS4} + array = _resolve(haystack, region) + height, width = array.shape[:2] + size = (max(1, round(width * float(scale))), max(1, round(height * float(scale)))) + return cv2.resize(array, size, interpolation=table[interp]) + + +def binarize(haystack: Optional[ImageSource] = None, *, + region: Optional[Sequence[int]] = None, method: str = "otsu", + block_size: int = 31, c: int = 11): + """Return a black/white image. ``method``: otsu / adaptive_mean / adaptive_gaussian.""" + import cv2 + gray = _gray(_resolve(haystack, region)) + if method == "otsu": + _, result = cv2.threshold(gray, 0, 255, + cv2.THRESH_BINARY + cv2.THRESH_OTSU) + return result + table = {"adaptive_mean": cv2.ADAPTIVE_THRESH_MEAN_C, + "adaptive_gaussian": cv2.ADAPTIVE_THRESH_GAUSSIAN_C} + if method not in table: + raise ValueError(f"unknown method: {method!r}") + block = int(block_size) | 1 # adaptiveThreshold needs odd + return cv2.adaptiveThreshold(gray, 255, table[method], cv2.THRESH_BINARY, + block, int(c)) + + +def denoise(haystack: Optional[ImageSource] = None, *, + region: Optional[Sequence[int]] = None, strength: int = 7): + """Return a denoised grayscale image (non-local means).""" + import cv2 + return cv2.fastNlMeansDenoising(_gray(_resolve(haystack, region)), None, + float(strength), 7, 21) + + +def enhance_contrast(haystack: Optional[ImageSource] = None, *, + region: Optional[Sequence[int]] = None, clip: float = 2.0, + grid: int = 8): + """Return a CLAHE contrast-enhanced grayscale image (rescues dark/low-contrast UI).""" + import cv2 + clahe = cv2.createCLAHE(clipLimit=float(clip), + tileGridSize=(int(grid), int(grid))) + return clahe.apply(_gray(_resolve(haystack, region))) + + +def detect_skew_angle(haystack: Optional[ImageSource] = None, *, + region: Optional[Sequence[int]] = None, + max_angle: float = 15.0) -> float: + """Return the text skew angle in degrees within ``[-max_angle, max_angle]`` (else 0).""" + import cv2 + gray = _gray(_resolve(haystack, region)) + _, mask = cv2.threshold(gray, 0, 255, + cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU) + coords = cv2.findNonZero(mask) + if coords is None: + return 0.0 + angle = cv2.minAreaRect(coords)[-1] % 90 # version-robust normalisation + if angle > 45: + angle -= 90 + return round(float(angle), 3) if abs(angle) <= float(max_angle) else 0.0 + + +def deskew(haystack: Optional[ImageSource] = None, *, + region: Optional[Sequence[int]] = None, max_angle: float = 15.0): + """Return the image rotated to remove text skew (no-op when none is detected).""" + import cv2 + array = _resolve(haystack, region) + angle = detect_skew_angle(array, max_angle=max_angle) + if abs(angle) < 1e-9: + return array + height, width = array.shape[:2] + matrix = cv2.getRotationMatrix2D((width / 2.0, height / 2.0), angle, 1.0) + return cv2.warpAffine(array, matrix, (width, height), + flags=cv2.INTER_CUBIC, borderMode=cv2.BORDER_REPLICATE) + + +def _step_grayscale(array, **_kwargs): + return to_grayscale(array) + + +def _step_upscale(array, *, scale: float, **_kwargs): + return upscale(array, scale=scale) + + +def _step_binarize(array, *, block_size: int, c: int, **_kwargs): + return binarize(array, block_size=block_size, c=c) + + +_STEPS = { + "grayscale": _step_grayscale, + "upscale": _step_upscale, + "binarize": _step_binarize, + "denoise": lambda array, **_kwargs: denoise(array), + "deskew": lambda array, **_kwargs: deskew(array), + "contrast": lambda array, **_kwargs: enhance_contrast(array), +} + + +def preprocess_image(haystack: Optional[ImageSource] = None, *, + region: Optional[Sequence[int]] = None, + steps: Sequence[str] = ("grayscale", "upscale", "binarize"), + scale: float = 2.0, block_size: int = 31, c: int = 11): + """Apply a pipeline of named preprocessing ``steps`` in order, returning the result. + + Steps: ``grayscale``, ``upscale`` (by ``scale``), ``binarize`` (otsu, tuned by + ``block_size`` / ``c`` only for the adaptive variants), ``denoise``, ``deskew``, + ``contrast`` (CLAHE). Unknown step names raise ``ValueError``. + """ + array = _resolve(haystack, region) + for step in steps: + if step not in _STEPS: + raise ValueError(f"unknown step: {step!r}") + array = _STEPS[step](array, scale=scale, block_size=block_size, c=c) + return array diff --git a/test/unit_test/headless/test_preprocess_batch.py b/test/unit_test/headless/test_preprocess_batch.py new file mode 100644 index 00000000..9488be66 --- /dev/null +++ b/test/unit_test/headless/test_preprocess_batch.py @@ -0,0 +1,113 @@ +"""Headless tests for OCR/match image preprocessing. No Qt.""" +import pytest + +import je_auto_control as ac + +np = pytest.importorskip("numpy") +cv2 = pytest.importorskip("cv2") + +from je_auto_control.utils.preprocess import ( # noqa: E402 + binarize, denoise, deskew, detect_skew_angle, enhance_contrast, + preprocess_image, to_grayscale, upscale, +) + + +def _color(): + img = np.zeros((40, 60, 3), dtype=np.uint8) + img[:, :30] = (80, 80, 80) + img[:, 30:] = (200, 200, 200) + return img + + +def _skewed(angle): + """A dark horizontal bar on a light background, rotated by ``angle`` degrees.""" + canvas = np.full((120, 200), 255, dtype=np.uint8) + cv2.rectangle(canvas, (40, 55), (160, 65), 0, -1) + matrix = cv2.getRotationMatrix2D((100, 60), angle, 1.0) + return cv2.warpAffine(canvas, matrix, (200, 120), borderValue=255) + + +def test_grayscale_drops_channels(): + assert to_grayscale(_color()).shape == (40, 60) + + +def test_upscale_doubles_dimensions(): + assert upscale(_color(), scale=2.0).shape[:2] == (80, 120) + + +def test_upscale_rejects_unknown_interp(): + with pytest.raises(ValueError): + upscale(_color(), interp="magic") + + +def test_binarize_otsu_is_two_valued(): + assert sorted(set(binarize(_color()).flatten().tolist())) == [0, 255] + + +def test_binarize_adaptive_keeps_shape(): + assert binarize(_color(), method="adaptive_gaussian").shape == (40, 60) + + +def test_binarize_rejects_unknown_method(): + with pytest.raises(ValueError): + binarize(_color(), method="triangle") + + +def test_denoise_and_contrast_keep_grayscale_shape(): + assert denoise(_color()).shape == (40, 60) + assert enhance_contrast(_color()).shape == (40, 60) + + +def test_detect_skew_recovers_angle_magnitude(): + assert 5.0 <= abs(detect_skew_angle(_skewed(10))) <= 15.0 + + +def test_detect_skew_clamps_to_zero_beyond_max(): + assert detect_skew_angle(_skewed(10), max_angle=3.0) == pytest.approx(0.0, abs=1e-9) + + +def test_deskew_reduces_skew(): + rotated = _skewed(10) + before = abs(detect_skew_angle(rotated)) + after = abs(detect_skew_angle(deskew(rotated))) + assert after < before + + +def test_pipeline_chains_steps(): + out = preprocess_image(_color(), steps=("grayscale", "upscale", "binarize")) + assert out.ndim == 2 and out.shape == (80, 120) + assert sorted(set(out.flatten().tolist())) == [0, 255] + + +def test_pipeline_rejects_unknown_step(): + with pytest.raises(ValueError): + preprocess_image(_color(), steps=("sharpen",)) + + +# --- wiring --------------------------------------------------------------- + +def test_wiring(): + assert "AC_preprocess_image" in set(ac.executor.known_commands()) + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert "ac_preprocess_image" in names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert "AC_preprocess_image" in specs + + +def test_facade_exports(): + for attr in ("preprocess_image", "to_grayscale", "binarize", "upscale", + "deskew", "enhance_contrast"): + assert hasattr(ac, attr) and attr in ac.__all__ + + +def test_executor_writes_output(tmp_path): + from je_auto_control.utils.executor.action_executor import _preprocess_image + src = tmp_path / "src.png" + cv2.imwrite(str(src), _color()) + out = tmp_path / "out.png" + result = _preprocess_image(str(out), source=str(src), + steps=["grayscale", "binarize"]) + assert result["path"] == str(out) and out.exists() + assert result["width"] == 60 and result["height"] == 40