Skip to content

Commit 4e404b1

Browse files
JE-ChenJE-Chen
authored
Merge pull request #231 from Integration-Automation/feat/screen-state-batch
Add semantic screen state: snapshot/diff and describe-screen
2 parents b96a650 + 4200b50 commit 4e404b1

15 files changed

Lines changed: 453 additions & 1 deletion

File tree

‎README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313

1414
## Table of Contents
1515

16+
- [What's new (2026-06-19) — Semantic Screen State](#whats-new-2026-06-19--semantic-screen-state)
1617
- [What's new (2026-06-19) — Set-of-Marks Overlay](#whats-new-2026-06-19--set-of-marks-overlay)
1718
- [What's new (2026-06-19) — Checkpoint & Resume](#whats-new-2026-06-19--checkpoint--resume)
1819
- [What's new (2026-06-19) — i18n / l10n Testing](#whats-new-2026-06-19--i18n--l10n-testing)
@@ -75,6 +76,13 @@
7576

7677
---
7778

79+
## What's new (2026-06-19) — Semantic Screen State
80+
81+
The semantic companion to the pixel diff, full stack. Full reference: [`docs/source/Eng/doc/new_features/v23_features_doc.rst`](docs/source/Eng/doc/new_features/v23_features_doc.rst).
82+
83+
- **Snapshot & diff** — `snapshot` / `diff_snapshots` / `snapshot_screen` / `screen_changed` (`AC_screen_snapshot` / `AC_screen_diff` / `AC_screen_changed`, `ac_*`): normalize the a11y tree to `{role, name, bbox}` and report what **appeared / vanished / moved** with a human-readable summary — the feedback signal an agent needs to verify a step ("Save dialog appeared").
84+
- **Describe the screen** — `describe_screen` (`AC_describe_screen`, `ac_describe_screen`): a compact "where am I" — role counts + interactive control labels.
85+
7886
## What's new (2026-06-19) — Set-of-Marks Overlay
7987

8088
The standard VLM-grounding format, full stack. Full reference: [`docs/source/Eng/doc/new_features/v22_features_doc.rst`](docs/source/Eng/doc/new_features/v22_features_doc.rst).

‎README/README_zh-CN.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目录
1414

15+
- [本次更新 (2026-06-19) — 语义屏幕状态](#本次更新-2026-06-19--语义屏幕状态)
1516
- [本次更新 (2026-06-19) — Set-of-Marks 叠图](#本次更新-2026-06-19--set-of-marks-叠图)
1617
- [本次更新 (2026-06-19) — 检查点与续跑](#本次更新-2026-06-19--检查点与续跑)
1718
- [本次更新 (2026-06-19) — i18n / l10n 测试](#本次更新-2026-06-19--i18n--l10n-测试)
@@ -74,6 +75,13 @@
7475

7576
---
7677

78+
## 本次更新 (2026-06-19) — 语义屏幕状态
79+
80+
像素差异的语义对应物,走完整五层。完整参考:[`docs/source/Zh/doc/new_features/v23_features_doc.rst`](../docs/source/Zh/doc/new_features/v23_features_doc.rst)。
81+
82+
- **快照与差异** — `snapshot` / `diff_snapshots` / `snapshot_screen` / `screen_changed`(`AC_screen_snapshot` / `AC_screen_diff` / `AC_screen_changed`、`ac_*`):把 a11y 树规范化为 `{role, name, bbox}`,报告**出现 / 消失 / 移动**并附人类可读摘要——agent 验证某步效果所需的反馈信号(「Save 对话框出现了」)。
83+
- **描述屏幕** — `describe_screen`(`AC_describe_screen`、`ac_describe_screen`):廉价的「我在哪」——各 role 计数 + 交互控件标签。
84+
7785
## 本次更新 (2026-06-19) — Set-of-Marks 叠图
7886

7987
VLM 定位的标准格式,走完整五层。完整参考:[`docs/source/Zh/doc/new_features/v22_features_doc.rst`](../docs/source/Zh/doc/new_features/v22_features_doc.rst)。

‎README/README_zh-TW.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目錄
1414

15+
- [本次更新 (2026-06-19) — 語意螢幕狀態](#本次更新-2026-06-19--語意螢幕狀態)
1516
- [本次更新 (2026-06-19) — Set-of-Marks 疊圖](#本次更新-2026-06-19--set-of-marks-疊圖)
1617
- [本次更新 (2026-06-19) — 檢查點與續跑](#本次更新-2026-06-19--檢查點與續跑)
1718
- [本次更新 (2026-06-19) — i18n / l10n 測試](#本次更新-2026-06-19--i18n--l10n-測試)
@@ -74,6 +75,13 @@
7475

7576
---
7677

78+
## 本次更新 (2026-06-19) — 語意螢幕狀態
79+
80+
像素差異的語意對應物,走完整五層。完整參考:[`docs/source/Zh/doc/new_features/v23_features_doc.rst`](../docs/source/Zh/doc/new_features/v23_features_doc.rst)。
81+
82+
- **快照與差異** — `snapshot` / `diff_snapshots` / `snapshot_screen` / `screen_changed`(`AC_screen_snapshot` / `AC_screen_diff` / `AC_screen_changed`、`ac_*`):把 a11y 樹正規化為 `{role, name, bbox}`,回報**出現 / 消失 / 移動**並附人類可讀摘要——agent 驗證某步效果所需的回饋訊號(「Save 對話框出現了」)。
83+
- **描述螢幕** — `describe_screen`(`AC_describe_screen`、`ac_describe_screen`):廉價的「我在哪」——各 role 計數 + 互動控制項標籤。
84+
7785
## 本次更新 (2026-06-19) — Set-of-Marks 疊圖
7886

7987
VLM 定位的標準格式,走完整五層。完整參考:[`docs/source/Zh/doc/new_features/v22_features_doc.rst`](../docs/source/Zh/doc/new_features/v22_features_doc.rst)。

‎docs/source/Eng/doc/new_features/v23_features_doc.rst‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
==================================================
2+
New Features (2026-06-19) — Semantic Screen State
3+
==================================================
4+
5+
The *semantic* companion to the existing pixel (visual-regression) diff:
6+
snapshot the accessibility tree, diff two snapshots into what **appeared /
7+
vanished / moved**, and get a compact structured **description** of the
8+
screen. This is the feedback signal an agent needs to verify a step's
9+
effect and orient itself. Pure standard library; full stack.
10+
11+
.. contents::
12+
:local:
13+
:depth: 2
14+
15+
16+
Snapshot & diff
17+
==============
18+
19+
::
20+
21+
from je_auto_control import snapshot, diff_snapshots, snapshot_screen, screen_changed
22+
23+
before = snapshot_screen() # baseline from the live a11y tree
24+
... # perform a step
25+
delta = screen_changed() # diff vs the baseline
26+
delta["summary"] # ["appeared: window Save", "moved: button OK"]
27+
28+
``snapshot`` normalizes elements to ``[{role, name, bbox}]`` (identity =
29+
``(role, name)``); ``diff_snapshots(before, after)`` returns ``added`` /
30+
``removed`` / ``moved`` lists plus a human-readable ``summary`` and
31+
``changed_count``. ``snapshot_screen`` / ``screen_changed`` capture and diff
32+
the *live* tree (caching the baseline). Exposed as ``AC_screen_snapshot`` /
33+
``AC_screen_diff`` / ``AC_screen_changed``.
34+
35+
36+
Describe the screen
37+
==================
38+
39+
::
40+
41+
from je_auto_control import describe_screen
42+
43+
describe_screen() # {app, element_count, by_role: {...}, controls: [...]}
44+
45+
A cheap "where am I" for an agent: counts per role and the labels of the
46+
interactive controls. Exposed as ``AC_describe_screen`` /
47+
``ac_describe_screen`` (and ``ac_screen_*`` for the diff family).

‎docs/source/Eng/eng_index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ Comprehensive guides for all AutoControl features.
4545
doc/new_features/v20_features_doc
4646
doc/new_features/v21_features_doc
4747
doc/new_features/v22_features_doc
48+
doc/new_features/v23_features_doc
4849
doc/ocr_backends/ocr_backends_doc
4950
doc/observability/observability_doc
5051
doc/operations_layer/operations_layer_doc

‎docs/source/Zh/doc/new_features/v23_features_doc.rst‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
==========================================
2+
新功能 (2026-06-19) — 語意螢幕狀態
3+
==========================================
4+
5+
既有像素(視覺回歸)差異的*語意*對應物:快照 accessibility 樹、把兩份
6+
快照差異成**出現 / 消失 / 移動**,並取得螢幕的精簡結構化**描述**。這是
7+
agent 驗證某步效果與自我定位所需的回饋訊號。純標準庫;走完整五層。
8+
9+
.. contents::
10+
:local:
11+
:depth: 2
12+
13+
14+
快照與差異
15+
==========
16+
17+
::
18+
19+
from je_auto_control import snapshot, diff_snapshots, snapshot_screen, screen_changed
20+
21+
before = snapshot_screen() # 從即時 a11y 樹取基準
22+
... # 執行某個步驟
23+
delta = screen_changed() # 與基準比對
24+
delta["summary"] # ["appeared: window Save", "moved: button OK"]
25+
26+
``snapshot`` 把元素正規化為 ``[{role, name, bbox}]``(識別 =
27+
``(role, name)``);``diff_snapshots(before, after)`` 回傳 ``added`` /
28+
``removed`` / ``moved`` 清單,加上人類可讀的 ``summary`` 與
29+
``changed_count``。``snapshot_screen`` / ``screen_changed`` 擷取並比對*即時*
30+
樹(會快取基準)。對應 ``AC_screen_snapshot`` / ``AC_screen_diff`` /
31+
``AC_screen_changed``。
32+
33+
34+
描述螢幕
35+
========
36+
37+
::
38+
39+
from je_auto_control import describe_screen
40+
41+
describe_screen() # {app, element_count, by_role: {...}, controls: [...]}
42+
43+
給 agent 的廉價「我在哪」:各 role 計數與互動控制項的標籤。對應
44+
``AC_describe_screen`` / ``ac_describe_screen``(差異家族則為 ``ac_screen_*``)。

‎docs/source/Zh/zh_index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ AutoControl 所有功能的完整使用指南。
4545
doc/new_features/v20_features_doc
4646
doc/new_features/v21_features_doc
4747
doc/new_features/v22_features_doc
48+
doc/new_features/v23_features_doc
4849
doc/ocr_backends/ocr_backends_doc
4950
doc/observability/observability_doc
5051
doc/operations_layer/operations_layer_doc

‎je_auto_control/__init__.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,11 @@
169169
from je_auto_control.utils.set_of_marks import (
170170
mark_click, mark_elements, mark_screen, render_marks, resolve_mark,
171171
)
172+
# Semantic screen state (snapshot/diff + structured description)
173+
from je_auto_control.utils.screen_state import (
174+
describe_screen, diff_snapshots, screen_changed, snapshot,
175+
snapshot_screen,
176+
)
172177
# Background popup/interrupt watchdog (unattended automation)
173178
from je_auto_control.utils.watchdog import (
174179
PopupWatchdog, WatchdogRule, default_popup_watchdog,
@@ -594,6 +599,8 @@ def start_autocontrol_gui(*args, **kwargs):
594599
"Checkpoint", "CheckpointStore", "run_resumable",
595600
"mark_click", "mark_elements", "mark_screen", "render_marks",
596601
"resolve_mark",
602+
"describe_screen", "diff_snapshots", "screen_changed", "snapshot",
603+
"snapshot_screen",
597604
# MCP server
598605
"AuditLogger", "HttpMCPServer", "MCPContent", "MCPPrompt",
599606
"MCPPromptArgument", "MCPResource", "MCPServer", "MCPTool",

‎je_auto_control/gui/script_builder/command_schema.py‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -662,6 +662,30 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None:
662662
_add_i18n_specs(specs)
663663
_add_checkpoint_specs(specs)
664664
_add_set_of_marks_specs(specs)
665+
_add_screen_state_specs(specs)
666+
667+
668+
def _add_screen_state_specs(specs: List[CommandSpec]) -> None:
669+
app = FieldSpec("app_name", FieldType.STRING, optional=True)
670+
specs.append(CommandSpec(
671+
"AC_screen_snapshot", "Native UI", "Screen: Snapshot Baseline",
672+
fields=(app,),
673+
description="Snapshot the a11y tree as a semantic-diff baseline.",
674+
))
675+
specs.append(CommandSpec(
676+
"AC_screen_diff", "Native UI", "Screen: Diff Snapshots",
677+
description="Semantic diff of 'before'/'after' snapshots (JSON view).",
678+
))
679+
specs.append(CommandSpec(
680+
"AC_screen_changed", "Native UI", "Screen: What Changed",
681+
fields=(app,),
682+
description="Diff the live screen against the last snapshot baseline.",
683+
))
684+
specs.append(CommandSpec(
685+
"AC_describe_screen", "Native UI", "Screen: Describe",
686+
fields=(app,),
687+
description="Structured 'where am I' (role counts + control labels).",
688+
))
665689

666690

667691
def _add_set_of_marks_specs(specs: List[CommandSpec]) -> None:

‎je_auto_control/utils/executor/action_executor.py‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2756,6 +2756,31 @@ def _mark_click(mark_id: int) -> Dict[str, Any]:
27562756
return {"clicked": mark_click(int(mark_id))}
27572757

27582758

2759+
def _screen_snapshot(app_name: Optional[str] = None) -> Dict[str, Any]:
2760+
"""Adapter: snapshot the live a11y tree as a diff baseline."""
2761+
from je_auto_control.utils.screen_state import snapshot_screen
2762+
return {"snapshot": snapshot_screen(app_name=app_name)}
2763+
2764+
2765+
def _screen_diff(before: List[Dict[str, Any]],
2766+
after: List[Dict[str, Any]]) -> Dict[str, Any]:
2767+
"""Adapter: semantic diff between two snapshots."""
2768+
from je_auto_control.utils.screen_state import diff_snapshots
2769+
return diff_snapshots(before, after)
2770+
2771+
2772+
def _screen_changed(app_name: Optional[str] = None) -> Dict[str, Any]:
2773+
"""Adapter: diff the live screen against the last snapshot baseline."""
2774+
from je_auto_control.utils.screen_state import screen_changed
2775+
return screen_changed(app_name=app_name)
2776+
2777+
2778+
def _describe_screen(app_name: Optional[str] = None) -> Dict[str, Any]:
2779+
"""Adapter: structured 'where am I' description of the live screen."""
2780+
from je_auto_control.utils.screen_state import describe_screen
2781+
return describe_screen(app_name=app_name)
2782+
2783+
27592784
class Executor:
27602785
"""
27612786
Executor
@@ -2968,6 +2993,10 @@ def __init__(self):
29682993
"AC_checkpoint_clear": _checkpoint_clear,
29692994
"AC_mark_screen": _mark_screen,
29702995
"AC_mark_click": _mark_click,
2996+
"AC_screen_snapshot": _screen_snapshot,
2997+
"AC_screen_diff": _screen_diff,
2998+
"AC_screen_changed": _screen_changed,
2999+
"AC_describe_screen": _describe_screen,
29713000
"AC_a11y_record_start": _a11y_record_start,
29723001
"AC_a11y_record_stop": _a11y_record_stop,
29733002
"AC_a11y_record_events": _a11y_record_events,

0 commit comments

Comments
 (0)