som

Warning

Deprecated: cua-som is no longer maintained and will not receive updates or fixes. It is licensed under AGPL-3.0, separately from the MIT-licensed Cua packages.

Som (Set-of-Mark) is a visual grounding component for the Computer-Use Agent (Cua) framework powering Cua, for detecting and analyzing UI elements in screenshots. Optimized for macOS Silicon with Metal Performance Shaders (MPS), it combines YOLO-based icon detection with EasyOCR text recognition to provide comprehensive UI element analysis.

Features

• Optimized for Apple Silicon with MPS acceleration
• Icon detection using YOLO with multi-scale processing
• Text recognition using EasyOCR (GPU-accelerated)
• Automatic hardware detection (MPS → CUDA → CPU)
• Smart detection parameters tuned for UI elements
• Detailed visualization with numbered annotations
• Performance benchmarking tools

System Requirements

• Recommended: macOS with Apple Silicon
  • Uses Metal Performance Shaders (MPS)
  • Multi-scale detection enabled
  • ~0.4s average detection time
• Supported: Any Python 3.11+ environment
  • Falls back to CPU if no GPU available
  • Single-scale detection on CPU
  • ~1.3s average detection time

Installation

# Using PDM (recommended)
pdm install

# Using pip
pip install -e .

Quick Start

from som import OmniParser
from PIL import Image

# Initialize parser
parser = OmniParser()

# Process an image
image = Image.open("screenshot.png")
result = parser.parse(
    image,
    box_threshold=0.3,    # Confidence threshold
    iou_threshold=0.1,    # Overlap threshold
    use_ocr=True         # Enable text detection
)

# Access results
for elem in result.elements:
    if elem.type == "icon":
        print(f"Icon: confidence={elem.confidence:.3f}, bbox={elem.bbox.coordinates}")
    else:  # text
        print(f"Text: '{elem.content}', confidence={elem.confidence:.3f}")

License

cua-som is licensed under AGPL-3.0-or-later. See LICENSE for the complete terms. Its dependencies retain their own licenses. Inspect the exact resolved Ultralytics dependency version and its notices before redistribution.

By default, OmniParser can download model files from Microsoft's separate OmniParser-v2.0 model repository. Those downloaded files are not relicensed by the cua-som package. Inspect the license and provenance published with the exact model revision before use or redistribution.