Visual Overlay API

Visualize detected elements directly in the browser with color-coded borders and importance indicators. Perfect for debugging element detection, understanding importance scoring, and verifying critical UI elements.

Overview

The Visual Overlay API displays a visual overlay highlighting elements in the browser without taking a new snapshot. This is useful for:

• Debugging - See exactly which elements Sentience detects on complex pages
• Analysis - Understand which elements rank highest by importance
• Validation - Verify critical buttons/links are being detected correctly
• Learning - Understand how importance scoring and visual cues work
• Target Highlighting - Show which element the agent will interact with

show_overlay()

Display a visual overlay highlighting elements in the browser with color-coded borders.

Function Signature

from sentience import show_overlay

show_overlay(browser, elements, target_element_id=None)

Parameters

• browser (SentienceBrowser) - Browser instance
• elements - Can be:
  • Snapshot object (will use snapshot.elements)
  • List of Element objects (from snapshot.elements)
  • List of raw element dicts (from API or snapshot result)
• target_element_id (int, optional) - ID of element to highlight in red

Visual Indicators

Color Coding

• 🔴 Red - Target element (when target_element_id is specified)
• 🔵 Blue - Primary elements (is_primary=true)
• 🟢 Green - Regular interactive elements

Additional Indicators

• Border thickness and opacity scale with importance score
• Semi-transparent fill for better visibility
• Importance badges showing numeric scores
• ⭐ Star icon for primary elements
• 🎯 Target emoji for the target element
• Auto-clear - Overlay disappears after 5 seconds

Basic Usage

from sentience import SentienceBrowser, snapshot, show_overlay

browser = SentienceBrowser()
browser.start()
browser.page.goto("https://example.com")

# Take snapshot once
snap = snapshot(browser)

# Show overlay anytime without re-snapshotting
show_overlay(browser, snap)

# Wait to inspect the overlay (auto-clears after 5 seconds)
import time
time.sleep(6)

browser.close()

Highlighting Specific Elements

Show overlay with specific elements or highlight a target element in red:

from sentience import SentienceBrowser, snapshot, show_overlay, find

browser = SentienceBrowser()
browser.start()
browser.page.goto("https://example.com")

# Take snapshot
snap = snapshot(browser)

# Option 1: Show all elements from snapshot
show_overlay(browser, snap)

# Option 2: Show only specific elements
show_overlay(browser, snap.elements)

# Option 3: Highlight a specific target element in red
button = find(snap, "role=button text~'Submit'")
if button:
    show_overlay(browser, snap, target_element_id=button.id)
    # The submit button will be highlighted in red

browser.close()

clear_overlay()

Manually clear the visual overlay before the 5-second auto-clear.

Function Signature

from sentience import clear_overlay

clear_overlay(browser)

Parameters

• browser (SentienceBrowser) - Browser instance

Usage

from sentience import SentienceBrowser, snapshot, show_overlay, clear_overlay
import time

browser = SentienceBrowser()
browser.start()
browser.page.goto("https://example.com")

snap = snapshot(browser)

# Show overlay
show_overlay(browser, snap)

# Do some analysis for 2 seconds
time.sleep(2)

# Clear immediately when done (before 5-second auto-clear)
clear_overlay(browser)

browser.close()

Using with Snapshot Options

You can also show overlays automatically when taking snapshots:

from sentience import SentienceBrowser, snapshot, SnapshotOptions
import time

browser = SentienceBrowser()
browser.start()
browser.page.goto("https://example.com")

# Automatically show overlay during snapshot
snap = snapshot(browser, SnapshotOptions(show_overlay=True))

# Overlay is displayed immediately
# Wait 6 seconds to inspect (overlay auto-clears after 5 seconds)
time.sleep(6)

browser.close()

Debugging Example

Use visual overlay to debug why an element isn't being detected:

from sentience import SentienceBrowser, snapshot, show_overlay, find, clear_overlay
import time

browser = SentienceBrowser()
browser.start()
browser.page.goto("https://example.com")

# Show overlay to see what's detected
snap = snapshot(browser)
show_overlay(browser, snap)
print(f"Found {len(snap.elements)} elements")

# Wait to inspect the overlay
time.sleep(6)

# Try to find a specific button
button = find(snap, "role=button text~'Submit'")
if not button:
    print("❌ Button not found - check the overlay to see what's detected")
    print("Top 5 elements by importance:")
    for i, elem in enumerate(snap.elements[:5]):
        print(f"  {i+1}. [{elem.id}] {elem.role}: {elem.text[:50]} (importance: {elem.importance})")
else:
    print(f"✅ Found button: {button.text}")
    # Highlight the target button in red
    show_overlay(browser, snap, target_element_id=button.id)
    time.sleep(3)
    clear_overlay(browser)

browser.close()

Use Cases

1. Debugging Element Detection

When an element isn't found, use overlay to see what Sentience detects:

# Check if critical buttons are detected
snap = snapshot(browser)
show_overlay(browser, snap)
time.sleep(6)

# Verify login button is present
login_btn = find(snap, "role=button text~'login'")
if not login_btn:
    print("⚠️ Login button not detected - check overlay!")

2. Understanding Importance Scoring

See which elements rank highest on a complex page:

# Show overlay with importance badges
snap = snapshot(browser)
show_overlay(browser, snap)

# Print top 10 elements by importance
for elem in snap.elements[:10]:
    print(f"{elem.importance}: {elem.role} - {elem.text[:40]}")

3. Validating Agent Actions

Highlight which element the agent will interact with:

# Find target element
snap = snapshot(browser)
target = find(snap, "role=button text~'Add to Cart'")

# Show what the agent sees (target in red)
show_overlay(browser, snap, target_element_id=target.id)
time.sleep(3)

# Proceed with action
click(browser, target.id)

4. Visual QA Testing

Verify critical UI elements are always detected:

# List of critical elements to verify
critical_elements = [
    "role=button text~'Checkout'",
    "role=link text~'Cart'",
    "role=textbox text~'Search'"
]

snap = snapshot(browser)
show_overlay(browser, snap)

for selector in critical_elements:
    elem = find(snap, selector)
    if not elem:
        print(f"❌ MISSING: {selector}")
    else:
        print(f"✅ Found: {elem.text}")

Pro Tips

1. Use show_overlay() separately when you want to re-visualize elements without re-snapshotting
2. Combine with find() or query() to highlight specific filtered elements
3. Perfect for debugging when elements aren't being detected as expected
4. Check importance scores via the overlay badges to understand element ranking
5. Use in headless mode by setting headless=False temporarily for debugging

Related APIs

• Snapshot API - Capture page state with show_overlay=True option
• Query API - Find elements to highlight with overlay
• Tracing & Debugging - Other debugging techniques

Next Steps

• Expect API → - Testing and assertions
• Snapshot API → - Capturing page state
• Examples → - More debugging examples