Docs/Studio/Debugging with Artifacts

Debugging AI Agent Runs with Failure Artifacts

Learn how to use visual failure artifacts to debug your AI agents when traces alone aren't enough.

Overview

Failure artifacts provide cinematic playback of exactly what went wrong when your AI agent encounters assertion failures. Unlike static screenshots, artifacts show the complete sequence of events leading to failure, making it easier to understand complex timing issues, visual bugs, and unexpected page behavior.

Artifacts debugging interface with video player and frame gallery

What Are Failure Artifacts?

Failure Artifacts are visual evidence automatically captured when your AI agent encounters assertion failures. They provide cinematic playback of exactly what went wrong, helping you debug complex issues that static screenshots can't capture.

Available Artifacts

When Artifacts Are Created

Accessing Artifacts

Finding Artifacts in Sentience Studio

On the Trace List Page

Failed trace runs show an artifact indicator when visual evidence is available:

Trace list showing artifact indicator on failed runs

In the Trace Debugger

Once you open a failed trace, artifacts appear in multiple places:

Debugger header showing artifact availability

Opening the Artifacts Panel

Method 1: From the Header

  1. Look for the artifact badge in the run header
  2. Click it to open the artifacts viewer

Method 2: From Failed Steps

  1. On the timeline, find steps marked with red camera icons
  2. Click the camera icon to jump to artifacts for that failure

Method 3: From Detail Panel

  1. Open any failed step's detail panel
  2. Click the "ARTIFACTS" tab

Understanding the Artifacts Interface

When you open artifacts, you'll see a two-panel layout designed for efficient debugging:

Artifacts interface with video player and metadata panel

Left Panel: Visual Evidence

The main area shows your visual debugging tools:

Video Player (when available):

Frame Gallery (fallback or alternative view):

Right Panel: Failure Context

The metadata panel provides essential context:

Failure Summary:

Step Timeline:

Step-by-Step Debugging with Artifacts

Step 1: Identify Failed Runs with Artifacts

Start by finding traces that have visual evidence available:

  1. Look for the artifact badge (🎬) on failed trace cards
  2. Filter by status to focus on failed runs
  3. Check recent runs for the most relevant failures

Step 2: Quick Video Assessment

Get the big picture with cinematic playback:

  1. Open the failed trace in the debugger
  2. Click the artifact badge in the header
  3. Play the video at normal speed (1x) first
  4. Note the exact moment when things go wrong

Pro Tip: Use 0.5x speed for complex sequences where timing matters.

Step 3: Detailed Frame Analysis

Dive deep with pixel-perfect inspection:

  1. Switch to frame gallery for precise analysis
  2. Enable diff overlays to see what changed
  3. Zoom in on error messages or unexpected elements
  4. Compare multiple frames to understand the sequence

When to use frames over video:

Step 4: Correlate with Trace Data

Connect visual evidence with agent behavior:

  1. Use timeline synchronization to link video to trace steps
  2. Check the LLM tab - what was the agent "thinking" at failure?
  3. Review verification signals - which assertions failed?
  4. Compare with action details - did execution match expectations?

Step 5: Identify Root Causes

Use the combined evidence to find the real issue:

Common Failure Patterns

Pattern 1: Wrong Element Selection

Video Evidence: Agent clicks a different button than expected
Trace Correlation: LLM chose element #42, but user wanted #38
Solution: Improve element descriptions or selectors

Pattern 2: Timing Issues

Video Evidence: Action fires before page finishes loading
Trace Correlation: Duration shows 50ms execution, but page needed 2s
Solution: Add proper wait conditions or delays

Pattern 3: Form Validation Problems

Video Evidence: User fills form, clicks submit, error appears
Trace Correlation: Verification expected success message
Solution: Update assertion logic for validation states

Pattern 4: Dynamic Content Changes

Video Evidence: Page layout shifts during interaction
Trace Correlation: Element becomes hidden after scroll
Solution: Use stable selectors or wait for stability

Advanced Artifact Analysis

Video Playback Techniques

Speed Control Strategies:

Timeline Navigation:

Frame Gallery Workflows

Comparative Analysis:

  1. Open two frames side-by-side in different browser tabs
  2. Use diff mode to highlight pixel-level changes
  3. Zoom to 200%+ for detailed element inspection

Pattern Recognition:

  1. Scan through frames chronologically looking for anomalies
  2. Note timestamps of important state changes
  3. Create frame sequences for documentation

Metadata Correlation

Step Timeline Analysis:

Failure Context:

Real-World Debugging Scenarios

Scenario 1: Form Submission Failure

Problem: Agent fills out a form and clicks submit, but the form shows validation errors instead of proceeding.

Artifact Analysis:

  1. Watch the video: See the exact sequence of typing and clicking
  2. Check form fields: Verify all required fields were filled
  3. Look for validation timing: Did errors appear immediately or after submission?
  4. Compare with trace: Check if verification expected the right success indicators

Common Causes:

Scenario 2: Navigation Timing Issues

Problem: Agent clicks a link, but the page doesn't navigate as expected.

Artifact Analysis:

  1. Slow down playback: Use 0.5x to see the click and response
  2. Check for loading states: Look for spinners or "Loading..." text
  3. Watch URL changes: See if navigation starts but gets interrupted
  4. Compare with verification: Check if URL change detection worked

Common Causes:

Scenario 3: Element Interaction Problems

Problem: Agent tries to click a button, but the interaction doesn't work.

Artifact Analysis:

  1. Zoom in on the target element: Verify it's actually clickable
  2. Check element state: Is it disabled, hidden, or covered?
  3. Watch for page changes: Does the element move or change before click?
  4. Compare bounding boxes: Did the agent target the right area?

Common Causes:

Best Practices for Artifact Debugging

Workflow Optimization

  1. Start with video overview - Get the complete story before diving into details
  2. Use appropriate playback speed - Match the analysis needs
  3. Take notes on key moments - Document important timestamps
  4. Correlate with trace data - Don't analyze artifacts in isolation

Team Collaboration

  1. Share specific video segments - Link to exact failure moments
  2. Export key frames - Include in bug reports and documentation
  3. Document findings - Combine artifact evidence with trace analysis
  4. Create reproducible test cases - Use artifacts to understand failure conditions

Performance Considerations

  1. Preload artifacts - Let them load in the background while you work
  2. Use frame gallery for quick checks - Faster than loading full videos
  3. Close unused tabs - Free up memory for large artifact sets
  4. Download for offline analysis - Work with local copies for intensive debugging

Troubleshooting Artifacts

Common Issues

"No artifacts available for this run"

"Video won't play or loads slowly"

"Frames don't show the failure moment"

"Can't find the ARTIFACTS tab"

Performance Tips

Next Steps

Previous
Detail Panel
Next
Troubleshooting