./osman-yigit-sokel

May 18, 2026May, 2026

Screenshot Intelligence — Inspectable AI Interface Analysis Pipeline

Screenshot Intelligence is a stateless multimodal analysis tool that turns UI screenshots into structured interface breakdowns, semantic regions, component inventories, UX feedback, and frontend structure suggestions.

#Ai Systems#Multimodal#Developer Tools
Role
Full-Stack Engineer & System Designer
Stack
Next.js · React · TypeScript · Tailwind CSS · shadcn/ui · OpenAI GPT-4o Vision · Zod · Vercel · Vitest
Links
LiveRepository

Problem

UI screenshots are usually reviewed manually and informally. Developers can describe what they see, but there is no structured way to extract layout sections, components, UX issues, and frontend architecture hints from a single screenshot.

Solution

I built a Next.js-based multimodal pipeline that validates and resizes uploaded screenshots, sends them to GPT-4o Vision, validates the response with Zod, normalizes the output, and renders the result as an inspectable interface with semantic overlays, structured panels, Markdown export, and local session history.

Decisions

  • Used semantic region approximation instead of CV bounding boxes to reduce complexity and instability
  • Kept the system stateless to avoid unnecessary infrastructure and persistence overhead
  • Used a single multimodal request instead of multi-agent orchestration for lower latency
  • Added normalization after Zod validation because schema correctness alone was not enough for stable rendering
  • Stored analysis history in localStorage instead of introducing accounts or databases
  • Focused the UI on inspectability and structure rather than AI-generated visual effects

Architecture highlights

  • Stateless multimodal pipeline built around a single GPT-4o Vision structured request
  • Zod schema validation + normalization layer prevents malformed AI output from reaching the UI
  • Semantic overlay system maps inferred layout regions to interactive frontend sections
  • Client-side image resize reduces payload size before server processing
  • Local-first session history stores recent analyses without backend persistence
  • Object-contain overlay alignment fixes region positioning on portrait screenshots

Outcomes

  • Built a complete screenshot-to-structured-analysis AI pipeline
  • Implemented inspectable semantic overlays synchronized with analysis sections
  • Created a reusable benchmark suite for multimodal UI analysis evaluation
  • Reduced unreliable AI rendering issues through normalization and validation layers
  • Achieved lightweight deployment with zero backend persistence requirements
  • Designed a developer-oriented interface inspection workflow instead of a generic AI demo

Overview

Screenshot Intelligence is a stateless multimodal interface analysis system that converts UI screenshots into structured, inspectable analysis results.

Instead of treating screenshots as static images, the system interprets them as interface structures. A single screenshot is transformed into semantic regions, component inventories, UX observations, and frontend structure summaries through a structured AI pipeline.

The project focuses on reliability and inspectability rather than raw AI output generation. The goal was not to build another “AI screenshot analyzer”, but to create a normalized analysis pipeline that developers can actually inspect, validate, and interact with.

Problem

UI reviews are usually manual and inconsistent.

Developers and designers can visually interpret screenshots, but most workflows still rely on informal feedback like:

  • “spacing feels off”
  • “CTA is weak”
  • “layout looks crowded”

These observations are subjective, difficult to structure, and hard to reuse programmatically.

Most existing AI screenshot tools also suffer from several issues:

  • generic feedback
  • unstructured outputs
  • hallucinated UI observations
  • no validation layer
  • no inspectable relationship between layout regions and analysis results

Additionally, many systems attempt pixel-perfect computer vision bounding boxes, which significantly increases complexity and brittleness for an MVP.

I wanted to explore a different approach: semantic interface understanding instead of precise CV detection.

Solution

I built a stateless analysis pipeline using Next.js API Routes, GPT-4o Vision, Zod validation, and a normalization layer.

The system works as follows:

  1. The user uploads a screenshot
  2. The browser resizes the image client-side
  3. The image is sent to a single multimodal GPT-4o request
  4. GPT output is validated through a strict Zod schema
  5. A normalization layer cleans and stabilizes the result
  6. The normalized structure is rendered through an inspectable UI

The final output includes:

  • screen type classification
  • semantic layout sections
  • component inventories
  • severity-tagged UX feedback
  • frontend structure summaries
  • clickable semantic overlays
  • Markdown export

The system intentionally avoids persistence and backend storage.

All analysis history is stored locally in localStorage with a lightweight session-based workflow.

Architecture

The project follows a fully stateless request pipeline.

Browser
  → upload + resize
  → POST /api/analyze

API Route
  → validate image
  → GPT-4o Vision structured call
  → Zod validation
  → normalization layer
  → normalized JSON response

Browser
  → semantic overlay rendering
  → analysis panels
  → localStorage history

This architecture kept the system lightweight while still allowing structured analysis workflows.

The backend acts as a transformation boundary rather than a persistent service layer.

Semantic Region System

One of the key architectural decisions was avoiding pixel-perfect computer vision overlays.

Instead of attempting exact coordinate detection, the system uses semantic region approximation:

  • top
  • bottom
  • left
  • right
  • center

GPT identifies semantic layout regions, and the frontend maps them into approximate overlay zones aligned to the rendered image bounds.

This approach dramatically reduced complexity while preserving enough structure for meaningful UI inspection.

The result is more stable and interpretable than brittle pseudo-CV bounding boxes.

Structured Output Reliability

A major focus of the project was output stability.

LLM outputs are unreliable when rendered directly into UI systems, so the pipeline includes multiple stabilization layers:

Zod Schema Validation

GPT responses are parsed through strict structured schemas before rendering.

This prevents malformed outputs from reaching the UI layer.

Normalization Layer

After validation, outputs pass through normalization logic that handles:

  • duplicate section cleanup
  • orphan section references
  • fallback generation
  • empty field handling
  • string trimming
  • UI-safe transformations

This converts raw AI output into a stable render model.

UX and Interaction Design

The frontend was designed around inspectability rather than visual novelty.

The interface includes:

  • semantic overlay ↔ section synchronization
  • clickable section navigation
  • structured result panels
  • Markdown export
  • retry flow after analysis failure
  • lightweight session history

The overlay system also handles object-contain image alignment to avoid incorrect positioning on portrait screenshots.

Benchmarking

The project includes a repeatable benchmark suite with multiple UI categories:

  • dashboards
  • analytics interfaces
  • notification systems
  • survey builders
  • CRM panels
  • link management tools

The benchmark runner generates structured JSON outputs and Markdown reports for inspection.

This helped evaluate:

  • semantic section quality
  • UX feedback specificity
  • component extraction consistency
  • response duration stability

Technical Decisions

Why stateless architecture?

The product did not require persistence, accounts, or collaborative workflows.

Removing backend storage significantly simplified deployment, security, and infrastructure complexity.

Why semantic regions instead of CV detection?

True computer vision overlays would require significantly more infrastructure and produce unstable MVP behavior.

Semantic approximation produced more predictable and inspectable results.

Why normalization after Zod?

Schema validation alone guarantees shape correctness, not render quality.

Normalization ensured the UI always received stable, cleaned data structures.

Why single multimodal call?

A single structured GPT-4o request reduced orchestration complexity and improved response latency for the MVP stage.

Outcomes

  • Built a fully stateless multimodal analysis pipeline
  • Created structured semantic UI interpretation workflows
  • Implemented reliable AI output validation + normalization layers
  • Developed an inspectable semantic overlay system
  • Added repeatable benchmark infrastructure for analysis quality evaluation
  • Designed a lightweight local-first session workflow without backend persistence

Reflection

The most important lesson from this project was that AI systems become significantly more useful when treated as structured transformation pipelines instead of raw text generators.

The real engineering challenge was not calling GPT-4o Vision.

It was designing the boundaries around the model: validation, normalization, inspectability, fallback handling, and UI-safe rendering.

This project pushed me toward thinking more carefully about reliability layers in AI-driven systems.

Related projects