The Lossy Format Translation Problem
TRIGGER
When providing visual design context to LLMs via screenshots, the model had to extrapolate and interpret images—spending compute on visual parsing rather than design generation, and introducing interpretation errors in the translation from pixels to structure.
APPROACH
Figma designer and advocate Greg Huntoon demonstrates how providing Figma frames instead of screenshots dramatically improves AI generation quality in Figma Make. Input: structured design data from Figma frames containing component hierarchy, properties, and relationships. Output: generated designs that build on the existing structure without needing to reverse-engineer it from pixels. When designers upload a screenshot, the model must extrapolate and interpret the image, spending compute on visual parsing rather than design generation and introducing interpretation errors. By providing a Figma frame instead, users pass well-structured design data containing component hierarchy, properties, and relationships that the model intuitively understands. Huntoon shows the difference using a recipe app example: an unstructured prompt yields results barely above a wireframe, while the same task with structured context via TC-EBC framework (Task, Context, Elements, Behavior, Constraints) produces richer, more visually appealing designs with better scaffolding to build on.
PATTERN
“Don't flatten to images then ask the model to reconstruct what you already had. Every format translation is lossy—when your source data has structure, preserve it through the pipeline.”
✓ WORKS WHEN
- Source data exists in structured format (design files, code, schemas) that can be serialized for LLM consumption
- The structure is semantically meaningful for the generation task (component hierarchies for UI generation)
- Model training included the structured format or can parse it reliably
- Preserving structure reduces ambiguity in what the model should generate vs. interpret
✗ FAILS WHEN
- Source is genuinely unstructured (hand-drawn sketches, whiteboard photos) where screenshot is the only option
- Structured export loses visual details that matter (exact spacing, visual polish, brand-specific rendering)
- Model performs better with visual input for the specific task (style transfer, visual similarity matching)
- Structured format serialization is verbose and consumes more tokens than image embedding