'Make It Real' AI Prototype Turns Drawings Into Working Software

An anonymous reader quotes a report from Ars Technica: On Wednesday, a collaborative whiteboard app maker called 'tldraw' made waves online by releasing a prototype of a feature called 'Make it Real' that lets users draw an image of software and bring it to life using AI. The feature uses OpenAI's GPT-4V API to visually interpret a vector drawing into functioning Tailwind CSS and JavaScript web code that can replicate user interfaces or even create simple implementations of games like Breakout. 'I think I need to go lie down,' posted designer Kevin Cannon at the start of a viral X thread that featured the creation of functioning sliders that rotate objects on screen, an interface for changing object colors, and a working game of tic-tac-toe. Soon, others followed with demonstrations of drawing a clone of Breakout, creating a working dial clock that ticks, drawing the snake game, making a Pong game, interpreting a visual state chart, and much more.

Tldraw, developed by Steve Ruiz in London, is an open source collaborative whiteboard tool. It offers a basic infinite canvas for drawing, text, and media without requiring a login. Launched in 2021, the project received $2.7 million in seed funding and is supported by GitHub sponsors. When The GPT-4V API launched recently, Ruiz integrated a design prototype called 'draw-a-ui' created by Sawyer Hood to bring the AI-powered functionality into tldraw. GPT-4V is a version of OpenAI's large language model that can interpret visual images and use them as prompts. As AI expert Simon Willison explains on X, Make it Real works by 'generating a base64 encoded PNG of the drawn components, then passing that to GPT-4 Vision' with a system prompt and instructions to turn the image into a file using Tailwind. You can experiment with a live demo of Make It Real online. However, running it requires providing an API key from OpenAI, which is a security risk.

Read more of this story at Slashdot.