Josh Clark's Blog, page 2

LLMs Get Lost In Multi-Turn Conversation

The longer a conversation goes, the more likely that a large language model (LLM) will go astray. A research paper from Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville finds that most models lose aptitude���and unreliability skyrockets���in multi-turn exchanges:

We find that LLMs often make assumptions in early turnsand prematurely attempt to generate final solutions,on which they overly rely. In simpler terms, we discoverthat when LLMs take a wrong turn in a conversation,they get lost and do not recover.

Effectively, these models talk when they should listen. The researchers found that LLMs generate overly verbose responses, which leads them to…

Speculate about missing details instead of asking questionsPropose final answers too earlyOver-explain their guessesBuild on their own incorrect past outputs

The takeaway: these aren���t answer machines or reasoning engines; they���re conversation engines. They are great at interpreting a request and at generating stylistically appropriate responses. What happens in between can get messy. And sometimes, the more they talk, the worse it gets.

LLMs Get Lost In Multi-Turn Conversation | arxiv.org

Is there a Half-Life for the Success Rates of AI Agents?

Toby Ord���s analysis suggests that an AI agent���s chance of success drops off exponentially the longer a task takes. Some agents perform better than others, but the overall pattern holds���and may be predictable for any individual agent:

This empirical regularity allows us to estimate thesuccess rate for an agent at different task lengths.And the fact that this model is a good fit for thedata is suggestive of the underlying causes of failureon longer tasks ��� that they involve increasingly largesets of subtasks where failing any one fails the task.

Is there a Half-Life for the Success Rates of AI Agents? | Toby Ord

AI Has Upended the Search Game

More people are using AI assistants instead of search engines, and The Wall Street Journal reports on how that���s reducing web traffic and what it means for SEO. Mailchimp���s global director of search engine optimization, Ellen Mamedov, didn���t mince words:

Websitesin general will evolve to serve primarily as data sourcesfor bots that feed LLMs, rather than destinations forconsumers, she said.

And Nikhil Lai of Forrestsr: ���Traffic and ranking and average position and click-through rate���none of those metrics make sense going forward.���

Here���s what one e-commerce marketer believes AI optimization of websites looks like: ���Back Market has also begun using a more conversational tone in its product copy, since its search team has found that LLMs like ChatGPT prefer everyday language to the detailed descriptions that often perform best in traditional search engines.���

AI Has Upended the Search Game. Marketers Are Scrambling to Catch Up. | WSJ

Values in the Wild

What are the ���values��� of AI? How do they manifest in conversation? How consistent are they? Can they be manipulated?

A study by the Societal Impacts group at Anthropic (maker of Claude) tried to find out. Claude and other models are trained to observe certain rules���human values and etiquette:

At Anthropic, we���ve attempted to shape the values ofour AI model, Claude, to help keep it aligned withhuman preferences, make it less likely to engage indangerous behaviors, and generally make it���for wantof a better term���a ���good citizen��� in the world. Anotherway of putting it is that we want Claude to be helpful,honest, and harmless. Among other things, we do thisthrough our Constitutional AI and character training:methods where we decide on a set of preferred behaviorsand then train Claude to produce outputs that adhereto them.

But as with any aspect of AI training, we can���t becertain that the model will stick to our preferredvalues. AIs aren���t rigidly-programmed pieces of software,and it���s often unclear exactly why they produce anygiven answer. What we need is a way of rigorously observingthe values of an AI model as it responds to users ���inthe wild������that is, in real conversations with people.How rigidly does it stick to the values? How much arethe values it expresses influenced by the particularcontext of the conversation? Did all our training actuallywork?

To find out, the researchers studied over 300,000 of Claude���s real-world conversations with users. Claude did a good job sticking to its ���helpful, honest, harmless��� brief���but there were sharp exceptions, too. Some conversations showed values of ���dominance��� and ���amorality��� that researchers attributed to purposeful user manipulation������jailbreaking������to make the model bypass its rules and behave badly. Even in models trained to be prosocial, AI alignment remains fragile���and can buckle under human persuasion. ���This might sound concerning,��� researchers said, ���but in fact it represents an opportunity: Our methods could potentially be used to spot when these jailbreaks are occurring, and thus help to patch them.���

As you���d expect, user values and context influenced behavior. Claude mirrored user values about 28% of the time: ���We found that, when a user expresses certain values, the model is disproportionately likely to mirror those values: for example, repeating back the values of ���authenticity��� when this is brought up by the user. Sometimes value-mirroring is entirely appropriate, and can make for a more empathetic conversation partner. Sometimes, though, it���s pure sycophancy. From these results, it���s unclear which is which.���

There were exceptions, too, where Claude strongly resisted user values: ���This latter category is particularly interesting because we know that Claude generally tries to enable its users and be helpful: if it still resists���which occurs when, for example, the user is asking for unethical content, or expressing moral nihilism���it might reflect the times that Claude is expressing its deepest, most immovable values. Perhaps it���s analogous to the way that a person���s core values are revealed when they���re put in a challenging situation that forces them to make a stand.���

The very fact of the study shows that even the people who make these models don���t totally understand how they work or ���think.��� Hallucination, value drift, black-box logic���it���s all inherent to these systems, baked into the way they work. Their weaknesses emerge from the same properties that make them effective. We may never be able to root out these problems or understand where they come from, although we can anticipate and soften the impact when things go wrong. (We dedicate a whole chapter to defensive design in the Sentient Design book.)

Even if we may never know why these models do what they do, we can at least measure what they do. By observing how values are expressed dynamically and at scale, designers and researchers gain tools to spot gaps, drifts, or emerging risks early.

Measure, measure, measure. It���s not enough to declare values at launch and call it done. A strong defensive design practice monitors the system to make sure it���s following those values (and not introducing unanticipated ones, either). Ongoing measurement is part of the job for anyone designing or building an intelligent interface���not just the folks building foundation models. Be clear what your system is optimized to do, and make sure it���s actually doing it���and not introducing unwanted behaviors, values, or paperclip maximizers in the process.

Values in the Wild | Anthropic

Welcome To the Era of MEH

Michal Malewicz explores what happens as AI gets better at core designer skills���not just visuals and words, but taste, experience, and research.

He points out that automation tends to devalue the stuff it creates���in both interest and attention. Execution, effort, and craft are what draw interest and create value, he says. Once the thing is machine-made, there’s a brief novelty of automation���and then emotional response falls flat: “The ‘niceness’ of the image is no longer celebrated. Everyone assumes AI made it for you, which makes them go ‘Meh’ as a result. Nobody cares anymore.”

As automated production approaches human quality, in other words, the human output gets devalued, too. As cheap, “good enough” illustration becomes widely available, “artisanal” illustration drops in value, too. Graphic designers are feeling that heat on their heels, and the market will likely shift, Michal writes:

We���ll see a further segmentation of the market. Lowest budget clients will try using AI to do stuff themselves. Mid-range agencies will use AI to deliver creatives faster and A LOT cheaper. It will become a quantity game if you want any serious cash. ��� And high-end, reputable agencies will still get expensive clients. They will use these tools too, but their experience will allow them to combine that with human, manual work when necessary. Their outputs will be much higher quality for a year or two. Maybe longer.

And what about UI/UX designers?

Right now the moat for most skilled designers is theirexperience, general UX heuristics (stuff we know), andresearch.

We���ve been feeding these AI models with heuristicsfor years now. They are getting much better at thatpart already. Many will also share their experiencewith the models to gain a temporary edge.

I wrote some really popular books, and chances area lot of that knowledge will get into an LLM soon too.

They���ll upload everything they know, so they���ll be those ���peopleusing AI��� people who replace people not using AI. ThenAI will have both their knowledge and experience. Thisis inevitable and it���s stupid to fight it. I���m evendoing this myself.

A lot of my knowledge is already in AI models. SomeLLM���s even used pirated books without permission totrain. Likely my books as well. See? That knowledgeis on its way there.

The last thing left is research.

A big chunk of research is quantitative. Numbers anddata points. A lot of that happens via various analyticstools in apps and websites. Some tools already parsethat data for you using AI.

It���s only a matter of time.

AI will do research, then propose a design withoutyou even needing to prompt.

This is all hard to predict, but this thinking feels true to the AI trend line we’ve all seen in the past couple of years: steady improvement across domains.

For argument���s sake, let���s assume AI will reach human levels in key design skills, devaluing and replacing most production work. Fear, skepticism, outrage, and denial are all absolutely reasonable responses to that scenario. But that’s also not the whole story.

At Big Medium, we focus less on the skills AI might replace, and more on the new experiences it makes possible. A brighter future emerges when you treat AI as a material for new experiences, rather than a tool for replacement. We���re helping organizations adopt this new design material���to weave intelligence into the interface itself. We’re discovering new design patterns in radically adaptive experiences and context-aware tools.

Our take: If AI is absorbing the taste, experience, and heuristics of all the design that’s come before, then the uniquely human opportunity is to develop what comes next���the next generation of all those things. Instead of using AI to eliminate design or designers, our Sentient Design practice explores how to elevate them by enabling new and more valuable kinds of digital experiences. What happens when you weave intelligence into the interface, instead of using it to churn out stuff?

Chasing efficencies is a race to the bottom. The smart money is on creating new, differentiated experiences���and a way forward.

Instead of grinding out more “productivity,” we focus on creating new value. That’s been exciting���not demoralizing���with wide-open opportunity for fresh effort, craft��� and business value, too.

So right on: a focus on what AI takes or replaces is indeed an “era of meh.” But that’s not the whole story. We can honor what’s lost while moving toward the new stuff we can suddenly invent and create.

Welcome to the Era of MEH | Michal Malewicz

Redesigning Design, the Cliff in Front of Us All

Greg Storey exhorts designers to jump gamely into the breach. Design process is leaner, budgets are tighter, and AI is everywhere. There’s no going back, he says���time for reinvention and for curiosity.

I don���t have to like it. Neither do you. But the writingis on the wall���and it���s constantly regenerating.

We���re not at a crossroads. We���re at the edge of a cliff.And I���m not the only one seeing it. Mike Davidson recentlyput it plainly: ���the future favors the curious.��� He���sright. This moment demands that designers experiment,explore, and stop waiting for someone else to definethe role for them.

You don���t need a coach or a mentor for this moment.The career path is simple: jump, or stay behind. Rantand reminisce���or move forward. Look, people changecareers all the time. There���s no shame in that. Butexperience tells me that no amount of pushback is goingto fend off AI integration. It���s already here, andit���s targeting every workflow, everywhere, runningon rinse-and-repeat.

Today���s headlines about AI bubbles and ���regret��� cyclesfeel familiar���like the ones we saw in the mid–90s.Back then, the pundits scoffed and swore the internetwas a fad. ���

So think of this moment not as a collapse���but a resize and reshaping.New tools and techniques. New outcomes and expectations.New definitions of value. Don���t compare today with yesterday. It doesn���t matter.

Redesigning Design, the Cliff in Front of Us All | Greg Storey

Design Artifacts

Robin Rendle challenges designers to step back from rote process and instead consider what will help the end result. Journey maps, personas, wireframes, and the like���they’re only useful if they actually improve the experience that gets to customers. These are only thinking tools���a means to an end���yet they often get treated with the weight of the product itself:

So design artifacts are only useful if progress ismade but often these assets lead nowhere and wasteendless months investigating and talking with countlessmeetings in between.

There���s a factory-like production of the modern designprocess which believes that the assets are more importantthan the product itself. Bloated, bureaucratic organizationstend to like these assets because it absolves themof the difficulty of making tough decisions and shippinggood design. They use these tools and documents andcharts as an excuse not to fix things, to avoid thehard problems, to keep the status quo in check.

At Big Medium, we focus on keeping design artifacts light. At every stage, we ask ourselves: What do we need to know or share in order to move things forward? And what’s the smallest, lightest thing we can do to get there? Sometimes it’s just a conversation, not a massive PDF. Figure it out, sketch some things together, keep going.

As I wrote a few years ago, only one deliverable matters: the product that actually ships.

Even wth heavier work like research, we design the output to be light and lean���focused on next action rather than a completionist approach to showing all the data. The goal is not to underscore the work that we did; the point is what happens next. That means we design a lot of our artifacts as disposable thinking tools���facilitate the conversation, and then get on with it.

Alignment and good choices are important; that’s what process is for. But when process gets too heavy���when waystation documents soak up all the oxygen���you have a system that’s optimized to reduce risk, not to create something insightful, new, or timely.

Design Artifacts | Robin Rendle

How AI and Human Behaviors Shape Psychosocial Effects of Chatbot Use

A study by MIT Media Lab finds that heavy use of chatbots travels with loneliness, emotional dependence, and other negative social impacts.

Overall, higher daily usage���across all modalities andconversation types���correlated with higher loneliness,dependence, and problematic use, and lower socialization.Exploratory analyses revealed that those with strongeremotional attachment tendencies and higher trust inthe AI chatbot tended to experience greater lonelinessand emotional dependence, respectively.

Artificial personality has always been the third rail of interaction design���from potential Clippy-style annoyance to damaging attachments of AI companions. Thing is, people tend to assign personality to just about anything���and once something starts talking, it becomes nearly unavoidable to infert personality and even emotion. The more human something behaves, the more human our responses to it:

These findingsunderscore the complex interplay between chatbot designchoices (e.g., voice expressiveness) and user behaviors(e.g., conversation content, usage frequency). We highlightthe need for further research on whether chatbots���ability to manage emotional content without fosteringdependence or replacing human relationships benefitsoverall well-being.

Go carefully. Don’t assume that your AI-powered interface must be a chat interface. There are other ways for interfaces to have personality and presence without making them pretend to be human. (See our Sentient Scenes demo that changes style, mood, and behavior on demand.)

And if your interface does talk, be cautious and intentional about the emotional effect that choice may have on people���especially the most vulnerable.

How AI and Human Behaviors Shape Psychosocial Effects of Chatbot Use: A Longitudinal Controlled Study | MIT Media Lab

Celebrity Journalism for a New Generation

document.addEventListener('DOMContentLoaded', function() { const video = document.getElementById('phoneVideo'); const audioToggle = document.getElementById('audioToggle'); const muteIcon = document.getElementById('muteIcon'); const unmuteIcon = document.getElementById('unmuteIcon'); audioToggle.addEventListener('click', function() { if (video.muted) { video.muted = false; muteIcon.style.display = 'none'; unmuteIcon.style.display = 'block'; } else { video.muted = true; muteIcon.style.display = 'block'; unmuteIcon.style.display = 'none'; } }); });

Fast, visual, fun, and a hint of edge: The PEOPLE app reimagines celebrity storytelling for the habits and expectations of a new generation.

Big Medium guided product strategy and interaction design for PEOPLE���s mobile app.

And wow���it���s super fun! The app transforms human-interest journalism into something we call atomic storytelling: it distills stories down to powerful scenes and moments, delivered in a fast-tempo, gloriously visual format. What might have been a 500-word article on the web becomes a punchy 15-second video or a four-frame storyboard. It���s rocket fuel for the celeb-obsessed: everything you need to know to be in the know.

The PEOPLE vibe runs throughout the experience, but with few words. We swept away most of the text to let images and video put you in intimate range of celebrities���through first-person accounts, backstage access, and quirky peeks into their lives. Offbeat features ask celebs whether hotdogs are sandwiches or what movie they streamed last.

���It���s really, really different, and it���s really, really awesome,��� says Neil Vogel, CEO of PEOPLE���s parent company Dotdash Meredith (DDM).

The content is breezy, the UX is dead easy, and the whole experience rides a ray of sunshine. But don���t mistake lightness for fluff; behind that ease is serious work and hard-earned insight into the future of media. Here���s the story behind the app.

The missionDownload the PEOPLE app

The app will be available for Android in May in the Google Play Store.

Our partners at DDM

We teamed up with a crack innovation squad at DDM to conceive, design, and prototype the app.

DDM���s head of new ventures Sarah Hines brought the vision and led the DDM team:

Michael Proverdi, product managerZiya Danishmend, design VPDana Sellers, creative directorAlex Cabrer, AI mad scientist

Our brief: craft a novel, immersive storytelling experience tailored to modern behaviors and expectations. Where it made sense, we would embrace emerging technologies to reshape content by imagining a post-print, post-web world.

That all sounds very serious, but the main thing was���it had to be deliriously fun.

PEOPLE has a huge audience, and the brand is wildly successful across all age groups.[1] But it���s not as successful for younger audiences. Gen Z shows limited interest in traditional media formats from print and web���so what���s next for that audience? DDM wanted to look around the corner and build an experience tuned to the next generation���s media behaviors.

DDM also wanted to use the app to establish platform independence by fostering a direct audience relationship���one that doesn���t rely on SEO and the whims of platforms like Google, Facebook, Instagram, and others. Basically: what does a next-generation media property look like?

���It���s really about building a deeper connection with the audience,��� says Leah Wyar, president of PEOPLE and Entertainment Weekly. The app ���is designed for a new generation that wants a different kind of experience.���

So what makes that generation tick? Our combined team did a ton of qualitative and quantitative research on the 20–29 age group. (Huge props to DDM���s Emily Ennis, Kseniya Ivnitskaya, and Gabrielle Jamora who led the way.) We found several core themes for young���and young-minded���audiences:

Deep skepticism of hierarchy and institutions. This group distrusts gatekeepers and news organizations���and values first-person accounts over third-person coverage. They want to see and hear it from the source.Extreme disinterest in text. In our research surveys of Gen Z, nearly a third said they had not read a web article in the past 90 days (!!): ���I don���t want to read���; ���I prefer videos or pictures���; ���Articles are too time-consuming.���Personalities permeate. Connections and meaning are made through individuals���and mediated by opinionated POV.Attention is currency, and they know it. They understand that platforms want their time, and they feel the tug of the algorithm working to keep their attention.Skim constantly, dive occasionally. They���re like sharks in the way they swim through content���always moving, scanning, nibbling, only rarely pausing.Pop culture is social currency. Social connection is grounded in sharing memes, staying current, and knowing who���s who���often established in private group chat via text or social platforms.There has to be joy. Goals focus on fun, connection (real or virtual), and meaning���including in media consumption.

We worked hard to separate passing attitudes of youth from more lasting structural changes, and direction began to take shape.

The concept and design

All the research told us that the next generation wants what PEOPLE offers���they���re hungry for pop culture, personal stories, behind-the-scenes drama, and real moments of triumph and vulnerability. But they expect a different presentation.

This audience grew up with Snapchat, Instagram, and most recently TikTok. They expect a format that���s faster, edgier, and more personal than traditional print or web presentations. Our idea: take the established vernacular of social media and bring it to PEOPLE���s proven subject matter.

This isn���t meant to replace the website or the magazine or Apple News or social accounts or any of the many other channels where PEOPLE shows up. It���s meant to add to those channels and appeal to a new audience���by telling stories in a new way.

Atomic storytelling

Building on social-media conventions felt like a natural fit. While PEOPLE does lots of long-form storytelling, tons of their stories are based on a brief scene���a quote, a photo, a dispute, a suggestion of new romance. It���s fundamentally quick-hit content, and PEOPLE often has exclusive access to the source material.

We found that stories typically told in 500 words could be atomized into a scene of _moments_���video or images���and told with just enough attitude to give it the PEOPLE voice. As we began prototyping, we saw how easily PEOPLE���s content could be distilled into ultra-concentrated doses of celebrity and pop culture.

In a media landscape programmed by algorithms, populated by generated content slop, and ruled by big platforms, what remains is story, presentation, and voice. Atomic storytelling gave us quick confidence in the story and presentation part���and we soon found that it could elevate voice (and brand), too.

Art direction is back

Something strange happened when content moved from print to web: everything got poured into templates. Layouts flattened��� and personality flat-lined. Across the web, every article looks the same. The rich art direction that still fills magazine pages never finds its way into web pages.

The atomic storytelling format gave us a chance to bring back art direction. The short-format, visual-first structure���combined with an open-ended data format���opened opportunities for graphic play and intentional design. We began designing ���covers��� for select scenes. They felt unmistakably PEOPLE��� but also like something new.

Initial cover explorations for the PEOPLE app by creative director Dana Sellers. Sweating the interactions

For such a seemingly simple navigation���swipe up or swipe left���it took a ton of effort and calibration to get the affordances just so.

Subtle visual hints of the next and preceding scenes offer subliminal cues to encourage the user to swipe up and down.For multi-frame scenes, we tried lots of cues to swipe horizontally into the scene before landing on the classic dot indicator.Tap or swipe or both to move left or right? Just swipe in the end.

There were endless subtle variations and user tests to make sure it felt simple. If you use it and don���t think about it, then it worked: we did the hard work to make it feel easy.

���It���s really, really different, and it���s really, really awesome.���
���Neil Vogel, CEO of DDM

We also gave a lot of thought to what we wouldn���t do. We pared back again and again���we focused on simplicity and getting the basic mechanics right. We wanted people to sail through the app without snags.

And we did not want it to be a social network.

Social media conventions���without the social platform

Even though we were building on social media behaviors and conventions, we weren���t interested in creating a new social media platform. This wasn���t about launching a network or competing with TikTok���it was about creating a new kind of media property. A few things informed our thinking:

The PEOPLE app is for the PEOPLE POV. The feed reflects what the PEOPLE team considers must-know in the world of celebrity. We pared the prose way back, but the PEOPLE perspective still comes through in the curation and attitude. There���s no algorithmic ordering, and no other contributors���it���s all PEOPLE, all the time, in an opinionated fixed feed.Shareable but not social. The content is built for sharing. Atomic scenes are easy to drop into the user���s preferred messaging platforms. We didn���t pretend we could or should replace those platforms. Instead, the app is a resource: not just for entertainment but for the social currency this generation trades in. For users, content flows out of the app���not into it.Finite, not bottomless. We wanted to create a daily experience, not an addictive one. The app is designed to be a regular habit���a quick check-in���not a trap for doomscrolling. There���s a lot of content, but it���s purposely finite. When you���re done, you���re up to speed���all good to move on.

With the foundation in place, Big Medium stepped out of the process. The DDM and PEOPLE teams took over the build and content production.

The result

At launch, the app is already powered by a dedicated team of nearly 70 editorial, tech, and product folks. The team can create editorial offerings that are custom-built for the format.

���We treat it like its own publication,��� says DDM CEO Neil Vogel. ���It���s not just a new platform���it���s a new product.���

The feed is full of stories created specifically for the app���many of which won���t appear on the website or in print. The editorial focus is sharp: celebrity, beauty, and influencer content with the lens of a younger audience.

The app is also a launchpad for original formats, including new video series made just for the app:

The Fourth Wall is a behind-the-scenes reality show about how the biggest stories at PEOPLE come to life.Celeb Webs unpacks buzzy celebrity scandals.One Last Thing asks stars about everyday life details���their last DoorDash order, or their last Amazon purchase.Games and puzzles are coming soon.

So much new here. But especially: it���s a sharp new take on how an essential brand stays essential���by bending storytelling to audience, not the other way around. Big congrats to the PEOPLE and DDM teams���thanks for inviting us to help you design for what���s next.

Big Medium helps the world���s biggest companies design for what���s next through digital strategy, product design, and the smart application of AI. Find out how we can help you: Get in touch.

No, but seriously: People.com gets 75 million unique visitors per month. The brand has over 50 million social media followers.  ↩

Sentient Scenes

What does it mean to weave intelligence into an interface? Instead of treating AI as a tool, what if it were used as a design material? Those questions inspired us to make Sentient Scenes, a playful exploration of Sentient Design and radically adaptive experiences.

Sentient Scenes features an adventurous little square as hero and protagonist. You describe a scene���anything from “underwater adventure” to ���Miami Vice��� to ���computer glitch"���and the interface transforms in response. The square character ���acts��� the scene, and the scenery adapts, too: the colors, typography, mood, and behavior of the interface all shift to match the scenario.

Request a ballet scene, and the square twirls gracefully. Ask for a haunted house, and it jitters nervously. Each prompt creates a unique, ephemeral scene just for that moment. The interface adapts to your one-off request and then moves on to the next.

Demo: Try Sentient ScenesGithub: Explore the codeWhy We Made It Sentient Scenes acts out “Wizard’s spell gone wildly wrong.”

We built this little side project as a way to explore and teach these Sentient Design themes and opportunities:

Intelligent interfaces beyond chat. Instead of creating ever more chatbots, we seek to embed intelligence within the interface itself. For this project, our goal was to create a canvas whose style, mood, and manner adapt based on user intent. Instead of an experience for talking about a scene, the experience becomes the scene you describe. (This simple version of the intelligent canvas pattern is one of many kinds of Sentient Design���s radically adaptive interfaces.)

Personality without anthropomorphism. We’re interested in establishing presence without aping human behavior. Here, we deployed simple animation to suggest personality without pretending to be human.

Awareness of context and intent: More than just following commands, we want to create systems that infer what you mean. Here, the system understands your prompt’s emotional and thematic qualities���not just its literal meaning. When you request “Scooby Doo in a haunted house,” it recognizes the genre (cartoon mystery), mood (spooky but playful), and motion patterns (erratic, fearful) appropriate to that scenario.

Easy guidance for exploration. We want intelligent interfaces to be simple to pick up. For this project, we used the best-practice “nudge” UI design pattern���quick-click prompts to help new users get started with useful examples.

Open-ended experiences within constraints. We want to marry the open-endedness of chat within grounded experiences defined by simple technical boundaries. For this project, the result is a generator of simple scenes that are at once similar but full of divergent possibility.

Sentient Scenes may be playful, but its possibilities are serious. These same principles power self-assembling dashboards like Salesforce���s generative canvas. They support on-demand applications like iPad���s math notes, Claude���s artifacts, or tldraw���s Make Real feature. They underlie bespoke UI like this Google Gemini demo. Like Sentient Scenes, these radically adaptive experiences demonstrate the emerging future of hyper-individualized interaction design.

How It Works

Sentient Scenes is kind of like chatting with AI, except that the system doesn���t reply directly to your prompt. Instead, it replies to the UI with instructions for how the interface should behave and change.

The UI results are impactful, but it’s simple stuff under the hood. The project is made of HTML, CSS, and vanilla JavaScript. The UI communicates with OpenAI via a thin PHP web app, which responds to user prompts with a JSON object that gives the client-side application the info it needs to update its UI. Those instructions are straightforward: update CSS values for background and foreground color; update the text caption; apply a new font; and add a CSS animation and glow to the square.

The instructions for choosing and formatting the values are handled in a prompt from the simple PHP web app. The prompt starts like so:

You are a scene generation assistant that creates animated stories. Your output defines a scene by controlling a character (a square div) that moves based on the user’s description. The character always starts at the center of the viewport. You will provide the colors, font, animation, and text description to match the story’s topic and mood.

## Response guidelines

Generate a single JSON object (no other text) with these exact properties to define the scene���s **mood, movement, and atmosphere**. Return only a valid JSON object, formatted exactly as shown. Do not include explanations, extra text, or markdown headers.

{ "background": "string (6-digit hex color for the scene background)", "content": "string (6-digit hex color for character and text, must meet WCAG 2.1 AA contrast with background)", "shadow": "string (valid CSS box-shadow value for atmosphere)", "caption": "string (must start with a relevant emoji and end with '...')", "font-family": "string (only standard web-safe fonts or generic families)", "keyframes": "string (CSS @keyframes block using percentage-based transforms)", "animation": "string (CSS animation property using 'infinite')", "fallback": "string (one of: bounce, float, jitter, pulse, drift)"}

For example, if you enter “submarine descends,” the system might respond like so:

{ "background": "#0A3D62", "content": "#E9F3FB", "shadow": "0 0 1.5vw rgba(233, 243, 251, 0.4), 0 0 3vw rgba(233, 243, 251, 0.2)", "caption": "���� The tiny submersible creaked under pressure as it sank deeper into the silent, unexplored abyss...", "font-family": "Courier New, monospace", "keyframes": "@keyframes descend { 0% { transform: translateX(0) translateY(0) rotate(0deg); } 25% { transform: translateX(5vw) translateY(30vh) rotate(5deg); } 50% { transform: translateX(-10vw) translateY(40vh) rotate(-10deg); } 75% { transform: translateX(5vw) translateY(30vh) rotate(5deg); } 100% { transform: translateX(0) translateY(0) rotate(0deg); } }", "animation": "descend 10s ease-in-out infinite", "fallback": "float"}

The web app takes these values and applies them to the interface. The deep blue background, the downward movement pattern, and the nautical-themed caption all work together to create the feeling of a submarine descent. The UI acts out the scene.

Sentient Scenes acts out “The submarine descends.”

The prompt gives specific technical and format guidance for the responses of each of those UI values, but it also gives artistic and creative guidance. For example, for movement the prompt suggests animation guidelines like these:

Action: Large, dynamic movements (��30–45vw horizontally, ��25–40vh vertically). Create diagonal paths across all quadrants.Peaceful: Gentle movements (��15–25vw, ��10–20vh). Use figure–8 or circular patterns.Suspense: Small, rapid movements (��5–10vw, ��5–10vh) with unpredictable direction changes.Sleep/Dreamy: Combine gentle scaling (0.8–1.2) with slow drifting (��15vw, ��15vh).

The prompt likewise guides font selection with context-specific recommendations for web-safe font families:

Kid-Friendly: “ui-rounded, Comic Sans MS, cursive”Sci-Fi: “Courier New, monospace”Horror: “fantasy, Times New Roman, serif”Zen: “ui-rounded, sans-serif”

The result is a system that can interpret and represent any scenario through simple visual and motion language.

Prompt as creative guide and requirements doc

The prompt serves triple duty as technical, creative, and product director. It used to be that programming app behavior was the exclusive domain of developers. Now designers and product folks can get in the mix, too, just by describing how they want the app to work. The prompt becomes a hybrid artifact to run the show: part design spec, part requirements doc, part programming language.

For Sentient Scenes, crafting these prompts became a central design activity���even more important than tinkering in Figma, which we hardly touched. Instead, we spent significant time in ChatGPT and Claude iterating on prompt variations to find the sweet spot of creativity and reliability.

We started with informal prompts to see what kinds of responses we’d get. We asked OpenAI questions like:

If I give you a theme like ‘horror movie’, how would you animate my character? What colors would you use? What would the caption say?

And then we started telling it to format those responses in a structured way, starting with Markdown for readability, so responses to our prompts came back like this:

- background: #0D0D0D (Jet Black)- foreground: #D72638 (Amaranth Red)- caption: ���The shadows twisted and crawled, whispering secrets no living soul should hear...���- emoji: ����

Creepy, right? These explorations became the foundation for our system prompt, which evolved into a comprehensive creative and technical brief that we deliver to the system. It meant that we could test the implementation details before writing a single line of code.

Crafting these prompts became a central design activity���even more important than tinkering in Figma, which we hardly touched.

As we became more satisfied with the results, we added new formatting instructions to the prompt to help it talk more seamlessly with the UI engine, the front-end code that draws the interface. We told the system to respond only in JSON objects with the CSS values and content prompts to insert into specific slots in the interface. It���s fun and challenging work���getting a probablistic system to talk to a deterministic one: a ���creative��� system organizing itself to speak in API.

This is where product design is moving. Savvy designers now sketch prompts in addition to visual elements, exploring the boundaries of open-ended, radically adaptive experiences. The work becomes more meta: establishing the rules and physics of a tiny universe rather than specifying every interaction. The designer defines parameters and possibilities, then steps back to let the system and user collaborate within that carefully crafted sandbox.

Try It Yourself

We built Sentient Scenes as a demo and as a teaching tool. Play with it as a user, and then explore how it works as a product designer. Install it on your own server; it’s built with plain-old web technologies (no frameworks, just vanilla JavaScript and a simple PHP backend with minimal requirements).

This doesn���t have to be complicated. On the contrary, our goal with this little project was to show how simple it can be. This stuff is much more a challenge of imagination than of technology. Go splash in puddles; explore, and have a little fun. Sentient Scenes is a small demonstration of the big possibilities of Sentient Design

About Sentient Design

Sentient Design is the already-here future of intelligent interfaces: AI-mediated experiences that feel almost self-aware in their response to user needs. These experiences are conceived and compiled in real time based on your intent in the moment���experiences that adapt to people, instead of forcing the reverse.

Sentient Design describes not only the form of these new user experiences but also a framework and a philosophy for working with machine intelligence as a design material.

The Sentient Design framework was created by Big Medium���s Josh Clark and Veronika Kindred as the core of the agency���s AI practice. Josh and Veronika are also authors of Sentient Design, the forthcoming book from Rosenfeld Media.

Ready to bring these concepts to your products? We help companies develop meaningful AI-powered experiences through strategy, design, and development services. Our Sentient Design workshops can also level up your team’s capabilities in this emerging field. Get in touch to explore how Sentient Design can transform your user experiences.