Models represent emotions
and it changes how they behave.
Happy Friday!
I’m going to dig a little deeper than usual in this fascinating paper today. As someone with a background in UX, and that wants to understand what happens inside these models, this is catnip for me.
Researchers from Anthropic looked at how Claude 4.5 (a recent and very good model) represents emotions internally, and how they impact how it behaves.
The way models work is they internally represent concepts in a vector space - concepts can be “New York” or “potatoe” or “eating”. Everything lives in a many-dimensional semantic space, based (simplified) on how often it’s mentioned together.
That internal representation is the model’s “world knowledge”.
The research shows that:
Models represent emotions that they’ve encountered in their training data. Sadness. Excitement. (Note: nobody here is saying that the models “feel”.)
While generating tokens, the model weights “track” the emotional vibe of what’s being said.
The key finding is that these emotions influence the output. Bad vibes make the model exhibit more “misaligned behaviors such as reward hacking, blackmail, and sycophancy.“
What does it all mean?
It’s useful to understand that emotions can throw a model off. When doing context design, for example.
It’s also important to wrap your head around the fact that, even though we use human words to understand the models (“emotion”, “reasoning”, …), what really happens is not at all the same thing that happens in humans. And yet, anthropomorphizing models by using human words and metaphors can be wildly useful, both for understanding what is happening, and for making predictions about things that might work well. Both those things are true.
Finally, it’s just fascinating to me that we now have model psychiatry. (Anthropic hired a psychiatrist to evaluate their latest Mythos model.) As we have more and more decisions being made by these models, the better we understand how they make those decisions, the better we can design a future that works for everyone.
Interesting this week:
Vidhya Srinivasan, VP of Google Ads, talking about ads in the context of the newly redesigned Google homepage (see below as well): “The best ads must be answers.” Ads are already running inside AI Mode. They’re not banners next to the output. They’re generated by Gemini to read as part of the conversation. And advertisers who want to appear in the new AI search? They must hand over creative and targeting control to Google’s system. “You can’t choose keywords anymore,” Srinivasan said. [...] When you search for a product, Gemini writes a custom explainer for the advertiser, framed as objective advice about why this product “may be the right choice for you.”
Check out the most dramatic redesign of the Google.com homepage in 25 years.
Anyone that has done qualitative user research (a lot of UX researchers) thinks the whole “synthetic users” thing is mostly useless and possibly actively dangerous. Bain published a more positive report that’s not terrible and worth a read. And in the spirit of keeping an open mind (and models become better fast), it may be that these things are becoming useful. I am still doubtful/conflicted about the real-ness of any insights you might get from AI representing users though. But if emotions are represented realistically in LLMs, maybe synthetic users can become a useful thing.
Google’s guide to optimizing for their AI overviews.
A key role for UX departments right now: track and understand how humans use AI, and how expectations are evolving quickly. Anthropic publishes some useful research regularly, openAI now also published some (high level) data.
Many Americans dislike data centers in their area.
Models are both becoming much more useful, and pricier. Anthropic was the expensive but very good model, then OpenAI increased prices for their best models, now Google is increasing pricing. Not a little, think 3X increases. This represents demand and somewhat limited supply: the latest models are so good at coding (agent tasks) that demand is through the roof, and paying $200/month, or much more expensive API pricing, doesn't seem too crazy. This also gives the labs a strong lever to keep you in their ecosystem - use their own tools, and you can use a subsidized subscription model.
Andrej Karpathy joins Anthropic.
Health and happiness,
Peter
PS: new AI courses on modelcontextexperience.com are on the way, it’s taking longer than I had planned, as it always does…