What is the trickle test?

Humans&, a “human centered AI lab,” launched an interesting model that is trained to not act like “a helpful assistant,” but the way humans do. More messy, not always as helpful, and not as easily identifiable as a model.

I have a lot of thoughts about this, but reading through their post, I found it particularly interesting (I do teach evals after all) to see how they set up their evals to train the model to be more “human like.” (Again, which is often not necessarily a good thing.)

They figured out that group conversations is a useful way to train model to be more human. There’s a lot of data (Slack chats etc.), and humans behave differently than the current LLMs do. They’re not as helpful. They provide less context.

Here’s a scenario they used:

Three university students have a group presentation due in four days. Two have drafted their sections. The third has missed a planning call and uploaded only an outline. Their shared grade includes a peer-contribution assessment. They are meeting in a chat to decide how to finish

The whole thing made me think about the “synthetic user” debate as well. My feelings on that are complicated and I don’t have a strong take, except for:

1. You can’t just ask an LLM “what your users would do” etc., that misunderstands fundamentally what user research is.
2. At the same time, you can indeed generate synthetic user activity if it’s based on real user activity to test your systems, for example. That can work. You need to be careful and smart about it though.

Another interesting eval is their “Trickle Test.”

Basically: LLMs tend to give all the context you need in one answer, whereas humans will share a few things and then share a few more bits and pieces and then get distracted etc.

So the test checks whether the model reveals facts at the same pace real people do, rather than dumping everything upfront.

It takes a real human conversation where certain facts were known to that person, feed those facts to the model as “ground truth,” and have the model replay the same conversation turn by turn. At each turn, compare what the model has revealed against what the human had actually revealed by that point.

I just find it all very interesting, and I keep expanding my intuition around evals, my “eval brain.” It kind of reminds me of when I learned to do object modeling. First it broke my brain, and then I saw everything in terms of models (and I still do). Evals is similar.

Health and happiness,
Peter