Stanford HAI study finds most Claude chats involve serious work

A Stanford team has read through a quarter of a million anonymized conversations between people and Claude, Anthropic's chatbot. The study is backed by the Stanford Institute for Human-Centered AI. The journal Nature Human Behaviour has accepted it for publication.

The headline figure is 56%. According to the Stanford HAI announcement^[1], well over half of the collaborations are rated "consequential" or higher.

Who did the measuring, and on what data

The study is built on real-world use. The conversations come from real users working on real tasks.

The researchers used Insights, Anthropic's analysis tool. They used it to answer three questions.

• Does the work people bring to AI carry real weight?
• How much control does the person keep in the conversation?
• Where does friction arise between person and AI, and when does that friction help or break the collaboration?

The senior author is Diyi Yang, a computer science professor at Stanford and a senior fellow at HAI. The co-first authors are Yijia Shao and Dora Zhao, PhD students in her lab.

More than half the work has serious effects

56% of collaborations are rated "consequential" or higher. The term describes tasks with serious effects that are hard to reverse.

Shao says users sometimes bring trivial tasks, such as drafting an email. Far more often they bring high-stakes work that is hard to undo.

Yang reads the figure as a counterpoint to the bleakest predictions about AI. In her view, the vast majority of users treat the model as a source of ideas and pushback. The model tests the reasoning of the person using it.

Three numbers the announcement leaves out

The 56% is the only percentage in the public text. Weighing it properly needs three things the announcement doesn't provide.

1. Exact sample. The announcement refers to "a quarter of a million" conversations. It gives no precise number, collection period or user profile.
2. Distribution. The 56% combines the "consequential" level with the levels above it. The share at each level, and the share of trivial tasks, are missing.
3. Classification method. The announcement names Insights as the tool. It doesn't say who assigns each conversation its level, or how that is checked.

The data and the tool come from Anthropic, the same company that makes Claude.

Higher stakes, longer exchanges

The second finding concerns behavior. According to Zhao, users engage more as the stakes rise. Conversations about serious tasks last almost twice as long as those about everyday ones.

The finding links how much a task matters to how long the exchange runs. It remains a correlation.

The announcement gives the ratio but not the average lengths. It also describes "friction" that can help or harm, and the available text stops before any data on that point.

Data from real work, method still under wraps

I have long thought that a benchmark measures the test and misses real work. Here the measurement comes from real work. That is why the study deserves attention.

The field that studies this kind of exchange is called human-AI interaction, as summarized in the Wikipedia entry^[2]. Much of that field relies on controlled experiments.

What the 56% is worth depends on the scale used. A scale defined one way produces this figure. A different scale could produce another.

Which argument holds up in front of the board

The data supports a specific argument. People use AI for serious work, and they stay longer in the conversation when the task matters.

The idea of AI as a shortcut for trivial tasks loses ground. By this measure, it describes a minority of cases.

For data and analytics leaders, the study shows that real-world use can be measured at scale. The result is only as good as the rule used to classify each conversation.

For now, the 56% is a production figure from a method that has not been made public. Cite it with its source and its limits.