One night in May 2020, during the height of lockdown, Deep Ganguli was worried.
Ganguli, then research director at the Stanford Institute for Human-Centered AI, had just been alerted to OpenAI’s new paper on GPT-3, its latest large language model. This new AI model was potentially 10 times more advanced than any other of its kind — and it was doing things he had never thought possible for AI. The scaling data revealed in the research suggested there was no sign of it slowing down. Ganguli fast-forwarded five years in his head, running through the kinds of societal implications he spent his time at Stanford anticipating, and the changes he envisioned seemed immeasurable. He knew he couldn’t sit on the sidelines while the tech rolled out. He wanted to help guide its advancement.
His friend Jack Clark had joined a new startup called Anthropic, founded by former OpenAI employees concerned that the AI giant wasn’t taking safety seriously enough. Clark had previously been OpenAI’s policy director, and he wanted to hire Ganguli at Anthropic for a sweeping mission: ensure AI “interacts positively with people,” in everything from interpersonal interactions to the geopolitical stage.
Over the past four years, Ganguli has built what’s known as Anthropic’s societal impacts team, a small group that’s looking to answer the thorniest questions posed by AI. They’ve written research papers on everything from AI’s economic impact to its persuasiveness, as well as explorations of how to mitigate elections-related risks and discrimination. Their work has, perhaps more than any other team, contributed to Anthropic’s carefully tended reputation as the “safe” AI giant dedicated to putting humans first.
But with just nine people among Anthropic’s total staff of more than 2,000, in an industry where mind-boggling profits could await whoever’s willing to move quickest and most recklessly, the team’s current level of freedom may not last forever. What happens when just a handful of employees at one of the world’s leading AI companies — one that nearly tripled its valuation to $183 billion in less than a year, and is now valued in the range of $350 billion — are given the blanket task of figuring out how the ultra-disruptive technology is going to impact society? And how sure are they that executives, who are at the end of the day still looking to eventually turn a profit, will listen?
““We are going to tell the truth.”
Nearly every major AI company has some kind of safety team that’s responsible for mitigating direct, obvious harms like AI systems being used for scams or bioweapons. The goal of the societal impacts team — which does not have a direct analog at OpenAI, Meta, or Anthropic’s other big competitors — is broader. Ganguli sees his job as finding “inconvenient truths” about AI that tech companies have incentives not to publicize, then sharing them with not only Anthropic leadership, but the rest of the world.
“We are going to tell the truth,” Ganguli said. “Because, one, it’s important. It’s the right thing to do. Two, the stakes are high. These are people. The public deserves to know. And three, this is what builds us trust with the public, with policymakers. We’re not trying to pull the wool over anyone’s eyes. We’re just trying to say what we’re seeing in the data.”
The team meets in the office five days a week, spending a good amount of time in Anthropic’s eighth-floor cafeteria, where Saffron Huang, one of the research scientists, usually grabs a flat white before a working breakfast with Ganguli and others. (“That’s the Kiwi in me,” says Huang, a New Zealander who founded a nonprofit in London before joining Anthropic in 2024.) Team members work out together at the gym and have late nights at the office and day trips to the beach. They’ve met each other’s mothers and ridden in each other’s cars while picking up their kids from school. They see so much of each other that Ganguli sometimes forgoes after-work hangouts — “I see you all more than my family!” a team member recalls him saying.
The result is a level of comfort voicing opinions and disagreements. The group is big on the “cone of uncertainty,” a phrase they use when, in true scientist fashion, they’re not sure about aspects of the data they’re discussing. It’s also the name of a literal traffic cone that research engineer Miles McCain and Anthropic’s facility team found, cleaned up, and fixed with googly eyes before installing it in the office.
The societal impacts team launched as Ganguli’s one-man operation when Anthropic was solely a research lab. Research scientist Esin Durmus joined him in February 2023, as Anthropic was gearing up to launch Claude the following month. Their work involved considering how a real future product might affect humanity — everything from how it could impact elections to “which human values” it should hold. Durmus’ first research paper focused on how chatbots like Claude could offer biased opinions that “may not equitably represent diverse global perspectives on societal issues.”
Around Claude’s launch, the team relied on testing models before deployment, attempting to anticipate how people would engage with them. Then, suddenly, thousands — later millions — of people were using a real product in ways the team had no way to gauge.
AI systems, they knew, were unpredictable. For a team designed to measure the impact of a powerful new technology, they knew frustratingly little about how society was using it. This was an unprecedented cone of uncertainty, spurring what eventually became one of the team’s biggest contributions to Anthropic so far: Claude’s tracking system, Clio.
One of the most “inconvenient truths” the team has released was the creation of “explicit pornographic stories with graphic sexual content.”
Anthropic needed to know what people were doing with Claude, the team decided, but they didn’t want to feel like they were violating people’s trust. “If we’re talking about insight versus privacy, you can have a ton of insight by having no privacy,” Ganguli said, adding, “You could also have a ton of privacy with zero insight.” They struck a balance after consulting with Anthropic engineers and external civil society organizations, resulting in, essentially, a chatbot version of Google Trends. Clio resembles a word cloud with clusters of topics describing how people are using Claude at any given time, like writing video scripts, solving diverse math problems, or developing web and mobile applications. The smaller clusters run the gamut from dream interpretation and Dungeons & Dragons to disaster preparedness and crossword puzzle hints.
Today, Clio is used by teams across Anthropic, offering insight that helps the company see how well safeguards and reinforcement learning are working. (There’s a Slack channel called Clio Alerts that shares automated flags on what each team is doing with the tool; Ganguli says he often stares at it.) It’s also the basis of much of the societal impacts team’s own work.
One of the most “inconvenient truths” the team has released came from using Clio to analyze Anthropic’s safety monitoring systems. Together with the safeguards team, Miles McCain and Alex Tamkin looked for harmful or inappropriate ways people were using the platform. They flagged uses like the creation of “explicit pornographic stories with graphic sexual content,” as well as a network of bots that were trying to use Claude’s free version to create SEO-optimized spam, which Anthropic’s own safety classifiers hadn’t picked up — and they published the research in hopes that it’d help other companies flag their own weaknesses. The research led to Anthropic stepping up its detection of “coordinated misuse” at the individual conversation level, plus figuring out how to monitor for issues they may not be able to even name yet.
“I was pretty surprised that we were able to just be quite transparent about areas where our existing systems were falling short,” said McCain, who built the Clio tool and also focuses on how people use Claude for emotional support and companionship, as well as limiting sycophancy. He mentioned that after the team published that paper, Anthropic made Clio an “important part of our safety monitoring stack.”
As team leader, Ganguli talks the most with executives, according to members — although the team presents some of their research results every so often on an ad hoc basis, he’s the one with the most direct line to leadership. But he doesn’t talk to Anthropic CEO Dario Amodei regularly, and the direct line doesn’t always translate to open communication. Though the team works cross-functionally, the projects are rarely assigned from the top and the data they analyze often informs their next moves, so not everyone always knows what they’re up to. Ganguli recalled Amodei once reaching out to him on Slack to say that they should study the economic impacts of AI and Anthropic’s systems, not realizing the societal impacts team had already been discussing ways to do just that. That research ended up becoming Anthropic’s Economic Index, a global tracker for how Claude is being used across each state and the world — and how that could impact the world economy.
When pressed on whether executives are fully behind the team’s work, even if it were not to reflect well on the company’s own technology, team members seem unfazed — mostly because they say they haven’t had any tangible reasons to worry so far.
“I’ve never felt not supported by our executive or leadership team, not once in my whole four years,” Ganguli said.
The team also spends a good bit of time collaborating with other internal teams on their level. To Durmus, who worked on a paper charting the types of value judgments Claude makes, the societal impacts team is “one of the most collaborative teams” at the company. She said they especially work with the safeguards, alignment, and policy teams.
McCain said the team has an “open culture.” Late last year, he said, the group worked closely with Anthropic’s safety team to understand how Claude could be used for nefarious election-related tasks. The societal impacts team built the infrastructure to run the tests and ran periodic analyses for the safety team — then the safety team would use those results to decide what they’d prioritize in their election safety work. And since McCain and his colleagues only sit a couple of rows of desks away from the trust and safety employees, they also have a good working relationship, he said, including a Slack channel where they can send concerns their way.
But there’s a lot we don’t know about the way they work.
Image: Cath Virginia / The Verge, Getty Images, Anthropic
There’s a tungsten cube on Saffron Huang’s desk, apparently. I have to take her word on that, as well as any other details about the team’s working environment, because most of Anthropic’s San Francisco headquarters is strictly off-limits to visitors. I’m escorted past a chipper security desk with peel-and-stick nametags and an artful bookshelf, and then it’s into the elevator and immediately to the office barista, who’s surrounded by mid-century modern furniture. (I’m proudly told by members of Anthropic’s public relations team, who never leave my side, that the office is Slack’s old headquarters.) I’m swiftly escorted straight into a conference room that tries to mask its sterile nature with one warm overhead light and a painting of a warped bicycle on the wall.
I ask if I can see Huang and the rest of the team’s workspace. No, I’m told, that won’t be possible. Even a photo? What about a photo with redacted computer screens, or getting rid of everything on the desks that could in any way be sensitive? I’m given a very apologetic no. I move on.
Huang’s tungsten cube probably looks just like any other. But the fact I can’t confirm that is a reminder that, though the team is committed to transparency on a broad scale, their work is subject to approval from Anthropic. It’s a stark contrast with the academic and nonprofit settings most of the staff came from.
“Being in a healthy culture, having these team dynamics, working together toward a good purpose, building safe AI that can benefit everyone — that comes before anything, including a lot of money.”
Huang’s first brush with Anthropic came in 2023. She’d started a nonprofit called the Collective Intelligence Project, which sought to make emerging technologies more democratic, with public input into AI governance decisions. In March 2023, Huang and her cofounder approached Anthropic about working together on a project. The resulting brainstorming session led to their joint “collective constitutional AI” project, an exercise in which about 1,000 randomly chosen Americans could deliberate and set rules on chatbot behavior. Anthropic compared what the public thought to its own internal constitution and made some changes. At the time of the collaboration, Huang recalls, Anthropic’s societal impacts team was only made up of three people: Ganguli, Durmus, and Tamkin.
Huang was considering going to grad school. Ganguli talked her out of it, convincing her to join the societal impacts team.
The AI industry is a small world. Researchers work together in one place and follow the people they connect with elsewhere. Money, obviously, could be a major incentive to pick the private sector over academia or nonprofit work — annual salaries are often hundreds of thousands of dollars, plus potentially millions in stock options. But within the industry, many employees are “post-money” — in that AI engineers and researchers often have such eye-popping salaries that the only reason to stay at one job, or take another, is alignment with a company’s overall mission.
“To me, being in a healthy culture, having these team dynamics, working together toward a good purpose, building safe AI that can benefit everyone — that comes before anything, including a lot of money,” Durmus said. “I care about this more than that.”
Michael Stern, an Anthropic researcher focused on AI’s economic impact, called the societal impacts team a “lovely mix of misfits in this very positive way.” He’d always had trouble fitting into just one role, and this team at Anthropic allowed him to combine his interests in safety, society, and security with engineering and policy work. Durmus, the team’s first hire after Ganguli himself, had always been interested in both computer science and linguistics, as well as how people interact and try to sway each other’s opinions online.
Kunal Handa, who now works on economic impact research and how students use Claude, joined after cold-emailing Tamkin while Handa was a graduate student studying how babies learn concepts. Tamkin, he had noticed, was trying to answer similar questions at Anthropic, but for computers instead. (Since time of writing, Tamkin has moved to Anthropic’s alignment team, to focus on new ways to understand the company’s AI systems and making them safer for end users.)
In recent years, many of those post-money people concerned with the advancement (and potential fallout) of AI have left the leading labs to go to policy firms or nonprofits, or even start their own organizations. Many have felt they could have more impact in an external capacity. But the societal impacts team’s broad scope and expansive job descriptions still prove more attractive for several team members.
“I am not an academic flight risk … I find Deep’s pitch so compelling that I never even really considered that path,” McCain said.
It’s a “lovely mix of misfits in this very positive way.”
For Ganguli himself, it’s a bit different. He speaks a lot about his belief in “team science” — people with different backgrounds, training, and perspectives all working on the same problem. “When I think about academia, it can be kind of the opposite — everyone with the same training working on a variety of different problems,” Ganguli said, adding that at Stanford, he sometimes had trouble getting people to emulate team science work, since the university model is set up differently. At Anthropic, he also values having access to usage data and privileged information, which he wouldn’t be able to study otherwise.
Ganguli said that when he was recruiting Handa and Huang, they were both deciding between offers for graduate school at MIT or joining his team at Anthropic. “I asked them, ‘What is it that you actually want to accomplish during your PhD?’ And they said all the things that my team was working on. And I said, ‘Wait, but you could just actually do that here in a supportive team environment where you’ll have engineers, and you’ll have designers, and you’ll have product managers — all this great crew — or you could go to academia where you’ll kind of be lone wolf-ing it.’”
He said their main concerns involved academia potentially having more freedom to publish inconvenient truths and research that may make AI labs look less than optimal. He told them that at Anthropic, his experience so far has been that they can publish such truths — even if they reveal things that the company needs to fix.
Of course, plenty of tech companies love transparency until it’s bad for business. And right now, Anthropic in particular is walking a high-stakes line with the Trump administration, which regularly castigates businesses for caring about social or environmental problems. Anthropic recently detailed its efforts to make Claude more politically middle-of-the-road, months after President Donald Trump issued a federal procurement ban on “woke AI.” It was the only AI company to publicly voice its stance against the controversial state AI law moratorium, but after its opposition earned it the ire of Trump’s AI czar David Sacks, Amodei had to publish a public statement boosting Anthropic’s alignment with aspects of Trump administration policy. It’s a delicate balancing act that a particularly unwelcome report could upset.
But Ganguli is confident the company will keep its promise to his team, whatever’s happening on the outside.
“We’ve always had the full buy-in from leadership, no matter what,” he said.
Image: Cath Virginia / The Verge, Getty Images, Anthropic
Ask each member of Anthropic’s societal impacts team about their struggles and what they wish they could do more of, and you can tell their positions weigh heavily on them. They clearly feel that an enormous responsibility rests upon their shoulders: to shine a light on how their company’s own technology will impact the general public.
People’s jobs, their brains, their democratic election process, their ability to connect with others emotionally — all of it could be changed by the chatbots that are filling every corner of the internet. Many team members believe they’ll do a better job guiding how that tech is developed from the inside rather than externally. But as the exodus of engineers and researchers elsewhere shows, that idealism doesn’t always pan out for the broader AI industry.
A struggle that the majority of team members brought up was time and resource constraints — they have many more ideas than they have bandwidth for. The scope of what the team does is broad, and they sometimes bite off more than they can chew. “There are more coordination costs when you’re 10 times the size as you were two years ago,” Tamkin said. That pairs, sometimes, with the late nights — i.e., “How am I going to talk to 12 different people and debug 20 different errors and get enough sleep at night in order to release a report that feels polished?”
The team, for the most part, would also like to see their research used more internally: to directly improve not only Anthropic’s AI models, but also specific end products like Claude’s consumer chatbot or Claude Code. Ganguli has one-on-one meetings monthly with chief science officer Jared Kaplan, and they often brainstorm ways to allow the societal impacts team to better impact Anthropic’s end product.
““There are more coordination costs when you’re 10 times the size as you were two years ago.”
Ganguli also wants to expand the team soon, and many team members hope that type of resource expansion means they’ll be able to better document how users are interacting with Claude — and the most surprising, and potentially concerning, ways in which they’re doing so.
Many team members also brought up the fact that looking at data in a vacuum or lab setting is very different from the effect AI models have in the real world. Clio’s analysis of how people are using Claude can only go so far. Simply observing use cases and analyzing aggregated transcripts doesn’t mean you know what your customers are doing with the outputs, whether they’re individual consumers, developers, or enterprises. And that means “you’re left to sort of guess what the actual impact on society will be,” McCain said, adding that it’s a “really important limitation, and [it] makes it hard to study some of the most important problems.”
As the team wrote in a paper on the subject, “Clio only analyzes patterns within conversations, not how these conversations translate into real-world actions or impacts. This means we cannot directly observe the full societal effects of AI system use.” It’s also true that until recently, the team could only really analyze and publish consumer usage of Claude via Clio — in September, for the first time, the team published an analysis of how businesses are using Claude via Anthropic’s API.
“Models and AI systems don’t exist in isolation — they exist in the context of their deployments, and so over the past year, we’ve really emphasized studying those deployments — the ways that people are interacting with Claude,” McCain said. “That research is going to have to also evolve in the future as the impacts of AI affect more and more people, including people who may not be interfacing with the AI system directly … Concentric circles outward.”
That’s why one of the team’s next big research areas is how people use Claude not just for its IQ, but also for its EQ, or emotional intelligence. Ganguli says that a lot of the team’s research to date has been focused on cut-and-dried answers and measurable impacts on the economy or labor market, and that its EQ research is relatively new — but the team will prioritize it in the next six months. “Once people leave the chatbot, we’re not entirely sure exactly how they were affected or impacted, and so we’re trying to develop new methods and new techniques that allow us to understand,” he said, referring to taking a more “human-centered approach” and doing more “social science research” akin to coupling data analysis with surveys and interviews.
“What does it mean for our world, in which you have a machine with endless empathy you can basically just dump on, and it’ll always kind of tell you what it thinks?”
Since people are emotionally influenced by their social networks, it stands to reason they can be influenced greatly by AI agents and assistants. “People are going to Claude … looking for advice, looking for friendship, looking for career coaching, thinking through political issues — ‘How should I vote?’ ‘How should I think about the current conflicts in the world?’” Ganguli said. “That’s new … This could have really big societal implications of people making decisions on these subjective things that are gray, maybe more matters of opinion, when they’re influenced by Claude, or Grok, or ChatGPT, or Gemini, or any of these things.”
By far the most pressing EQ-related issue of the day is widely known as “AI psychosis.” The phenomenon references a range of conditions where AI leads a user down a delusional spiral and causes them, on some level, to lose touch with reality. The user typically forms an emotional bond with a chatbot, made more intense by the chatbot’s memory of previous conversations and its potential to drift away from safety guardrails over time. Sometimes this can lead to the user believing they’ve unearthed a romantic partner “trapped” inside the chatbot who longs to be free; other times it can lead to them believing they’ve discovered new secrets to the universe or scientific discoveries; still other times it can lead to widespread paranoia and fear. AI psychosis or delusion has been a main driver behind some teen suicides, as well as ensuing lawsuits, Senate hearings, newly passed laws, and parental controls. The issue, experts say, is not going anywhere.
“What does it mean for our world, in which you have a machine with endless empathy you can basically just dump on, and it’ll always kind of tell you what it thinks?” Ganguli said. “So the question is: What are the kinds of tasks people are using Claude for in this way? What kind of advice is it giving? We’ve only just started to uncover that mystery.”