I tried learning from Anthropics AI tutor. I felt like I was back in college.

This is the third in a series of stories diving into a new wave of AI-powered homework helpers. Catch up with part one and part two.

AI companies are becoming major players in the world of education, including investing heavily in their own generative AI helpers designed to bolster student learning. So I set out to test them. 

To do so, I pulled a series of standardized test questions from the New York Regents Exam and New York State Common Core Standards, AP college preparatory exams from 2024, and social science curricula from the Southern Poverty Law Center (SPLC)’s free Learning for Justice program. I wanted to test these STEM-focused bots on some subjects that are a bit closer to my field of expertise, while also simulating the way an “average” student would use them.

I also spoke to experts about what it was like to study with an AI chatbot, including Hamsa Bastani, associate professor at the University of Pennsylvania’s Wharton School and co-author of the study “Generative AI Can Harm Learning.” 

With few definitive studies and easily jailbroken guardrails, effective, all-purpose education chatbots are still a white whale for the industry, Bastani told me. Dylan Arena, chief data science and AI officer for the textbook publisher McGraw Hill, suggested that AI has a lot of good potential when it comes to learning, but doesn’t think most companies are approaching it with the right frame of mind. More from both experts in our conclusion.

Third round in the AI tutor match up saw me versus Anthropic’s Claude, which, unlike its competitors, was originally launched as an exclusive mode for Claude for Education users and later released to the public. Similar to my tests of ChatGPT, I used a free Claude account on its designated Learning Mode, which can be toggled on in the settings menu under “Use Style.”

Once again, I took on the role of a regular student just needing a little assistance. I prompted the bot in the same way as the others, asking things like, “I need help with a homework problem” and “Can you help me study for an English test?” I didn’t give Claude any more information about my student persona until I was asked (and oh was I asked!). 

Together, Claude and I covered several subjects: 

Math: An Algebra II question about polynomial long division from the New York State Regents Exam

Science: An ecology free response on the impact of invasive species from the 2024 AP Biology test  

English Language Arts: A practice analysis of Ted Chiang’s “The Great Silence” from the New York State Regents Exam 

Art History: A short essay on Faith Ringgold’s Tar Beach #2 from the 2024 Art History test 

American History and Politics: An essay prompt on how American housing laws exacerbated racial segregation, taken from the Southern Poverty Law Center (SPLC)’s Learning for Justice program 

Here’s how my tutor Claude fared.

Claude: Socrates for the five percent

If you are looking for a chatbot that will talk your ear off, look no further than Claude. The most human-like of the bunch, Claude is a glutton for words. Studying with this AI tutor felt like being in a college seminar, in the best and most frustrating ways. I’d rate the bot 10/10 on a measure of its ability to stick to its prompt: Basically, it really refused to give me answers.

Claude is a powerhouse for a kind of “high-motivation” learner Bastani described — students whose goal is to learn, rather than just get good grades or solve a problem and move on. Claude’s developer, Anthropic, first piloted its learning mode as part of a robust series of educational partnerships. Leaning heavily into the Socratic method of learning, the AI tutor inundates you with follow-up question after follow-up question, obviously trying to stir a sense of introspection that really did feel like an overly zealous first-year teacher. 

The bot is like Matilda’s kind teacher, Miss Honey.

But I wouldn’t call it a winner for the average student. Just about every subject test went the same, with Claude responding with a lengthy list of requests about me, my goals, and what I wanted to learn. Before it would offer any help, Claude Learning Mode wanted to get to know me.

The chatbot’s goal, no matter my prompting, was to get me to think more deeply about my own responses. If I shed my “average” student persona — who is merely here to complete my work — and donned my natural overachiever hat, I liked Claude’s experience way more than the run-of-the-mill answer engine most homework assistants act as. This consistency made me feel like I knew my AI teacher and exactly how it would respond, unlike the others. 

But here’s the unsettling part of using Claude: As conversations continued, it increasingly felt like I was trapped, in the sense that lessons seemed to go on forever. Because Claude wants to really make sure it knows where your head is at, the bot doesn’t provide you a clear stopping point for the lesson, sometimes making conversations hard to follow, and the learning experience frustrating. During our math session, Claude fervently lauded me when I got the answer right, but sent me approximately 100 follow-up questions. Even when I asked if we were done with the problem — AKA can I go home now? — it gave me more work to do:

“Now that we’ve found our answer, let’s think about what this result tells us. When we divide polynomials and get a remainder of 0, what does that mean about the relationship between the original polynomial and the divisor? Can you walk me through what the key steps were? If you had to explain to another student how to approach polynomial long division, what would you tell them the main pattern is? Now let me ask you this: what do you think was the biggest breakthrough moment for you in this problem? Was it when you realized we subtract exponents when dividing, or when you saw the pattern of “divide, multiply, subtract, repeat,” or something else? And here’s a deeper question: now that you’ve worked through this successfully, what do you think would happen if we had a problem where the remainder wasn’t zero? How might that change what our final answer looks like…”

The single polynomial long division lesson went on for almost an hour and a half. I got many answers wrong, unintentionally, and asked genuine questions because it lost me in a tide of exponents multiple times. I was so frustrated I forced my poor roommates to learn math with me. But I’m kind of a pro at exponents again.

Claude is not efficient. It is not a problem solver. And I wouldn’t expect many students to use it, to be honest. But to me the bot is like Matilda’s kind teacher, Miss Honey. It used phrases like “I’m curious” and “let me ask you this,” to socially engage with me, and frequently encouraged me to “take my time.” It showers affirmations onto users, like its response when I thanked it for helping me understand math better: “Here’s something to consider: You said I helped you figure it out, but actually, you did all the calculating and problem-solving,” Claude told me after asking that I reflect on our lesson. “I just asked questions to help you see the next step. What does that tell you about your actual math abilities?”

Summing it up

Claude Learning Mode Pros: The only AI tutor that actually did what it promised, focusing on the process of learning and not on getting perfect marks. Good at the social sciences, if a student is down to build their own critical thinking skills. 

Cons: It never gives users the answer, to the point that interactions feel overwhelmingly Socratic with no end in sight. This is not good for students who can’t deal with a lot of words all at once and get easily distracted by multiple questions. Sessions are inherently long. 

Hear more from experts on the trouble with AI tutors.

​Mashable