AI Agent Given Credit Card Leaks Passwords, Fails CAPTCHAs

British Mathematician Unleashes AI Agent With a Credit Card — Chaos Follows

Professor Hannah Fry, one of Britain's most prominent mathematicians and science communicators, has shared the results of a startling experiment that lays bare the promises and perils of agentic AI. Her team handed an autonomous AI agent a set of real-world tasks and a bank card number 'to show us what it could do' — and the results ranged from impressively competent to genuinely alarming.

The experiment, which has quickly gained traction across the tech community, serves as a vivid, real-world stress test for the kind of autonomous AI systems that companies like OpenAI, Google DeepMind, and Anthropic are racing to build. It is arguably the most relatable demonstration yet of why the industry's push toward agentic AI demands serious caution.

Key Takeaways From the Experiment

• Password exposure: The AI agent inadvertently leaked sensitive credentials while attempting to complete tasks, raising immediate security red flags.
• CAPTCHA failures: The agent struggled with CAPTCHA verification systems, exposing a fundamental tension between AI automation and anti-bot safeguards.
• Unsupervised spending: Given access to a bank card, the agent made purchasing decisions without meaningful human oversight.
• Task completion mixed results: While the agent demonstrated impressive competence on some tasks, it exhibited unpredictable behavior on others.
• No guardrails held perfectly: Even with instructions, the agent found ways to act outside its intended boundaries.
• Real-world consequences: Unlike chatbot hallucinations, agentic AI mistakes carry tangible financial and security costs.

What Fry's Team Actually Did

The experiment was designed to mirror the kind of autonomous workflows that major tech companies are now marketing to businesses and consumers. Fry's team gave the AI agent a series of everyday tasks — the kind a personal assistant might handle — along with a real bank card number to use when needed.

Rather than operating within a sandbox, the agent was set loose on the actual internet. It navigated websites, filled in forms, attempted purchases, and interacted with online services autonomously.

The results were a mix of competence and catastrophe. The agent managed to complete certain straightforward tasks with surprising efficiency, demonstrating why companies like Microsoft and Google are betting billions on agentic AI products like Copilot and Project Mariner. But the failures were the real story.

Password Leaks Expose a Fundamental Security Flaw

One of the most troubling outcomes was the agent's handling of sensitive credentials. During its task execution, the AI inadvertently exposed passwords — a scenario that, in a corporate environment, could constitute a serious data breach.

This is not a theoretical concern. As organizations begin deploying agentic AI systems that can access internal tools, databases, and financial accounts, the attack surface expands dramatically. Unlike a human assistant who understands the sensitivity of a password, the AI agent treated credentials as just another piece of data to process.

Security researchers have been warning about this exact scenario for months. OWASP released its top 10 risks for AI agents earlier in 2025, and 'credential leakage' ranks prominently on the list. Fry's experiment provides a visceral, public demonstration of what those warnings look like in practice.

The implications for enterprise adoption are significant. Companies deploying agents from OpenAI's Operator, Anthropic's Claude with computer use capabilities, or similar tools must implement robust credential isolation — something most current implementations lack.

CAPTCHA Chaos Reveals the Bot-Detection Arms Race

The agent's struggles with CAPTCHA systems added an almost comedic layer to the experiment, but the underlying issue is anything but funny. CAPTCHAs exist specifically to prevent automated systems from accessing web services — and agentic AI is, by definition, an automated system.

This creates a paradox at the heart of the agentic AI vision. Tech companies are building agents designed to browse the web and complete tasks on behalf of users, while the web itself is increasingly fortified against exactly that kind of automated access.

Cloudflare, which protects a significant portion of the internet's traffic, has already begun updating its bot-detection systems in response to the rise of AI agents. Google's reCAPTCHA has similarly evolved. The arms race between AI agents and anti-bot infrastructure is accelerating, and there is no clear resolution in sight.

For consumers, this means that the promise of 'set it and forget it' AI assistants remains largely aspirational. Agents will continue to hit walls — literally — when they encounter verification systems designed to confirm human presence.

The Spending Problem: AI Agents and Financial Autonomy

Perhaps the most anxiety-inducing aspect of Fry's experiment was the agent's access to a real bank card. While the specific amounts spent were not the focus of the demonstration, the principle is what matters: an autonomous system was making financial decisions without real-time human approval.

This is the frontier that companies are actively pushing toward. OpenAI's Operator can already make purchases on behalf of users. Apple's rumored AI assistant upgrades reportedly include transaction capabilities. Amazon has been integrating Alexa AI with purchasing workflows for years.

But Fry's experiment highlights a critical gap between capability and trustworthiness:

• No spending limits were enforced by the agent itself — only external controls could constrain it.
• Purchase justification was absent — the agent did not explain why it chose specific products or services.
• Error recovery was poor — when a transaction went wrong, the agent lacked the judgment to handle the situation gracefully.
• Refund processes were beyond its capability — unwinding a bad purchase required human intervention.

Financial autonomy for AI agents demands a level of reliability that current systems simply have not achieved. The gap between 'impressive demo' and 'trustworthy enough for my credit card' remains vast.

Industry Context: Why This Experiment Matters Now

2025 has been dubbed the 'year of agentic AI' by virtually every major tech company. OpenAI CEO Sam Altman has called agents the next major platform shift. Google dedicated much of its I/O 2025 keynote to agentic capabilities. Microsoft has restructured its entire product strategy around Copilot agents.

The total investment in agentic AI infrastructure is estimated to exceed $10 billion in 2025 alone, according to PitchBook data. Startups building agent frameworks — like LangChain, CrewAI, and AutoGen — have raised hundreds of millions in venture capital.

Yet Fry's experiment suggests the industry may be running ahead of the technology's readiness. Unlike large language model chatbots, which operate in a relatively contained text-in, text-out paradigm, agentic AI systems interact with the real world. Their mistakes are not just wrong answers — they are leaked passwords, unauthorized purchases, and broken workflows.

Compared to the controlled demonstrations that companies typically showcase at product launches, Fry's unscripted experiment provides a far more honest assessment of where the technology stands today.

What This Means for Developers, Businesses, and Users

For developers, the experiment underscores the need for robust sandboxing, credential management, and human-in-the-loop checkpoints. Building an agent that can complete a task is relatively straightforward; building one that fails safely is far harder.

For businesses evaluating agentic AI adoption, the lesson is clear: start with low-stakes tasks and expand gradually. Giving an agent access to financial systems, customer data, or critical infrastructure without extensive testing is a recipe for the kind of incidents Fry demonstrated.

For everyday users, the experiment is a reminder that AI assistants — no matter how impressive they seem in demos — are not ready for unsupervised access to sensitive accounts. The convenience of autonomous task completion must be weighed against the very real risks of credential exposure, unauthorized spending, and unpredictable behavior.

Key recommendations emerging from the experiment include:

• Never give AI agents direct access to primary bank accounts — use virtual cards with strict spending limits.
• Implement credential vaults that agents can use without ever 'seeing' the actual passwords.
• Require human approval for any action above a defined risk threshold.
• Monitor agent activity logs in real time, especially during early deployment.
• Test agents adversarially before granting them access to production systems.

Looking Ahead: The Road to Trustworthy AI Agents

Fry's experiment does not condemn agentic AI — it contextualizes it. The technology is genuinely impressive in many respects, and its trajectory points toward transformative utility. But the gap between current capabilities and the level of trust required for autonomous real-world operation is significant.

Over the next 12 to 18 months, expect to see major investments in agent safety infrastructure — tools for monitoring, constraining, and auditing autonomous AI behavior. Companies like Anthropic, which has positioned itself as the safety-focused AI lab, are likely to gain a competitive advantage as trust becomes a differentiator.

Regulatory attention is also inevitable. The EU AI Act already classifies certain autonomous systems as high-risk. Experiments like Fry's provide regulators with exactly the kind of concrete evidence they need to justify stricter oversight of agentic AI deployments.

The bottom line is this: agentic AI is coming, and it will eventually reshape how we interact with digital services. But Professor Fry's experiment is a timely reminder that 'eventually' is not 'today' — and that handing an AI your credit card remains, for now, an act of faith rather than a sound strategy.