TokenMeter: A Vibe-Coded AI API Speed Testing Tool

Developer Builds AI API Benchmark Tool Without Writing a Line of Code

A developer has released TokenMeter, an open-source tool designed to measure and compare the output speed of various AI API providers — and the twist is that the entire project was built using 'vibe coding,' a growing trend where developers rely entirely on AI to generate their codebase.

The project, available on GitHub, is still in its demo stage but already delivers a clean dashboard showing real-time speed metrics for different API endpoints. A live demo is accessible at the developer's status page.

What TokenMeter Actually Does

TokenMeter focuses on one of the most quantifiable aspects of AI API performance: token output speed. Rather than attempting to evaluate response quality — which remains a notoriously difficult problem — the tool zeroes in on raw throughput.

Key features include:

• Real-time speed monitoring across multiple AI API providers
• Dashboard visualization of tokens-per-second output rates
• Open-source codebase available for community contribution
• Lightweight deployment suitable for personal benchmarking setups
• Comparative view showing how different providers stack up side by side

The creator candidly describes the project as 'for entertainment purposes,' noting that no thorough code review has been conducted yet. Still, for developers shopping between providers like OpenAI, Anthropic, Google, or various proxy services, even a rough speed comparison offers practical value.

Vibe Coding Gains Momentum as a Development Approach

Vibe coding — the practice of describing what you want and letting AI write all the code — has exploded in popularity in 2025. Coined by Andrej Karpathy earlier this year, the term captures a workflow where developers act more as product managers than engineers, guiding AI coding assistants through natural language prompts.

TokenMeter serves as a real-world example of what vibe coding can produce today. The developer used an AI coding plan to generate the entire project, from backend logic to frontend dashboard. While the result is admittedly rough around the edges, it demonstrates that functional developer tools can emerge from pure AI-assisted workflows in a matter of hours rather than weeks.

The Harder Problem: Measuring AI Quality

The creator raises an important point that speed alone doesn't tell the full story. Measuring response quality across AI providers remains an open challenge the developer says they are actively researching.

This mirrors a broader industry tension. Benchmarks like MMLU, HumanEval, and Arena Elo attempt to capture quality differences, but real-world API performance often depends on specific use cases, prompt structures, and context window utilization. A tool that could reliably measure both speed and quality in production environments would fill a significant gap in the developer tooling ecosystem.

What Comes Next

TokenMeter is early-stage, but it points to a growing need. As the number of AI API providers proliferates — from major players like OpenAI and Anthropic to dozens of inference providers and regional alternatives — developers need better tools to make informed decisions about which services to use.

For now, TokenMeter offers a simple, transparent way to compare one critical metric. The project is open source, and the developer appears open to community contributions that could push it beyond its current demo state.