The Human Creativity Benchmark: A New Yardstick for AI Creative Capabilities

Filling the Gap in Creative Assessment

When ChatGPT can write poetry, Midjourney can paint, and Suno can compose music, a fundamental question emerges — how do we scientifically measure AI's creativity? Traditional AI benchmarks have largely focused on "hard skills" such as logical reasoning, mathematical computation, and knowledge-based Q&A. Yet for creative expression — a core domain of human intelligence — the academic community has long lacked systematic evaluation tools.

The Human Creativity Benchmark is a brand-new evaluation framework designed to address this very challenge. It attempts to establish a standardized methodology that quantitatively assesses generative AI's performance in creative work across multiple dimensions, providing reproducible and comparable scientific evidence for AI creativity research.

Core Design: Deconstructing "Creativity" Across Multiple Dimensions

The definition of creativity itself is deeply contested. The benchmark's core contribution lies in its decision not to produce a single "creativity score," but rather to decompose creativity into multiple independently assessable sub-dimensions.

Novelty: To what extent does AI-generated content transcend existing patterns found in training data? This dimension examines the model's ability to produce genuinely "never-before-seen" content, rather than simple recombination and stitching.

Usefulness & Appropriateness: Creativity is not random, unbounded output. Excellent creative work must carry practical value within a specific context. This dimension evaluates whether AI can demonstrate creativity while satisfying given constraints.

Surprise: True creativity often carries the quality of being "unexpected yet perfectly logical." The benchmark designs specialized test scenarios to measure whether AI can produce delightfully surprising answers rather than merely "correct" ones.

Diversity: Given the same task, can AI generate multiple solutions with distinctly different styles and approaches? This dimension directly reflects whether a model has fallen into fixed generation patterns.

The benchmark spans multiple creative domains including writing, visual design, music composition, and problem-solving, striving for comprehensive coverage of generative AI's creative application scenarios.

Why Existing Evaluation Systems Fall Short

Current mainstream AI benchmarks — whether MMLU, HumanEval, or various leaderboards — fundamentally measure "correctness." They presuppose standard answers or verifiable outputs, while the defining characteristic of creative work is precisely that "there is no single correct answer."

Previous assessments of AI creative ability have relied primarily on two approaches: first, manual review, where experts or general users subjectively score AI-generated work; second, directly applying creativity tests from psychology (such as the Torrance Tests of Creative Thinking) to AI. The former is costly and difficult to standardize, while the latter faces the fundamental challenge of whether tests designed for humans are applicable to AI.

The Human Creativity Benchmark seeks to strike a balance between the two — preserving the irreplaceable role of human judgment while enhancing comparability and reproducibility through a structured evaluation framework and quantitative metrics. Some dimensions also incorporate automated evaluation methods, such as semantic similarity analysis to quantify novelty and output distribution analysis to measure diversity.

Preliminary Findings: AI Creativity's "Ceiling" and "Blind Spots"

Although comprehensive evaluation results await further research validation, preliminary experiments based on this framework have already revealed several noteworthy trends:

Pronounced pattern tendencies: Current mainstream large language models exhibit significant "convergence" in creative tasks. Results generated across multiple runs show high similarity in structure, style, and even word choice — a stark contrast to the diverse expression of human creators.

Strong appropriateness but weak novelty: AI performs excellently at "completing tasks" — it can accurately understand creative briefs and deliver competent output. However, when it comes to truly breaking conventions and proposing unexpected creative solutions, a clear gap remains.

Significant domain differences: In language-intensive tasks such as writing, AI's creative performance is relatively strong. But in tasks requiring cross-modal thinking, metaphorical association, or deep cultural understanding, performance drops substantially.

These findings point to a key insight: current generative AI functions more like an "efficient creative executor" than a true "creative initiator." It excels at high-quality creative fill-in within established frameworks but still relies on human guidance when it comes to defining problems and breaking paradigms.

Industry Impact: From "Can It Be Used?" to "How Well Does It Perform?"

The release of this benchmark carries multiple implications for the AI industry.

For model developers, it provides clear direction for optimizing creative capabilities. Previously, improvements in the creative dimension lacked quantitative evidence, making it difficult for developers to determine whether architectural adjustments or training strategy changes truly enhanced creativity. The emergence of a standardized benchmark will change this dynamic.

For AI application companies, it helps more precisely define the boundaries of product capabilities. In industries highly dependent on creativity — such as advertising, content marketing, and game design — companies need to clearly understand where AI tools' creative "ceiling" lies, enabling them to design human-AI collaboration workflows accordingly.

For policymakers and the public, it provides an empirical foundation for discussions about AI creativity, helping move beyond the oversimplified narrative of "will AI replace human creators" toward more constructive dialogue.

Controversies and Limitations

It is worth noting that any attempt to quantify creativity inevitably faces scrutiny. Critics argue that creativity is deeply culturally embedded and historically situated, and that decomposing it into quantifiable dimensions may lose what is most essential. Furthermore, having human reviewers judge AI's creativity inherently presents the epistemological dilemma of "measuring non-human intelligence by human standards."

Other researchers have noted that the benchmark currently focuses primarily on "product-level creativity" — the creative quality of final outputs — while neglecting "process-level creativity" — the exploration, trial-and-error, and moments of insight during the creative generation process. The latter is arguably the most fascinating aspect of human creativity research.

Looking Ahead: Toward Scientific Assessment of AI Creativity

Despite the controversies, the emergence of the Human Creativity Benchmark marks an important pivot in the field of AI evaluation — expanding from a sole focus on the "instrumental" dimension of intelligence to a systematic exploration of its "expressive" dimension.

As generative AI continues to penetrate creative industries, establishing a scientific, fair, and actionable creative evaluation system is no longer an academic luxury but an industry necessity. It is foreseeable that more specialized benchmarks targeting specific creative domains — such as screenwriting, architectural design, and scientific hypothesis generation — will emerge in the future, collectively building a multi-layered AI creativity evaluation ecosystem.

In this era where AI and human creativity are deeply intertwined, we may ultimately discover that the process of measuring AI creativity is also a process of re-understanding the essence of human creativity itself.