Reading Notes

Personal reading notes. Original paper: arXiv:2606.11762v1.

arxiv_id: 2606.11762v1

Paper Reading Note: Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

1. Paper Overview

With significant progress made by Large Language Models (LLMs) in language understanding, reasoning, and generation, their creative potential has attracted widespread attention. However, existing creativity evaluation metrics are often tightly coupled with specific tasks and embed domain assumptions, limiting the scalability and generality of the evaluation.

To address this gap, the authors propose an automated, domain-independent framework for quantifying LLM creativity in open-ended tasks. The core of this framework lies in separating the measurement apparatus from the creative tasks themselves, thereby achieving scalable, task-agnostic evaluation. The paper argues that creativity comprises two dimensions: Divergent Thinking and Convergent Thinking, and designs corresponding automated evaluation methods for each.

2. Core Methodology

The authors decompose creativity evaluation into two independent parts: Divergent Creativity (measuring novelty and diversity) and Convergent Creativity (measuring task completion).

2.1 Divergent Creativity Evaluation: Semantic Entropy

Divergent thinking refers to the ability to generate diverse, novel, and innovative ideas. The authors hypothesize that divergent creativity is reflected in the diversity of reasonable semantic directions a model can explore in open-ended tasks. To quantify this attribute, the paper adopts Semantic Entropy (SE).

Semantic Entropy is a sampling-based, reference-free metric used to quantify the variability of model-generated outputs. Unlike traditional uncertainty-based hallucination detection, the authors reinterpret it as an operational definition of divergent creativity in open-ended generation.

The paper illustrates the calculation logic of semantic entropy through Figure 1. Model generation steps are clustered by similarity, and entropy is calculated over cluster probabilities.

As shown in the figure, the top image depicts a model with lower creativity, producing surface-level rewrites resulting in low semantic entropy; the bottom image depicts a model with higher creativity, producing truly distinct ideas resulting in high semantic entropy. The authors note that this method requires no reference ground truth and, validated by human annotation, LLM novelty judgment, and baseline diversity metrics, faithfully reflects the core markers of divergent creativity.

2.2 Convergent Creativity Evaluation: Retrieval-based Multi-Agent Framework

Convergent thinking involves synthesizing information to produce solutions tailored to specific goals and contexts. Evaluating this aspect is inherently subjective. To address this, the authors propose an adaptive, autonomous multi-agent LLM evaluation framework where agents collaborate to assess different aspects of task completion.

To solve the problem of low computational efficiency in traditional discussion-based evaluation methods, the authors introduce a retrieval-based discussion framework, which simplifies the review process and makes large-scale benchmarking more feasible. The paper claims this method improves efficiency by over 60% in context-sensitive evaluation of task completion.

Figure 2 shows the architecture of this retrieval-based multi-agent framework.

The framework includes three specialized LLM agents: Problem Agent, Solution Agent, and Judge Agent, which collaborate via a retrieval mechanism to efficiently evaluate model outputs.

3. Experimental Setup and Benchmarks

To demonstrate the generality and practical value of the framework, the authors validated it across three qualitatively different creativity domains:

1. Problem-solving: Using the MacGyver dataset.
2. Research ideation: Using the HypoGen dataset.
3. Creative writing: Using the BookMIA dataset.

These three domains capture complementary aspects of creativity: functional reasoning under constraints, logical synthesis of scientific ideas, and narrative originality.

Figure 3 shows an overview of the benchmarks for all three datasets.

The experiments applied 300 questions in each domain and benchmarked diverse LLMs. The authors analyzed how model size, recency, temperature, and reasoning enhancement affect creativity performance.

4. Results and Analysis

Empirical results indicate that the framework can reliably capture key aspects of creativity, including novelty, diversity, and task completion.

4.1 Semantic Entropy Analysis

The authors conducted an in-depth analysis of semantic entropy performance across different steps. Figure 4 shows the semantic entropy analysis results, where part (a) displays the relationship between step-level semantic entropy and the number of semantic clusters formed at each step.

This analysis validates the effectiveness of semantic entropy as a metric for divergent creativity, indicating a correlation between the number of semantic clusters generated by the model and entropy values, reflecting the model’s ability to explore different solution directions.

4.2 Impact of Model Properties

The paper further explores the impact of model properties on creativity performance. Figure 5 shows the impact of various parameters, including model size on the left, model recency in the middle, and reasoning capability on the right.

Experimental results reveal how model scale, training data recency, sampling temperature, and reasoning enhancement mechanisms specifically affect the model’s creative output. This provides empirical evidence for understanding the boundaries of LLM creativity.

4.3 Metric Comparison

To verify the effectiveness of the proposed method, the authors compared semantic entropy with different convergent creativity metrics. Figure 6 shows the comparison of semantic entropy with other convergent creativity metrics (Y-axis) on the MacGyver dataset.

The results show that the framework not only generalizes across different domains but also systematically analyzes the impact of model characteristics on creativity performance, establishing a reproducible and scalable automated LLM creativity evaluation standard.

5. Summary and Outlook

This paper proposes a fully automated, domain-general LLM creativity evaluation framework. The main contributions include:

1. Introducing reference-free, automated divergent creativity evaluation through Semantic Entropy.
2. Developing a computationally more efficient multi-agent LLM evaluation framework for assessing convergent creativity.
3. Conducting comprehensive empirical benchmarking on MacGyver, HypoGen, and BookMIA.

The authors believe this work paves the way for scalable benchmarking and will accelerate the progress of creative AI. Future work may involve further expanding evaluation domains and exploring potential connections between creativity evaluation and model alignment.