In my previous article, “In Silico Ideation,” I reviewed academic literature describing the application of LLM algorithms to generating new product ideas. Now, I want to review what is known about LLMs’ ability to generate other creative content. This article is about poetry (and short stories). 

Can You Tell Who Wrote That Poem?

If you think you can easily spot the difference between AI-generated and human-written poetry, think again. In a 2024 study by Porter and Machery, 1,634 participants were randomly assigned to evaluate poetry from 10 well-known poets and poems generated by ChatGPT-3.5 written in the style of each poet.

Guess what? The participants failed to tell the difference between the two sets. Even more surprising, they were more likely to mistake AI-generated poems for human work than the other way around. Moreover, ChatGPT-3.5-generated poetry not only passed as human-written but was rated higher for overall quality, rhythm, and beauty compared to works by famous poets.

The researchers call this the “more-human-than-human” effect. When people like a poem, they tend to assume it must have been written by a human. This bias plays out consistently across experiments, regardless of participants’ experience with poetry.

However, there was a twist: when explicitly told that a poem was AI-generated, participants rated it lower than when told it was human-written, revealing persistent biases against machine creativity.

Enhancing Human Creativity

AI isn’t just creating content on its own—it’s also changing how humans create it. A 2024 study by Doshi and Hauser found that prior access to a pool of AI-generated “seed” ideas improved the novelty and usefulness of human-written short stories by 6.7% and 6.4% respectively. Stories inspired by AI prompts were also rated as more enjoyable and better written.

The most intriguing finding? AI appears to be a great equalizer. Writers with lower measured creative abilities saw improvements of up to 11% when using AI “seed” ideas, effectively closing the gap between them and their more naturally creative peers. 

The Collaboration Sweet Spot

It also appears that generating creative content is more effective when humans collaborate with LLMs rather than when either party works alone. A 2023 study by Hitsuwari and colleagues found that while AI-generated haiku and human-made haiku were rated equally beautiful, AI-generated haiku with human intervention received the highest beauty ratings. 

Again, participants couldn’t reliably distinguish between human and AI authors. Moreover, the higher the AI-generated haiku was rated, the more likely people were to believe it was human-made.

The Diversity Angle

There’s a potential downside to AI-induced creative enhancement. The Doshi and Hauser study found that AI-assisted stories showed higher similarity to one another and the AI-generated prompts. This suggests a reduction in the diversity of creative output, raising questions about AI’s role in fostering true originality over time.

Implications for the Future

These studies collectively point to several important implications:

1. Indistinguishable creation: The line between human and AI creativity is rapidly blurring, at least for shorter creative formats like poetry.

2. Democratization of creativity: AI tools can help level the playing field, potentially allowing those with less natural creative talent to produce work of similar quality to highly creative individuals.

3. The collaboration advantage: The highest quality creative output may come from human-AI partnerships rather than either working independently.

As AI continues to evolve, so will our understanding of creativity itself. Rather than seeing AI as a replacement for human creativity, these studies suggest we might be moving toward a future where AI becomes an extension of human creative capabilities—enhancing, equalizing, and potentially transforming how we create art.

The question isn’t whether AI can be creative, but how our collaboration with LLM systems will reshape the very notion of creativity itself. I’ll come back to this topic in my future articles.