日本語版はこちら
Ohakonban’nichiwa! I’m RYO from the Rikigaku Observation Institute!
“Which do people prefer: fiction written by ChatGPT or fiction written by a human?”
A study by researchers at Villanova University, recently covered by CNET Japan, produced an intriguing result: participants rated the ChatGPT-generated stories more highly than the human-written ones.
In the first experiment, 1,682 adults aged 18 to 81 evaluated short stories written either by humans or by ChatGPT, rating their quality and how absorbing they found them. The AI-generated stories received higher ratings overall. Yet another effect appeared at the same time: stories described as “human-written” were rated more favorably, regardless of who had actually written them.
In subsequent experiments, participants were asked to distinguish human-written stories from AI-generated ones. Their performance showed that telling the two apart was far from straightforward.
So far, so interesting. But while reading the Japanese article, something else caught my attention.
In Japanese, the AI Version Really Is Easier to Read
The CNET Japan article includes Japanese translations of excerpts from the stories used in the study.
In the AI-generated story, the narrator sits beside a pond, remembers her mother, watches autumn leaves fall and koi swim beneath the surface, and reflects on change and constancy in life.
Mother / pond / autumn / falling leaves / koi / water / change / comfort.
The prose may have that slightly familiar ChatGPT tendency to wrap things up a little too neatly, but its meaning comes across immediately.
The human-written story, by contrast, develops a metaphor around childbirth. One character is described as being in labor; the narrator casts herself as a midwife; the husband becomes an anxious father; and the metaphor expands toward the idea of a sacred event.
Of course, these are only excerpts from longer works, so it would be unfair to judge either story as a whole from these passages alone. But when I read the two excerpts in Japanese, I found the AI-generated one noticeably smoother and easier to follow.
The human-written passage felt different. In Japanese, it had the flavor of an older translated novel—or, to exaggerate slightly, something produced by an earlier generation of machine translation.
There is a particular kind of translation effect that Japanese readers sometimes encounter in American films, advertisements or tech presentations: a perfectly ordinary English phrase crosses into Japanese and somehow comes out sounding grand, solemn and vaguely philosophical.
To invent an exaggerated example:
“This is not merely a device. It is a new way to experience tomorrow.”
Perfectly plausible in an English-language presentation. Translate that too literally into Japanese, however, and suddenly it sounds as though Apple has started writing philosophy.
The human-written excerpt gave me a little of that feeling.
But Wait—These Stories Weren’t Written in Japanese
And then an obvious point occurred to me: the experiment was conducted in English.
Japanese readers of the CNET Japan article are therefore not reading exactly what the participants in the study read. What reaches us has already traveled through another process:
English original → Japanese translation → Japanese reader
That distinction matters particularly when the subject is fiction. In a news report, a translation can often do its job as long as factual information—dates, numbers, events and statements—is transferred accurately. Fiction is different. Word order, rhythm, ambiguity, metaphor and cultural association can all be part of the work itself.
A metaphor that feels natural and understated in English may become conspicuous, overly dramatic or strangely explicit when its structure is carried directly into Japanese.
So by the time I compare these two excerpts in Japanese, I am no longer observing only ChatGPT vs. human. I am also observing ChatGPT in English → Japanese translation versus human writing in English → Japanese translation.
There is another lens between the original text and me: translation.
The AI Wasn’t Writing from Nothing
Looking at the original paper reveals another important detail. The researchers used three human-written stories and asked GPT-4 to generate a corresponding story for each one.
For example, the AI story Reflections in Still Water was paired with the human-written story FISH. GPT-4 was given fairly specific instructions involving themes such as life and death across generations, uncertainty, koi as a symbol, and the perspective from which the story should be told.
In other words, the researchers did not simply tell GPT-4:
“Write me a story.”
Themes, symbols, narrative perspective and other elements were extracted from the human-written works and used to construct corresponding prompts for the AI.
That is a reasonable way to make the stories comparable in an experiment. But when the study is reduced to the popular question “Which writes better fiction, AI or humans?”, this experimental condition is worth remembering.
Using Saussure as a Measuring Stick
This is where Ferdinand de Saussure becomes useful—not as a subject for a linguistics lecture, but as a tool for observation.
To simplify his theory considerably, Saussure described the linguistic sign through the relationship between the signifier—the form of a word or expression—and the signified—the concept it evokes.
Now look again at the AI-generated passage.
Mother / pond / autumn / falling leaves / koi / water / change / comfort.
These signs connect to their meanings in a relatively straightforward way. Leaves fall in autumn / koi swim beneath the water / the narrator remembers her mother / something constant offers comfort amid a changing life.
Of course, none of these meanings is completely independent of language or culture. But the relationships among them seem relatively easy to preserve when the passage moves from English into Japanese.
The human-written passage works differently:
Childbirth → labor → midwife → anxious father → sacred event.
Here, meaning develops through an extended metaphor. The effect depends not only on what each individual word signifies, but also on the network of associations created among those words within a particular linguistic and cultural context.
That network may not survive translation in exactly the same form. The words can all be translated correctly, and the tone can still shift. What felt literary in English may become unusually explicit in Japanese. A metaphor may remain perfectly understandable while becoming heavier, more conspicuous, or simply more “translated.”
In other words, translation does not merely replace one signifier with another. It has to reconstruct relationships among signs in another linguistic system.
And that raises another question.
Are AI-Generated Texts More “Translation-Resistant”?
From this point on, I am no longer describing a finding from the Villanova study. This is a hypothesis that occurred to me while reading the Japanese translations.
Perhaps AI-generated prose is not simply easier to read because it is more direct. Perhaps it also tends to preserve its semantic relationships more easily when moved from one language to another.
Large language models learn from enormous quantities of text. In doing so, they may gravitate toward patterns and structures that recur across many examples of language. That can certainly be a weakness: AI prose can feel averaged out / less idiosyncratic / overly polished / strangely familiar.
But turn the same characteristic around, and it suggests another possibility: those more widely shared structures may also be easier to carry across languages.
Human literary writing can derive much of its richness from exploiting the peculiarities of a particular language, culture, voice or network of associations. Precisely because those relationships are so specific, some of that richness may be difficult to reproduce elsewhere.
AI-generated prose may sacrifice some of that specificity. But could the same sacrifice make it more portable?
Less dependent on a particular system of signifiers → less lost when those signifiers have to change?
I don’t know. And the Villanova study does not answer that question.
Testing it would require a different experiment—one designed specifically to compare how human-written and AI-generated texts behave across translation. But that is exactly why the Japanese version of the article interested me. Translation may have introduced a new variable that the original experiment was never designed to examine.
The Researchers Themselves Don’t Say “AI Won Because It’s Easier to Read”
There is another point worth keeping in mind. Popular coverage naturally tends to focus on a simple explanation: AI-generated stories may have been preferred because they were more direct, concise and easier to understand.
The original paper is more cautious.
The researchers discuss ease of interpretation as one possible explanation for the higher ratings, not as a conclusion established by the experiments. They also point out an obvious complication: literary fiction is not necessarily “better” simply because it is easier to understand. Ambiguity, complexity and room for interpretation can be part of what gives a story its value.
The later experiments produced an even more interesting result. Participants who relied on wording as a clue to authorship tended to be worse at identifying whether a story had been written by a human or by ChatGPT. In one experiment, participants who used their own enjoyment of a story as a clue were also more likely to get the answer wrong.
In other words, an intuition such as “This is easy to read, so it must be AI” may not help us identify AI writing at all. It may even push us in the wrong direction.
That makes the result more interesting, not less.
How Long Will “AI vs. Human” Remain a Useful Comparison?
There is a broader problem with the question itself. In a controlled experiment, separating “AI-written” from “human-written” text makes perfect sense. Outside the laboratory, however, that boundary is already becoming difficult to maintain.
Consider professional shogi. Today’s top players study moves suggested by AI, including moves that previous generations of human players might have considered unnatural or even poor. They examine the reasoning behind those moves, understand their value, and incorporate what they learn into their own play.
When a professional later plays such a move in an actual match, whose move is it? The human’s? The AI’s? The question quickly becomes awkward.
Something similar happened long ago with spreadsheets. Before software such as Microsoft Excel, enormous amounts of human time were spent performing calculations and organizing data manually. Today, we do not normally look at a spreadsheet produced with software and say, “This is not human work because a computer calculated it.”
The software has become part of the human workflow. Generative AI may be moving in the same direction.
Human → AI → Human
Writing is already beginning to look like this:
A human develops the idea → ChatGPT produces a draft → the human spots what feels wrong → the AI generates alternatives → the human rejects some, keeps others and rewrites the result.
Who wrote the finished text?
“The human” and “the AI” are both incomplete answers.
The more interesting change may therefore be not that AI is becoming capable of writing better stories than humans, but that humans and AI are beginning to alter one another’s output.
Humans learn from AI. Humans incorporate those techniques into their own writing. AI systems, in turn, learn from human-produced language in an environment that is itself increasingly influenced by AI.
Human → AI → human → AI.
Shogi offers an early example of this cycle. What begins as an “AI move” can eventually become part of ordinary human theory.
If something similar happens to writing, the clean boundary required by the question “AI or human?” may become increasingly artificial.
Perhaps the more useful question will eventually be not “Who wrote this?”, but “What forces shaped the text that ended up in front of us?”
English → Japanese (translator unknown) → ChatGPT → English.
At this point, even Saussure might ask for a system update.😂
