---

Abstract #

Oakhollow and Quarthaven, the two browser games of this series, treat human players and language-model characters as the same kind of entity, and the client is never told which is which. We tested whether that holds from the player’s side. First, we audited every channel a player can observe in the shared engine and measured each one on a private copy of Quarthaven. Second, four LLM judges and a surface-feature classifier labelled 118 replies, each written either by a human crowdworker from the LIGHT fantasy dialogue corpus or by our unchanged NPC mind for the same context. Third, we traced the provenance of every logged “human” session. The game interface identifies humans with 100% accuracy: the relations matrix leaves a human’s row empty (126 of 126 rows), and sign-up refuses all 12 NPC names. Text alone gives much less away. The best judge reached 62.93% (95% CI 53.86-71.17%), but a 14-feature classifier reached 75.42% (66.94-82.31%), mostly from punctuation and length. Timing leaks too: 99.07% of 429 NPC lines follow the engine’s fixed typing rate of 90 ms per character to within 0.25 s. Finally, none of the 21 logged sessions comes from a confirmed independent player: the 14 human lines in the logs were written by the developer. The games are not yet indistinguishable, and the leaks lie in the interface and timing, not in the language.

1. Introduction #

The previous article in this series measured what zero-script characters cost and how they react when every step, word and blow comes from one model decision [1]^[2]. It built on a classification of AI integration surfaces that placed our characters on an authoritative server, receiving structured perceptions [2]^[3]. One design promise has not yet been tested. In both games a human joins as a resident like any other, nobody is labelled, and the engine’s own comment says that clients “never learn which is which”. If the promise holds, a player can never be sure whether the mage across the plaza is a person or a model. If it fails, it fails in a specific place, and that place can be fixed.

Turing-style studies now report that a well-prompted model can be judged human more often than real humans in five-minute text chats [3]^[4]. A game world, however, shows much more than text. Players see positions, speeds, icons above heads, opinion panels and names, and they see when someone answers. The question is therefore not whether the model’s sentences sound human. It is whether everything around the sentences does too.

RQ1: Which channels of the shared engine reveal whether a resident is human or NPC, and how accurately does each one identify the humans when measured from the player’s side? RQ2: How accurately can LLM judges and a simple feature classifier tell our NPCs’ lines from human lines, in matched replies and in live scenes with timestamps? RQ3: How much of the “human” traffic in the live logs comes from real, independent players, and what does that evidence support?

2. Existing Approaches (2025–2026 state of the art) #

2.1 Interactive Turing tests #

Jones and Bergen ran two pre-registered three-party tests in which interrogators chatted for five minutes with a human and a model at the same time. With a humanlike persona prompt, GPT-4.5 was chosen as the human 73% of the time, significantly more often than the real humans it was compared with [3]^[4]. In their earlier public test, interrogators decided mainly on linguistic style (35%) and socio-emotional traits (27%), not on knowledge or reasoning [4]^[5]. These tests restrict the judge to text, with no world around the speakers; a game cannot.

2.2 Displaced and inverted judges #

In most games a player overhears more than they interrogate. Rathi et al. gave Turing-test transcripts to GPT-3.5, GPT-4 and displaced human readers. All three were less accurate than the live interrogators, below chance overall, and all three judged the best GPT-4 witness human more often than the human witnesses [5]^[6]. Kabbach argues that a pass requires imperfect, “normal” behaviour and that the verdict belongs to a jury of judges, not to one judge [6]^[7]. Hence we use several judges and read their biases as well as their accuracy.

2.3 Process over output #

A second line of work looks at how behaviour is produced rather than at what is produced. The Process Turing Test of Rmus et al. separates humans from frontier systems through process-level features of cognitive tasks, with a classifier AUC of 0.88 even when task performance is matched [7]^[8]. Mitropolsky et al. generalise the test to arbitrary interactive systems and find that distinguishers use stylistic signatures as well as substantive probes [8]^[9]. Keystroke dynamics separate assisted from genuine writing better than text-only detectors do [9]^[10]. The same signal can be forged, though. Timing-forgery attacks that sample inter-key intervals from human distributions evade such detectors in at least 99.8% of cases [10]^[11]. For a game, the relevant processes are typing, movement and response timing.

2.4 Behavioural fingerprints in online games #

Anti-cheat research separates automated players from people by behaviour; recent work in multiplayer online battle arenas shows that behavioural fingerprints within one account are distinguishable from those across accounts, even with few labels [11]^[12]. Our problem is the inverse of cheat detection: a game that wants its automated residents to be mistaken for people has to remove exactly the fingerprints that anti-cheat systems look for.

2.5 NPC studies with players #

Generative agents showed that model-driven characters can produce believable daily life in a sandbox town [12]^[13]. Believability is a weaker requirement than indistinguishability: a believable character can still be recognisably artificial. In a randomised study with 130 players, model-driven NPCs raised cognitive load and lowered perceived usability and trust compared with scripted NPCs [13]^[14]. In virtual reality, a behavioural filler during response delays improved perceived response time and humanlikeness, while symbolic indicators such as a thinking bubble did not improve the impression of the character [14]^[15]. Another VR study measured a 7 s speech-to-speech cycle and a believability score of 6.67 out of 10 for two GPT-4 Turbo characters [15]^[16].

flowchart TD
    T[Is this resident human?] --> A[Interactive: judge talks to the resident]
    T --> B[Displaced: judge reads a transcript]
    T --> C[Process: judge sees how behaviour is produced]
    T --> D[Interface: judge reads panels and rules of the game]
    A --> A1[Text style, socio-emotional cues]
    B --> B1[Our scene test with LLM judges]
    C --> C1[Typing rate, reply latency, speed]
    D --> D1[Relations rows, profiles, name rules, looks]

None of these studies measures a shared world in which humans and model-driven residents move, fight and speak under the same rules. The interface channel is specific to games and, as Section 4 shows, decisive.

3. Method and Data #

3.1 Channel audit #

We read the shared engine (simulation, mind, server, client) and the two game definitions, and listed everything the server sends to a client or exposes over HTTP: state frames at 10 Hz (position, health, activity icon, faction flag, looks), chat, voice, death notices, the relations matrix, profiles, the token log and the public statistics. The live standalone Oakhollow build uses the same constants for speed, typing, dozing and the relations matrix as the shared engine.

3.2 Engine run from the player’s side #

On 2 November 2026 we started a private copy of Quarthaven on a spare port, with its own log directory, speech synthesis disabled and the engine code unchanged. Two scripted player seats joined with an OpenRouter key: one moved with direction keys, the other by clicking, and both greeted the nearest mage from a short list of fixed lines. For 150 s, seat A recorded every state frame, every chat line it heard, nine relations snapshots and every resident’s profile, exactly as a browser receives them. At the end, a further connection tried to sign up under each of the 12 NPC names, under three free player names and under one name already in use. The world made 79 model calls.

3.3 Typing delay #

Every NPC line in the zero-script logs of article 2 and in this run was paired with the decision that produced it, 429 lines in all. The engine shows a writing icon and holds the line for 90 ms per character, clamped to 1.5-9 s, so the delay between decision and speech can be compared with that rule line by line.

3.4 Matched-reply judge test #

Live human data being scarce (Section 4.3), we used human-written in-character dialogue as a proxy. LIGHT is a crowdsourced fantasy text adventure in which humans play characters in grounded dialogue [16]^[17]. We sampled 60 test dialogues (seed 20261103) and picked one turn at position 2, 4 or 6 in each. The human reply is the crowdworker’s actual turn. The NPC reply comes from our engine’s own decision function, with the same rules, prompt layout, model (gemini-2.5-flash-lite), temperature 0.9 and JSON schema as in the games. The setting becomes the lore, the persona becomes the NPC persona, earlier turns become memories and the partner’s last line becomes the event. One NPC reply was empty, which left 59 pairs, or 118 replies.

Four judges labelled each reply as human or AI, one reply per call, at temperature 0: gemini-2.5-flash-lite, mistral-small-3.2, gpt-4.1-mini and claude-sonnet-4.5 as the stronger one. They saw the setting, up to four previous turns and the reply, and were told that about half of the replies were human. In parallel, a logistic regression on 14 surface features (length, punctuation density, lowercase start, missing end punctuation, ellipses, contractions without apostrophes, odd spacing, word overlap with the partner’s line and similar) was evaluated with 5-fold cross-validation grouped by dialogue. We report accuracy with 95% Wilson intervals and a two-sided binomial test against 50%.

3.5 Scene judge test #

The same four judges labelled every speaker in seven chat transcripts with times. Four natural scenes come from the live Oakhollow logs of 23 and 24 October 2026 and contain the only hand-typed or spoken human lines we have. They were recorded under the earlier v0 and v4 builds, which still had routines. Three probe scenes are the first three minutes of each article-2 probe run on the zero-script engine, in which the “players” were scripted bots repeating fixed template lines. We report the probe scenes as rates of “called human”, not as accuracy, because their players were not human.

3.6 Provenance #

Each of the 21 joins in the live logs between 23 October and 2 November 2026 was matched against the timestamps of development test scripts that opened a websocket to the game, and against requests the developer wrote minutes after a manual session. The analysis keeps only in-game pseudonyms; no IP addresses were used. All experiments together cost USD 0.308497 in model usage, against a cap of USD 1.50.

Code, data and outputs: stabilarity/hub/research/living-npcs-can-players-tell-who-is-human-an-indistinguishability-test-on-zero-script-llm-villagers.

4. Results #

4.1 RQ1: The interface gives humans away completely #

Table 1 lists every channel in which the engine treats the two kinds of resident differently, with the measured or code-derived strength of the cue.

Channel	What differs	Measured value	Strength
Relations matrix	Only NPCs have opinion rows; a human’s row is empty	126 of 126 rows classified correctly over 9 snapshots	Complete
Profile panel	Only NPCs have outgoing opinions	14 of 14 profiles	Complete
Sign-up names	NPC names are reserved; player names come from a separate pool	12 of 12 NPC names refused, 3 of 3 free player names accepted; pool overlap 0 in both games	Complete
Lore	The default lore names residents	7 of 10 NPCs named in Oakhollow, 4 of 12 in Quarthaven	Partial roster
Looks (Oakhollow)	NPC tunic colours are outside the player palette	0 of 10 NPC colours selectable by players	Complete in 2D
Movement speed	Players 3.2 tiles/s, NPCs walk at 2.3 and run at 3.1	Median 3.2 vs 2.3 tiles/s; 6 of 6 moving residents classified correctly	Strong while NPCs walk
Typing delay	NPCs hold a line for 90 ms per character	425 of 429 NPC lines within 0.25 s of the rule	Strong
Thinking icon	NPCs show it for the model call, including silent decisions	Median 0.801 s over 53 icons; 9 of 79 decisions produced no line	Moderate
Line rate	Humans may speak every 1.2 s; NPCs deliberate at most every 7 s	Code constants	Moderate
Dozing	Idle humans show a sleep icon after 60 s at any hour	Code constant	Moderate
Death	A dead NPC returns after 30 s under the same name; a dead human leaves	Code constants	Strong when observed
Melee damage	Fixed for players; per persona for NPCs	No Oakhollow NPC shares the player value of 10; 1 of 12 Quarthaven NPCs shares the value of 8	Weak

Source: our code audit and engine run of 2 November 2026; typing over the zero-script logs of article 2 and this run.

Four channels identify humans without error, and none of them involves language. The relations panel is the clearest case, and the interface says so itself. Its help text reads “People keep their thoughts to themselves, so their rows are empty”, and a profile without outgoing opinions says that the person “keeps their thoughts to themselves”. Sign-up is a second oracle: any player can test whether a name belongs to a resident by trying to take it. In Oakhollow, every NPC also wears one of ten tunic colours that no player can choose.

The process channels are nearly as strong. Chart 3 plots the delay between decision and speech against line length for all 429 NPC lines. The median absolute deviation from the rule is 0.035 s. The 395 lines between the clamps were held a median of 89.99 ms per character, 33 lines sat at the 1.5 s minimum and 1 at the 9 s maximum. Only 4 lines deviated, all by 39.94 to 64.05 s. A player who watches the writing icon above a resident can read off the length of the coming line, and every NPC “types” at exactly the same speed. Human typing varies from person to person and from one line to the next, so this regularity is the kind of process signal that keystroke-based detectors exploit [9]^[10].

Movement separates the two almost as cleanly (Chart 4). Between consecutive state frames, the player seats moved at a median 3.2 tiles/s and the NPCs at 2.3 tiles/s. 90.9% of player frame pairs and 5.0% of NPC frame pairs were faster than 2.75 tiles/s. The cue weakens when an NPC runs to attack, flee or catch up: a player is only 3.2% faster than an NPC running at 3.1 tiles/s.

Reply latency was not a reliable cue: the developer answered after a median 10.674 s (9 replies), the zero-script NPCs after 7.324 s (126 replies), and the shares under 4.5 s (settle time plus minimum typing time) were 22.22% and 23.02%. NPCs can beat the gate when they had already decided to address the person.

flowchart LR
    S[Server state per tick] --> F[State frames: position, icon, looks, flag]
    S --> C[Chat and voice]
    S --> P[Relations matrix and profiles]
    S --> N[Sign-up: name accepted or refused]
    F --> L1[Speed 3.2 vs 2.3, writing icon 90 ms per char, sleep icon]
    C --> L2[Line length, punctuation, timing]
    P --> L3[Empty human rows, no outgoing opinions]
    N --> L4[NPC names reserved]
    L3 --> X[Human identified with certainty]
    L4 --> X
    L1 --> Y[Human identified with high probability]
    L2 --> Z[Weak to moderate signal]

4.2 RQ2: Text alone is a weak signal, but style still leaks #

Chart 1 shows the matched-reply results. Three judges were within chance: gemini-2.5-flash-lite reached 58.93% (95% CI 49.67-67.6%, p = 0.07213), mistral-small-3.2 55.08% (46.09-63.76%) and gpt-4.1-mini 54.24% (45.26-62.95%). Claude-sonnet-4.5 reached 62.93% (53.86-71.17%, p = 0.006832, AUC 0.6801), and the majority of four judges 62.73% (53.41-71.19%). The feature classifier did better than any judge, at 75.42% (66.94-82.31%, AUC 0.8337), and reply length alone gave 68.64%.

Judge	Accuracy (%)	95% CI (%)	Human replies called human (%)	NPC replies called human (%)	Replies answered
gemini-2.5-flash-lite	58.93	49.67-67.6	52.7	35.1	112
mistral-small-3.2	55.08	46.09-63.76	13.6	3.4	118
gpt-4.1-mini	54.24	45.26-62.95	88.1	79.7	118
claude-sonnet-4.5	62.93	53.86-71.17	86.2	60.3	116
Majority of 4	62.73	53.41-71.19	51.9	26.8	110
Feature classifier	75.42	66.94-82.31	n/a	n/a	118

Source: our experiment, 3 November 2026; human replies from LIGHT test dialogues [16]^[17]. gemini-2.5-flash-lite left 6 replies unanswered after truncated output; 34 of its verdicts were read from truncated JSON.

The judges also differ in bias. Mistral-small-3.2 called almost everything AI, and gpt-4.1-mini called almost everything human. Claude-sonnet-4.5 accepted 60.3% of NPC replies as human. Their explanations named “formal” 84 times, “generic” 51 and “emotion” 46, but “typos” only twice.

The classifier shows what the judges only partly used. The strongest feature was punctuation density (coefficient -1.167: more punctuation per character means NPC). Next came odd spacing (+0.813, human), word overlap with the partner’s line (-0.792, NPC), missing end punctuation (+0.749, human) and ellipses (+0.663, human). The style figures make the difference concrete. The LIGHT humans wrote a median of 73 characters and left 18.6% of replies without end punctuation. The engine’s NPC wrote 46 characters and did so in 1.7% of replies. Across 310 zero-script NPC lines in the probe logs, not one started with a lowercase letter, against 28.6% of the developer’s 14 live lines. Our system prompt asks for “usually one short sentence”, and the model complies with clean, punctuated, slightly echoing sentences that are easy to tell from casual typing.

The scene test adds timestamps and context. In the four natural scenes the pooled accuracy was 58.33% (95% CI 50.17-66.07%, 144 speaker judgments). Claude-sonnet-4.5, however, found the human in all four scenes and called only 2 of 32 NPC speaker labels human, for 94.44% (81.86-98.46%). Its stated cues were the developer’s lowercase fragments (“help pls”) against full medieval sentences. Gemini-2.5-flash-lite and mistral-small-3.2 scored only 33.33% each and called most NPCs human.

The probe scenes reproduce the inversion of displaced tests [5]^[6]. Claude-sonnet-4.5 called none of the 12 template-scripted players human and 30 of 34 zero-script NPCs human (88.2%), noting that four players “used identical scripted phrases”. Pooled over the four judges, 88 of 136 NPC labels (64.7%) and 19 of 48 template-player labels said human. In the engine run, a mage noticed the same thing about a repetitive seat and remarked: “You ask that a lot, Caelum.” An NPC that talks in its own words is more convincing than a player who repeats himself. That makes template bots a poor stand-in for human players in any future test.

4.3 RQ3: There are no independent players in the logs yet #

Class	Sessions	Lines typed or spoken	Evidence
Scripted test	11	5	A test script joined within seconds, or the name is hard-coded in the test scripts
Connection check	4	0	Joined and left within 1 s without any action
Developer, manual	4	14 (10 by speech-to-text)	Feature requests written minutes after the session describe that play
Unknown	2	0	Two short Quarthaven visits, 73 s and 20 s, spells only, no model call
Confirmed independent player	0	0	—

Source: live Oakhollow and Quarthaven logs, 23 October to 2 November 2026; our provenance analysis.

Of 21 logged sessions, none can be attributed to an independent member of the public. Every hand-typed or spoken line in the logs came from the developer, under builds that still had routines, and the zero-script Oakhollow build has received a single join since deployment: the smoke test. The two unknown Quarthaven visitors did not speak, and no model call was made while they were present, so the NPCs could not have answered them at all, because a world without a player key does nothing. The judge results in Section 4.2 therefore rest on a proxy (LIGHT crowdworkers) and on one person’s play. They are evidence about the text channel, not about how real players would judge the villages.

5. Discussion #

What the games actually hide. The language model is the best-hidden part of the system. Even the strongest judge was right only 62.93% of the time on matched replies, while the interface identified humans every time. Engineering hygiene, not model quality, is what separates our games from indistinguishability. The same split appears in process-based detection, where how something is done beats what is produced [7]^[8].

Design fixes, recommended and not yet implemented. Each leak in Table 1 has a concrete remedy:

• Relations and profiles: give every human synthetic outgoing opinions, derived from what they did (lines spoken, blows struck, help given) with the same clamping and drift as NPC opinions, and remove the help text that explains empty rows. Alternatively, hide every resident’s outgoing row unless it is the viewer’s own.
• Names: merge the two name pools, release NPC names for players in private worlds, and stop listing residents in the default lore, or list players there too.
• Looks: draw NPC tunics from the player palette in Oakhollow.
• Movement: one speed table for everyone. Players walk at the NPC walking speed and run on a key, at the NPC running speed.
• Typing and thinking icons: sample each NPC’s typing rate per line from a spread of human rates instead of a constant, and add pauses. Show the thinking icon for humans when they open the chat box (as now), and for NPCs only when they will speak. Timing forgery of exactly this kind defeats keystroke-based detectors [10]^[11]. The VR evidence suggests a gestural filler would help more than an icon [14]^[15].
• Death and dozing: let dead humans respawn under the same name after the same 30 s, and let idle NPCs doze on the same 60 s rule.
• Style: ask for typing as players type, with occasional lowercase starts, missing end punctuation and less echoing of the partner. A humanlike persona prompt is what moved GPT-4.5 past the human witnesses [3]^[4], and normal imperfection is the target [6]^[7].

flowchart TD
    G[Goal: no channel separates humans from NPCs] --> I[Interface]
    G --> P[Process]
    G --> T[Text]
    I --> I1[Synthetic outgoing opinions for humans]
    I --> I2[One name pool, one palette]
    P --> P1[One speed table]
    P --> P2[Sampled typing rate and pauses]
    P --> P3[Same respawn and doze rules]
    T --> T1[Casual typing style in the prompt]
    T --> T2[Less echo of the partner line]

Event-driven social simulators already emulate human-like temporal patterns [17]^[18]. Our results suggest that a game which mixes humans and NPCs should do the same.

Should the games be indistinguishable at all? Article 50(1) of the EU AI Act requires providers to ensure that people interacting directly with an AI system are informed of it, unless this is obvious to a reasonably well-informed person given the context [18]^[19]. Both game pages already state that model-driven residents live side by side with people and that nobody is told who is who, so the open question is the individual resident, not the existence of AI. Closing every leak is a research configuration. A public deployment may prefer one honest, optional label per resident. Players’ trust in model-driven NPCs is already fragile [13]^[14], and the engine should make that label a switch rather than an accident of the code.

Limitations and threats to validity. The matched-reply test uses crowdworkers writing for a dataset in 2019, not players of our games. They wrote longer replies than our players and in a different register, so the classifier partly detects that register. The natural scenes contain one human per scene, always the developer, and earlier builds whose NPCs spoke far more often than the zero-script ones. All judges were language models reading transcripts, and displaced readers do worse than interactive interrogators [5]^[6]. Our seats were scripted and never sent activity icons, so human icon timing was not measured. The engine run lasted 150 s, with 6 residents moving long enough to be measured. One NPC reply was empty, and gemini-2.5-flash-lite truncated 40 judgments, 34 of which we could still read.

What would change the conclusion. Real players typing in the zero-script villages, judged interactively by other players, would replace both proxies. If the interface and timing leaks were fixed, the text channel, at 62.93% for the best judge and 75.42% for a simple classifier, would become the binding one. A persona prompt tuned for casual style could move it toward chance.

6. Conclusion #

RQ1 Finding: The interface, not the language, gives humans away. Measured by the accuracy of each channel at separating humans from NPCs, the relations matrix scored 100% (126 of 126 rows), profiles 100% (14 of 14), sign-up names 100% (12 of 12 NPC names refused), Oakhollow tunic colours 100% (0 of 10 shared) and movement speed 100% of 6 moving residents (3.2 vs 2.3 tiles/s). The typing rule held for 99.07% of 429 NPC lines. This matters for the series because the cheapest gains in believability are interface fixes, not better models.

RQ2 Finding: Text alone is a weak but real signal. Measured by accuracy on 118 matched replies, the best LLM judge reached 62.93% (95% CI 53.86-71.17%), three judges were within chance, and a 14-feature classifier reached 75.42% (66.94-82.31%). In scenes with timestamps, the strongest judge found the human in 4 of 4 developer scenes (94.44%) but called 88.2% of zero-script NPCs human against template-scripted players. This matters for the series because NPC prose already outperforms scripted test players, so the remaining text cue is style (punctuation, length, echo), which can be controlled in the prompt.

RQ3 Finding: The live logs contain no independent players yet. Measured by session provenance, 0 of 21 sessions are confirmed independent: 11 are scripted tests, 4 connection checks and 4 developer sessions with 14 human lines, and 2 unknown visits contain no line at all. This matters for the series because any claim about real players must wait for real players, and test traffic must never be reported as such.

The next question for the series is whether players can still tell once these leaks are closed, measured with people playing against each other and against the villagers.