---

Abstract #

Where should a language model attach to a game, and which faculties of a character should it control? We propose a two-axis classification: the integration surface (a mod of a shipped game, an engine SDK, an agent on an authoritative server, or an agent that reads pixels and presses keys) and seven augmentation types graded 0 to 3 (dialogue, vision, hearing, memory, time perception, action, social state). We coded 33 public systems, from Skyrim mods to Generative Agents, Project Sid, SIMA 2 and NVIDIA ACE titles, and measured the decision step itself by sending a structured villager perception to six low-cost models in 144 calls. Three numbers frame the design space. The median model needs 1.113 s for a complete decision, about 22 ticks of a 20 Hz server, so the model cannot sit inside the game loop. A 120-entry memory log barely changes latency (median ratio 1.017) but multiplies cost by a median of 6.162. And explicit social state is the emptiest cell of the map: only 3 of 33 systems (9.1%) keep an opinion or relationship variable. The series will therefore build a server-authoritative 2D browser village in which model-driven characters act through a perception, decision and action interface and an opinion score decides whether they attack.

1. Introduction #

Mantella lets a Skyrim or Fallout 4 player speak to more than 3,000 non-player characters (NPCs) through speech-to-text, a language model and text-to-speech [1]^[2]. Engine vendors sell the same capability as SDKs, and research groups run hundreds of model-driven agents on Minecraft servers [2]^[3]. A practitioner who wants characters that “live their life” in a game must choose between these routes. Surveys organise the field by the role a model plays in a game [3]^[4] or by agent architecture and genre [4]^[5]. Neither answers the engineering question of where to attach the model and what it should control.

Two observations motivate a different cut. The integration surface fixes what the model can see and change before any prompt is written: a mod sees what the script hooks expose, a pixel agent only the screen. And the faculties people mean by “intelligent NPC” are separable: a character can talk fluently and remember nothing, or fight competently without a word. Time perception, the question of how often a character takes in the world and when it is allowed to act, is the faculty most often left implicit, although it decides whether a slow model is usable at all.

RQ1: Along which axes can AI integration for game characters be classified, and where do 33 public systems fall on them? RQ2: How long does one model-driven NPC decision take in seconds and in game ticks, what does it cost, and how reliable is its structured output? RQ3: Which combination of surface and augmentations is the simplest credible testbed for characters that live, talk and fight in a shared world?

Article 1 classifies; article 2 builds and measures the testbed chosen here.

2. Existing Approaches (2025–2026 state of the art) #

2.1 Modding a shipped game #

The cheapest way to get a talking NPC is to patch a game that already exists, and Bethesda titles are the reference case. The CHIM server, formerly Herika, describes itself as the bridge between an SKSE plugin and AI providers for speech recognition, speech synthesis and chat, and lists long-term memory, “deep world awareness” and function-calling commands such as “trade with me” or “attack that monster” [5]^[6]. Mantella’s current documentation claims persistent memory, awareness of in-game events, NPCs that “can see what you see”, and more than 20 actions including follow, attack, trade and travel [1]^[2].

Modding inherits the game’s art, world and combat for free, which is why it dominates hobby work. Its limits are structural: the model can only call actions the mod author has wired to the game’s existing AI, perception is whatever the hooks report, and each patch is tied to one game build. Latency hides behind conversation, because players tolerate a pause when a human would pause too.

2.2 Engine SDKs and plugins #

When the developer owns the source, the model can be integrated inside Unity or Unreal Engine as a component. Three families exist. Reinforcement-learning toolkits, led by Unity ML-Agents, let games and simulations serve as environments for training characters with deep reinforcement learning and imitation learning [6]^[7]. Local inference runtimes such as Unity’s Inference Engine import trained networks and run them on the player’s CPU or GPU on all Unity platforms [7]^[8]. Conversational character SDKs package the whole talking-character stack: the Convai Unity SDK streams microphone audio for recognition, generates the reply and voice, and adds opt-in modules for actions, long-term memory, vision through a Unity camera and injection of runtime state [8]^[9].

Hardware vendors push the same stack on-device. NVIDIA’s ACE for Games offers small speech, language and vision models and a scheduler that runs inference alongside graphics [9]^[10]. Its January 2025 announcement describes PUBG Ally, a teammate built on the Mistral-Nemo-Minitron-8B-128k-instruct small language model that talks in game lingo, shares loot, drives vehicles and fights, and a local-inference AI teammate for NARAKA: BLADEPOINT [10]^[11].

The SDK route gives full control but ties a studio to a vendor’s roadmap. Inworld retired its hosted Character Studio, which held characters, scenes, knowledge and triggers, and asks developers to rebuild them with its Agent Runtime SDKs, which include relationship modelling [11]^[12]. Research prototypes show what the route buys: a Unity combat game in which a local Mistral 7B reads the live state every five seconds and assigns one of four tactical tags to a trained policy more than doubled the win rate against an adaptive opponent, from 11% to 24% [12]^[13]; panoramic renders segmented into a JSON scene description let NPC dialogue refer to nearby objects, and players preferred it to a context-free baseline [13]^[14]; and fine-tuned small models with swappable memory modules run NPC dialogue on consumer hardware [14]^[15].

2.3 Agents on an authoritative server #

In networked games the server owns the truth, and an agent can join as a client or run inside the server. This is where most research on “living” characters happens. Generative Agents populated a Sims-like town of 25 characters whose memory stream, reflections and plans produced emergent social behaviour [15]^[16]. Humanoid Agents added basic needs, emotion and an explicit closeness value for each relationship [16]^[17]. AI Town packages the idea as a deployable browser game in which AI characters “live, chat and socialize” [17]^[18]. On Minecraft servers, Voyager writes its own skill library as executable code [18]^[19], MINDcraft lets several model-driven characters collaborate and finds that performance drops by up to 15% when they must communicate detailed plans [19]^[20], and Project Sid’s PIANO architecture runs 10 to more than 1,000 agents in real time, which develop specialised roles, follow and change collective rules, and transmit culture [2]^[3].

CICERO reached human-level play in online Diplomacy by combining a dialogue model with strategic planning [20]^[21], and a 2025 state-machine prompting framework for StarCraft II won 60% of games against the hardest built-in AI where its baselines won none [21]^[22]. Orak exposes 12 commercial games to model agents through the Model Context Protocol [22]^[23]. In this class perception arrives as structured state and the server validates actions, so a slow or wrong model cannot corrupt the world.

2.4 Pixels and virtual input #

The most general surface treats the game as a black box. Cradle restricts its agent to screenshots in and keyboard and mouse out, and completes tasks in commercial games without built-in APIs [23]^[24]. SIMA 2, built on Gemini, acts in many 3D worlds, converses with the user and learns new skills from self-generated tasks [24]^[25]. Lumine processes raw pixels at 5 Hz, emits keyboard and mouse actions at 30 Hz, invokes reasoning only when necessary, and completes the five-hour Mondstadt storyline of Genshin Impact [25]^[26]. The cost of generality is time. VideoGameBench found inference latency to be a major limitation for frontier vision-language models in real time, and its “Lite” variant pauses the game while the model thinks; even then the best models completed 1.6% of the benchmark [26]^[27].

flowchart TD
    R[AI integration for a game character] --> S[Axis A: integration surface]
    R --> G[Axis B: augmentation type, level 0-3]
    S --> S1[Mod of a shipped game]
    S --> S2[Engine SDK or plugin]
    S --> S3[Authoritative server or protocol]
    S --> S4[Pixels and virtual input]
    G --> G1[Dialogue]
    G --> G2[Perception: vision and hearing]
    G --> G3[Memory]
    G --> G4[Time perception]
    G --> G5[Action and agency]
    G --> G6[Social state]

3. Method and Data #

The classification (E2). We coded 33 public systems on both axes: 5 mods, 14 engine integrations, 10 server or protocol agents and 4 pixel agents, 25 of them from 2025 or 2026. Each row cites the paper or documentation used. Each augmentation type is graded from 0 to 3; the full scale is in the repository README. For example, vision runs from structured state as text (1) to rendered images sent to a vision model on demand (2) to pixels as primary input (3); time perception runs from a paused, turn-based game (0) to event-triggered decisions (1), a periodic asynchronous loop (2) and continuous fixed-rate perception and action (3); social state is 1 when relationships live only in memory text and 2 when an explicit opinion variable is kept. The codes are our judgement from the cited sources, published as a CSV so others can dispute them.

The decision-latency measurement (E1). A village weaver receives a perception JSON (health, position, inventory, two nearby entities with distance, armament and current opinion, one heard utterance, two world events) and must reply with one JSON object: an action from eight verbs, a target, a spoken line, an integer opinion change from -10 to 10, and a short reason. Six utterances rotate across calls, two hostile (“Give me your bread or I will make you regret it.”), two friendly and two neutral. A second condition prepends a 120-entry episodic memory log of about 3,000 tokens. We called seven low-cost chat models through OpenRouter on 23 October 2026 with list prices from its public model catalogue [27]^[28]: Claude Haiku 5.5, GPT-6 Luna, Gemini 3.1 Flash-Lite, Qwen3.7 Flash, DeepSeek V4 Flash, Llama 3.1 8B and Ministral 8B. Each model received 12 calls per condition, streamed, with at most 200 output tokens and reasoning disabled. We recorded time to first token, total latency, billed cost, JSON validity as returned and after extracting the first object, and schema conformance. Ministral 8B returned HTTP 429 (upstream rate limit) on 43 of 48 attempts, including a sequential retry, and is excluded from latency statistics.

Conversions (our calculation). Ticks elapsed equal latency multiplied by the server tick rate. A village estimate assumes 20 NPCs that each decide every 5 s for one hour of play, which is 14,400 decisions. All code, data and outputs are in stabilarity/hub/research/living-npcs-a-classification-of-ai-integration-approaches-for-game-characters.

4. Results #

4.1 RQ1: Two axes, an uneven map #

Table 1 summarises the four surfaces; latency, cost and control follow from the surface before any model is chosen.

Surface	What the model sees	What it can change	Who owns time	Integration effort (our assessment)	Examples
Mod of a shipped game	Hooked state and events; screenshots in some mods	Commands the mod wires to existing game AI	The game; it never waits	Low to start, fragile across game updates	Mantella, CHIM
Engine SDK or plugin	Anything the developer exposes	Anything, through game code	The developer; can pause or budget	Medium; tied to SDK vendor	Convai, Inworld Runtime, ACE titles, ML-Agents
Authoritative server or protocol	Structured server state	Intents the server validates	The server tick	Medium; needs an agent API	Generative Agents, AI Town, Project Sid, CICERO
Pixels and virtual input	Rendered frames	Keys and mouse	Real time, unless paused	High; slow and costly per step	Cradle, SIMA 2, Lumine

Source: our classification of the systems cited in section 2; the effort column is a qualitative judgement.

Chart 4 shows where the 33 systems invest. Dialogue is the most crowded augmentation: 63.6% of systems reach level 2 or higher, and every system with voice in and out is a mod or an engine SDK. Memory follows at 57.6% and action at 54.5%. The surfaces specialise. All four pixel agents reach level 2 or higher on vision and action, and three of four on time perception, but none keeps social state and only one talks. Server agents are strongest on memory (80%) and executed action (70%) and weakest on perception: one of ten uses images and none processes audio. Mods are rich in dialogue and memory (80% each) but none runs a periodic or continuous decision loop; their characters think when spoken to.

The emptiest column is social state. Only 3 of 33 systems (9.1%) keep an explicit opinion or relationship variable: Inworld’s runtime, Humanoid Agents and a Stardew Valley mod that spreads gossip through a weighted relationship network (linked in our dataset). Only Humanoid Agents combines it with actions executed in the world. Almost everywhere else, opinion lives as text in memory that the model may or may not consult.

Time perception is the axis on which the surfaces differ most. A character perceives on one clock, decides on another and acts on a third, and the game decides what happens in between.

flowchart LR
    W[World state at tick t] --> P[Perception snapshot]
    P --> Q{Trigger: proximity, speech, timer}
    Q -->|no| W
    Q -->|yes| D[Model decision: about 1 s]
    D --> V[Server validates intent at tick t + k]
    V -->|still legal| A[Action applied]
    V -->|stale| F[Fallback behaviour]
    A --> W
    F --> W

On a server, an NPC decides on a trigger rather than every tick, and its intent is checked against the world as it is when the answer arrives, not as it was when the question was asked.

4.2 RQ2: A decision costs about a second and a fraction of a cent #

Table 2 lists the measured values for the perception-only condition, and Chart 1 adds the memory condition.

Model	TTFT p50 (s)	Total p50 (s)	Total p90 (s)	20 Hz ticks at p50	Strict JSON (%)	Cost per 1,000 decisions (USD)	Opinion change, hostile vs friendly
Llama 3.1 8B	0.581	0.904	2.466	18.1	91.7	0.0108	0.0 vs 0.0
DeepSeek V4 Flash	0.712	1.047	4.13	20.9	100	0.0426	-3.75 vs 0.0
Gemini 3.1 Flash-Lite	0.725	1.066	1.273	21.3	100	0.1845	-6.0 vs 0.5
Claude Haiku 5.5	0.705	1.16	1.25	23.2	100	0.0914	-3.75 vs 1.5
Qwen3.7 Flash	0.641	1.397	1.495	27.9	100	0.0206	-5.5 vs -2.75
GPT-6 Luna	0.894	1.535	2.288	30.7	91.7	0.0642	-0.75 vs 1.0

Source: our measurement via OpenRouter, 23 October 2026, 12 calls per model, perception-only prompt; prices as listed that day [27]^[28]. TTFT: time to first token.

Latency. The first token arrives after a median of 0.708 s across models and the complete decision after 1.113 s; per model, the median ranges from 0.904 s (Llama 3.1 8B) to 1.535 s (GPT-6 Luna). For a game, tails matter more: the 90th percentile ranges from 1.25 s to 4.13 s, and DeepSeek V4 Flash, routed across six different upstream providers, had both a fast median and the worst tail. Adding the memory log changed the median model’s latency by a factor of only 1.017, because these short replies are dominated by fixed overhead and output generation rather than by reading the prompt.

Game time. Chart 2 converts latency into ticks. At 20 Hz, a decision lands 18.1 to 30.7 ticks after its perception was taken at the median, and up to 82.6 ticks at the 90th percentile. At 60 Hz the median decision spans 54.2 to 92.1 frames. No tested model decides within a tick, so the model must run beside the game loop and the server must treat each reply as a proposal about a world that has since moved on. Lumine’s split between 5 Hz perception and 30 Hz control [25]^[26] and the pause in VideoGameBench Lite [26]^[27] are two answers to the same problem. The Real-Time Reasoning Gym frames it as a choice between reactive agents with bounded reasoning and planning agents with extended reasoning, and finds that combining both works best as time pressure rises [28]^[29]. Speculative execution can hide part of the wait: predicting likely next actions with a faster model reached up to 55% accuracy and cut latency by up to 20% in its authors’ environments [29]^[30].

Reliability. Of 144 successful calls, 4 were not valid JSON as returned: Llama 3.1 8B once wrapped the object in a code fence, and GPT-6 Luna three times appended stray text after a valid object. Extracting the first JSON object recovered all 144, after which one reply failed the schema with an opinion change of -19, outside the allowed range. Chart 3 plots strict parse rates against price. Whatever the model, the server must parse defensively and clamp every numeric field.

Cost. Per 1,000 perception-only decisions, billed cost ranged from USD 0.0108 (Llama 3.1 8B) to USD 0.1845 (Gemini 3.1 Flash-Lite); with memory, from USD 0.0482 to USD 1.0886. Memory multiplied the cost of the median model by 6.162, while DeepSeek V4 Flash rose only from USD 0.0426 to USD 0.0482 because its provider served 75.6% of the memory prompt tokens from cache. Prefix caching rewards prompts whose stable part comes first, which matters for NPCs whose memory changes slowly [30]^[31]. Our estimate for a village of 20 NPCs deciding every 5 s is USD 0.156 to 2.657 per hour of play without memory and USD 0.694 to 15.676 with it, depending on the model. Cost does not constrain a prototype; it constrains a persistent world with many characters and long memories.

Social signal. Five of six models lowered their opinion after hostile speech and raised it, or lowered it less, after friendly speech, with separations between 1.75 and 6.5 points in the perception-only condition. Llama 3.1 8B returned an opinion change of exactly 0 for every hostile and friendly utterance in both conditions. A game whose combat depends on an opinion threshold would never fight with that model, so the opinion update must be tested per model and partly computed by game rules.

4.3 RQ3: The testbed for this series #

A mod is the fastest way to a talking character but cannot host a decision loop, and its actions are borrowed from someone else’s AI. A pixel agent is general but spends most of its budget on seeing. An authoritative server is the only surface where perception is structured and cheap, actions are validated, time is owned by a tick the designers control, and humans and agents share one world on equal terms. It is also where the emptiest cell, explicit social state combined with executed action, is easiest to fill.

The series will therefore build a server-authoritative 2D browser MMO set in a feudal village enclosed by a wooden palisade. Each NPC is controlled by a language model through the interface measured in E1: the server sends a perception snapshot, the agent returns an intent, and the server validates and applies it. Players and NPCs use the same avatars and the same chat, so neither can tell which is which; a 2025 study in which GPT-4.5, prompted with a humanlike persona, was judged to be the human 73% of the time in five-minute conversations suggests that the disguise can hold in text [31]^[32]. Proximity opens a conversation, every exchange updates an opinion score, an NPC whose opinion of someone falls below a threshold attacks, and every character’s goal is to stay alive. A first version, Oakhollow, is already open to play at hub.stabilarity.com/living-village^[33]; it is the testbed for article 2, and no results from it are reported here.

flowchart TD
    C[Characters meet] --> X{Within chat radius?}
    X -->|no| M[Move, work, idle]
    X -->|yes| T[Chat exchange]
    T --> O[Opinion update, clamped by server rules]
    O --> Y{Opinion below threshold?}
    Y -->|no| M
    Y -->|yes| K[Attack intent]
    K --> S[Server resolves combat]
    S --> L{Still alive?}
    L -->|yes| M
    L -->|no| E[Respawn as a stranger]

The design follows from the measurements. Decisions are triggered by proximity and speech, not by every tick. The opinion score is a server variable that the model proposes changes to but cannot set, which guards against both the out-of-range reply and the silent zero observed here. Combat is resolved by game rules, so latency changes who acts first but never what is legal.

5. Discussion #

Implications for practitioners. The simplest way to connect a model and patch a specific game depends on what the character must do. To add conversation to a single-player game you already own, a mod framework or a conversational SDK is the shortest path, and conversation hides latency. To make characters act on their own in a shared world, put the model behind an authoritative server, give it structured perception and a small verb set, and let the server own time and rules. Reserve pixel agents for games without any interface, and budget for their cost in time.

flowchart TD
    Q1{Do you control the game source?} -->|no| Q2{Does the game have a mod or script API?}
    Q2 -->|yes| A1[Mod: dialogue, memory, menu actions]
    Q2 -->|no| A2[Pixels and virtual input]
    Q1 -->|yes| Q3{Shared world with many characters?}
    Q3 -->|no| A3[Engine SDK, local or cloud model]
    Q3 -->|yes| A4[Authoritative server with agent API]

Players are not uniformly delighted. A randomised study with 130 participants found that model-driven NPCs raised cognitive load compared with scripted ones and did not significantly improve the overall gaming experience; perceived autonomy rose while usability and trust fell [32]^[34]. Free-form dialogue also opens an attack surface: one 2025 study examined whether adversarial prompt injection can make NPCs reveal hidden backstory secrets [33]^[35]. Rule-heavy interactions need scaffolding. A trading prompt that decomposes negotiation into six states reached more than 97% state compliance and 99.7% calculation precision [34]^[36], which supports keeping prices, inventories and combat in game code and letting the model choose among legal moves.

Limitations and threats to validity. The classification codes are our judgement from documentation, which describes intended features rather than measured behaviour, and the sample over-represents engine prototypes from papers and under-represents commercial titles that publish little. With four pixel agents and five mods, the shares for those rows rest on few systems. The latency measurement used 12 calls per cell from one client in Europe on one day, through one routing service whose provider choice varied, so tails are noisy and absolute values depend on region, load and provider. We tested only small models with reasoning disabled; larger models or reasoning would raise latency and, likely, quality. The memory log was synthetic and the opinion test used six fixed utterances, so the social-signal result shows that models differ, not how well any of them models opinion.

What would change the conclusion. On-device models that decide within a frame would weaken the case for an asynchronous loop, and an engine SDK with a game-validated relationship model would fill the social-state gap.

6. Conclusion #

RQ1 Finding: AI integration for game characters can be described by four integration surfaces and seven augmentation types, and the 33 coded systems cluster unevenly on them. Measured by the share of systems at level 2 or higher = 63.6% for dialogue versus 9.1% for explicit social state, with 0% of mods running a periodic decision loop and 100% of pixel agents reaching level 2 on vision and action. This matters for the series because it locates the open cell, social state combined with executed action, that the testbed is designed to fill.

RQ2 Finding: One structured NPC decision from a low-cost model takes about a second and costs a fraction of a cent, which places the model beside the game loop, never inside it. Measured by the median model’s total latency = 1.113 s (p50 range 0.904 to 1.535 s, p90 up to 4.13 s), 18.1 to 30.7 ticks of a 20 Hz server at the median, 4 of 144 replies needing defensive parsing, and USD 0.0108 to 0.1845 per 1,000 decisions, rising by a median factor of 6.162 with a memory log. This matters for the series because it fixes the testbed’s time model: event-triggered decisions validated by the server, with memory cost controlled by caching.

RQ3 Finding: A server-authoritative 2D browser world with structured perception, a small verb set and a server-owned opinion score is the simplest credible testbed for characters that live, talk and fight. Measured by the opinion separation between hostile and friendly speech = 1.75 to 6.5 points for five models and exactly 0 for one, which shows that the opinion rule must be enforced and tested by the game rather than assumed from the model. This matters for the series because article 2 will build that village and measure whether its humans and model-driven characters can tell each other apart.