Agentic XR development

§ 1Why this guide

The agentic XR conversation has moved fast in 2026. Meta’s DevRel ships full VR/MR games end-to-end with Claude Code + Unity AI Assistant. WebXR developers wire IWSDK’s ECS into emulator MCPs and step through frames from the chat window. None of this is a single tool decision — it is a small set of patterns that show up in every successful workflow, regardless of engine.

This guide extracts those patterns. It is not a comparison between Unity and the web; it is a catalog of techniques you can mix and match. Every pattern is shown twice: once as it appears in a Unity-side demo (Dilmer Valecillos’ GDC 2026 basketball build) and once as it appears in the author’s own IWSDK projects (Virtual Makerspace, Circuit Playground). The point is the underlying move, not the brand of the tool.

§ 2How to read it

Three reading paths depending on where you are starting from:

Each pattern card shows the rule, why it matters, and how it shows up in two stacks (Unity-side and IWSDK-side) — plus a “when not to apply” line so you don’t over-engineer.

§ 3Quick start — your first agentic XR session

If you have already cloned an IWSDK project (e.g. the official scaffold or Virtual Makerspace) and have Claude Code installed, the following 20-minute walkthrough exercises five of the patterns at once. Skip ahead to the catalog if you prefer to read first.

1. Open the project in Claude Code. Read CLAUDE.md if present — it encodes pitfalls and conventions for the project (pattern 01).
2. Run the tool check (pattern 02) as your literal first message:

   What custom tools do you have available?
   Group by MCP server. If any are missing,
   stop and tell me which.

3. Confirm the dev server. If the agent reports the IWER MCP is connected, the session is already running — do not start another. Otherwise npm run dev in a separate terminal and re-check.
4. Smoke-test one MCP call (pattern 02 deepening): ask the agent to take a screenshot, then describe what it sees:

   Call browser_screenshot. In one paragraph,
   describe what you see in the image and
   which entity is in the foreground.

   Anything wrong here (missing tool, blank image, wrong scene) blocks every other pattern — fix it before continuing.
5. Attach a plan file. Pick a small task (“add a new colour to the LED palette”) and write a 5-line plan in docs/plans/led-palette.md with goal, components touched, telemetry events emitted. In chat: “Read docs/plans/led-palette.md and implement it; ask before editing the telemetry schema.”
6. Verify with a screenshot loop (pattern 05). After the agent reports done, ask for a fresh browser_screenshot and a one-sentence visual diff vs. the previous one.
7. Stop and human-check. Read the diff yourself. If the change touched a system you weren’t expecting (e.g. scene.add snuck in despite CLAUDE.md) revert and re-prompt with the rule made explicit.

§ 4Pattern catalog

Eleven patterns organized roughly from the start of a session to the end, with the meta-pattern (agent reviewing agent) closing the catalog. None is mandatory; each is independently useful. The first three are the highest-leverage if you are picking only a few; pattern 11 unlocks the next scale of work.

At a glance — pattern × stack matrix

Where each pattern lands on the stacks covered in this guide. ✓✓ = native tooling exists, drop-in. ✓ = works with manual setup. ○ = build it yourself. Use this to decide which patterns are quick wins on your stack and which require investment.

Reading the table: patterns 01, 02, 04, 08, 11 are universal — portable across any stack with no setup cost. Pattern 03 (domain MCP) and pattern 07 (ECS-aware debugging) are where IWSDK’s open-source MCP toolchain has the biggest head start in 2026-05. Three.js bare and Babylon.js builders pay a one-time setup cost on those two patterns but everything else transfers cleanly.

Can you implement a basketball
hoop using the details from
this file?

What custom tools do you have
available?

mcp__iwsdk-dev-mcp__
xr_get_session_status

Pattern 03

Domain knowledge in an MCP

Don’t spend context tokens making the LLM recall what your platform’s docs already know — front it with an MCP that searches authoritative sources.

The model can guess how Quest passthrough works, but it will be wrong half the time and slow always. An MCP that semantically searches your platform docs gives the agent a primary source instead of a hallucinated one — better answers, faster session, fewer confidently-wrong rabbit holes. The same logic applies to your own domain knowledge (curriculum, experiment design, code conventions): expose it through an MCP rather than hoping context survives compaction.

Unity stack HZDB

Meta Horizon Developer Hub MCP exposes the entire Meta Quest documentation set plus connected-device inspection. Sample queries:

• “How do I implement passthrough in Unity?”
• “What is the Quest 3 battery level?”
• “Search Meta’s 3D model library for ‘office chair’”

IWSDK stack IWSDK-RAG

Semantic code search across @iwsdk/core, elics, and super-three. Returns actual source — not summaries — so the answer is verifiable.

mcp__iwsdk-rag-local__
search_code: "how to subscribe
to query qualify events"

For your own project: compile your codebook / R-script library / pedagogy charter into a small RAG MCP and expose it the same way.

When not to apply: exploratory sessions where you want the model’s best guess to challenge your assumptions; brainstorming.

Building your own domain MCP

Most readers want to expose their own domain knowledge — a curriculum, a codebook, a pedagogy charter, a research-protocol manual. The cheapest path:

1. Start with a directory of markdown. Even a flat docs/ folder is enough to RAG.
2. Use a generic local MCP server — e.g. an fs-mcp that exposes search across a path, or one of the open-source RAG MCP templates listed at the Model Context Protocol directory. Configure it to index your folder.
3. Wire it into Claude Code via ~/.claude/mcp_servers.json; the agent will list it under its tools after restart.
4. Smoke-test with the inventory check (pattern 02): the new tool should appear with a meaningful name.
5. Quality-control the retrieval. Ask deliberately ambiguous questions; if the model invents a section that isn’t in your docs, the indexer is too lossy — raise chunk size, lower top-k, or add structural metadata.

For research projects: a small RAG over your docs/pedagogy-charter.md + experimental-protocol PDFs gives the agent reviewer-anchored constraints without re-pasting them every session. The author plans a follow-up guide on this; until then, the IWSDK-RAG implementation in node_modules/@felixtz/iwsdk-rag-mcp is a working public example to read.

Pattern 05

Screenshot validation loop

After every visual change, take a screenshot and feed it back to the agent. Trust the eye, not the diff.

Code that compiles and runs can still produce wrong output: a misplaced anchor, a transparent material, a Z-fight. The fix is to give the agent vision in the loop. Unity AI Assistant captures the editor view automatically after each action; on the web you call browser_screenshot through the IWER MCP. Either way, the agent sees what you see and can correct without you typing the description.

Unity stack

Unity AI Assistant captures the scene view automatically after each action and feeds the image back. No extra prompt needed.

IWSDK stack

Manual but cheap:

mcp__iwsdk-dev-mcp__
browser_screenshot

Combine with scene_get_hierarchy for ground-truth entity positions when the screenshot is ambiguous.

When not to apply: pure backend / data-pipeline work where there is no visual; performance tuning where you should be reading the profiler instead.

vision-model limits

LLM vision is reliable for which entity is where and does the layout look broken. It is unreliable for pixel-precise alignment, very small text (HUD timer at 8 px), fast motion (frame between gestures), and colour accuracy in low light. For these, pair the screenshot with a structured query — scene_get_object_transform, ecs_query_entity — and treat the image as a sanity check, not as the source of truth.

Pattern 06

Force prefab / entity spawns

Tell the agent explicitly to spawn from project assets, not to construct primitives at runtime. You will want to tweak it later.

Agents default to building from primitives because primitives need no asset dependency — but the resulting object is invisible to the inspector and impossible to edit by hand. Forcing prefab / entity-spawn semantics keeps every agent-generated artifact tweakable through the normal authoring path.

Unity stack

Default Dilmer build had ball launcher creating a primitive at runtime. Author corrected: “use the project prefab.” Result: ball mesh now editable in inspector, physics tunable without re-prompting.

IWSDK stack

Equivalent rule: prefer entity spawners with declared components, not scene.add(mesh). The author’s spawn-components.ts does the right thing — every part starts as a properly-componented entity, queryable by every system.

world.createTransformEntity(
  mesh, parentEntity);
// not:
// scene.add(mesh);

When not to apply: truly throwaway primitives for visual debugging (a red box marking a target).

Input modality note — controllers, hands, voice, gaze

The patterns in this guide assume controllers because that is where the agentic tooling matured first. Each alternative input modality changes one or more pattern semantics; design for it explicitly rather than letting the agent default.

Modality	What changes	Where it bites
Controllers (default)	Trigger / squeeze / thumbstick discrete events; DistanceGrabbable with ray; deterministic input.	Reliable. Use as the baseline experimental condition unless your study question is specifically about embodiment.
Hand tracking	No buttons — grab semantics shift to proximity + pinch confidence. OneHandGrabbable on IWSDK and Unity hand subsystems use closeness + finger-pose heuristics. Telemetry events fire on pinch-detect, not button-down, with non-zero false-positive rates.	Snap reliability drops; near-miss events become noisier. If your study reports manipulation precision, control for hand-tracking confidence in the analysis. The IWER MCP supports xr_set_input_mode="hand" for emulator testing — use it before assuming hand parity with controllers.
Voice	Continuous input layer: STT → intent → agent response → spatial-audio TTS. Adds an LLM in the participant loop, not just the dev loop.	Easily breaks Productive Failure designs. A voice agent that volunteers help during struggle collapses PF into guided practice. Pre-register the intervention policy (silence vs prompt) before any data collection. Voice introduces a second IRB consideration: audio capture and storage. See § 10 researcher concerns.
Gaze	Quest Pro / Vision Pro / Pico 4 Enterprise expose eye-tracking. Gaze gives attention allocation as a continuous signal — an order of magnitude richer than controller telemetry.	Gaze is a strong debugging signal: where did the participant look during failure? It is also strong study evidence for theories that hinge on attention (CAMIL, joint-attention CSCL). Storage policy: gaze data may identify individuals; treat it like biometric PII per § 10. Hardware availability is uneven — not on Quest 2 / 3 base.

Practical rule: when an agent suggests “add hand tracking” or “add voice” mid-build, treat it as a condition change, not a feature. Both modalities shift what the same participant’s telemetry means; introducing them after Phase 1 data collection breaks comparability with prior runs.

ecs_pause
ecs_snapshot label="before"
ecs_step count=1
ecs_snapshot label="after"
ecs_diff from="before" to="after"

Pattern 08

Telemetry-first design

Decide what events are emitted before you decide what scenes look like. Agentic builds drift; the telemetry contract holds them in place.

If you don’t commit to a telemetry schema early, the agent will happily add features that no measurement instrument captures. With a schema in docs/ the agent treats it as an interface contract — every new interaction emits a known event, and quantitative analysis stays possible. This applies to research projects (the obvious case) and consumer products alike (analytics, A/B tests, telemetry-driven QA).

Unity stack

Equivalent: define a Unity Analytics event taxonomy or a custom event bus, document it in markdown, pass it to the agent. New gameplay features wire into the bus, not new bespoke logging.

IWSDK stack VM example

Schema lives in docs/mvp-scope.md. Every event shares one envelope (event_id, session_id, frame_time_ms, optional poses) — the schema specifies UUID v7 for time ordering, the implementation uses crypto.randomUUID() (v4) until a v7 library is wired in. Agent-added systems emit events from the existing catalog; error_recovery on undo, socket_connect on snap, circuit_state_change on topology delta. NDJSON export at session_end.

When not to apply: throwaway prototypes you will not ship and not study.

Telemetry → analysis bridge

A schema is only useful if the analysis pipeline can read it. NDJSON exports drop straight into:

• Sequence mining — treat event_type as the alphabet, session_id as the unit. See the author’s Sequence Mining Blueprint (TNA, LSA, SPM, HMM) for end-to-end R templates.
• Ordered Network Analysis — group events by participant and condition, encode co-occurrence within a sliding window. The ONA in Practice guide covers the agentic-figure workflow.
• Knowledge tracing — if the task has scoreable steps, step_advance + correctness become the input; Knowledge Tracing Blueprint covers BKT / DKT / R / Python.
• Stealth assessment — backward-design from outcome to ECD evidence; see the Stealth Assessment draft for the full pipeline.

The point: design the telemetry schema once, then pick the analysis lens later. The four guides above all consume the same NDJSON shape.

npx -y @meta-quest/hzdb mcp install claude-code

---
description: One-line trigger for when this skill applies.
  Used by Claude Code's matcher.
argument-hint: "[optional positional arg]"
---

You are running the **/<skill-name>** workflow. The user typed
`/<skill-name> $ARGUMENTS` because they want X.

## Pipeline

1. Confirm prerequisite: e.g. dev server running, MCP healthy.
2. Read the relevant project files into context (list 2-3).
3. Apply the verb: do exactly the recurring task this skill exists for.
4. Verify with the standard check (screenshot, type-check, test).

## House rules

- Project conventions to enforce (e.g. "import Three from @iwsdk/core,
  never from 'three'").
- Anti-patterns to refuse.

## Output contract

What artifacts this skill produces, where they go, and how the user
verifies success.

device_battery
device_info
get_device_logcat
  --tag=Unity --level=W

Use the iwsdk-project-code-reviewer agent
to review the last commit against
docs/pedagogy-charter.md.

§ 5Tooling spectrum

Five stacks with different deployment, license, and openness trade-offs. Pick by where the build needs to run and what you want to control directly.

Other stacks the author has not deployed personally — included for orientation, not endorsement: Three.js bare (vanilla three + WebXRManager; trade IWSDK’s ECS for full control); Babylon.js (built-in WebXR helpers, node-material editor; mature engine, but the IWSDK-equivalent ECS-aware MCP toolchain is not something the author has tested); Unreal Engine (high-fidelity VR; agentic tooling is less mature than Unity AI in 2026-05 and the author has no first-hand workflow data); Unity authoring → web runtime (Unity + Meta Spatial Editor for content, then glTF/GLXF export to IWSDK or Three.js for browser deploy — valid in principle, two stacks to maintain). For each of these, patterns 01–04 transfer; patterns 05, 07, 10 depend on stack-specific MCPs you would need to source or build yourself.

§ 6Asset pipeline — Blender MCP

Code is half the agentic XR build; the other half is 3D assets. Once the agent can generate meshes, search asset libraries, and run Blender scripts on your behalf, the bottleneck shifts decisively from authoring to direction. Blender’s MCP server is the cleanest way to give an agent that capability today — it lets Claude Code call execute_blender_code, search Polyhaven and Sketchfab, generate via Meshy or Hunyuan3D, and import the result straight into the scene without leaving the chat.

What the Blender MCP exposes

A representative tool surface (this section’s code blocks use the same prefix the author’s MCP exposes — yours may differ slightly):

Install workflow (10 minutes)

1. Blender 4.x — download from blender.org. The MCP server runs as a Blender add-on, so you need a working install before anything else.
2. Get the Blender MCP add-on. The most active community implementation as of 2026-05 is ahujasid/blender-mcp. Clone it and follow its README — typically: copy the add-on folder into Blender’s scripts/addons/ directory, enable in Edit → Preferences → Add-ons, then start the MCP server from the add-on’s panel.

   # macOS / Linux example — adapt for your OS
   git clone https://github.com/ahujasid/blender-mcp ~/blender-mcp
   ln -s ~/blender-mcp/addon ~/Library/Application\ Support/Blender/4.2/scripts/addons/blender_mcp
   # then in Blender: Preferences → Add-ons → search "MCP" → enable → "Start MCP Server"

3. Wire the server into Claude Code. Add an entry to your Claude Code MCP config (typically ~/.claude/mcp_servers.json or via claude mcp add). The add-on README has the exact command and stdio/SSE transport choice; follow it verbatim.
4. Restart Claude Code. The Blender tools should appear in the inventory.
5. API keys for Meshy / Hunyuan3D / Sketchfab go in the add-on’s preferences panel inside Blender, not in Claude Code config — the agent never sees them.
6. Smoke test (pattern 02) the new server right away:

   List the Blender MCP tools you can call.
   Then call get_scene_info.
   Then call get_polyhaven_status.

   Three results, each non-empty → you’re live.

Standard agentic asset workflow

The recipe below is the one that works most reliably across XR projects. Plug in your stack’s import path at the end:

1. Search first — search_polyhaven_assets("breadboard") or search_sketchfab_models("low-poly basketball"). Free CC0 from Polyhaven beats anything you generate; only fall through to text-to-3D if nothing matches.
2. Generate fallback — if no good search result, generate_hyper3d_model_via_text with a constrained prompt (“Single low-poly object on neutral background, real-world scale, single material slot”). Poll the job status; import on success.
3. Inspect in Blender — get_viewport_screenshot. Eyeball it. Pattern 05 applies inside Blender too.
4. Cleanup pass — execute_blender_code with a small Python snippet to: decimate to a target poly count, rotate to your axis convention (Y-up vs Z-up), centre the origin, apply scale. The agent can write this snippet from your project conventions.
5. Texture — set_texture with a Polyhaven HDRI for environment, or attach project-specific PBR maps.
6. Export — execute_blender_code running bpy.ops.export_scene.gltf(filepath="public/gltf/asset.glb"). Glb because IWSDK and Three.js consume it natively.
7. Wire into the scene — declare in your AssetManifest (IWSDK) or GLTFLoader (vanilla Three.js); spawn entities as the project conventions dictate (pattern 06).

Where Blender MCP fits in the pattern catalog

• It is a domain MCP (pattern 03) — the “domain” here is 3D content creation rather than platform docs.
• It pairs with the screenshot loop (pattern 05) via get_viewport_screenshot, so the agent sees the result of every modelling step.
• It survives the prefab rule (pattern 06) only if you set up a project-specific export script that targets the runtime asset folder; otherwise the agent will save to /tmp.
• It is not a substitute for a 3D artist on production work. For research-grade or commercial deliverables, treat agent-generated meshes as fast scaffolding that an artist will replace before launch.

Common failures (specific to asset pipelines)

§ 7Case studies

Five end-to-end views on the patterns combined — four from the author’s own work, one from the broader community wave around GPT-5.5. Pick the one closest to your project shape.

Unity · Basketball MR — Dilmer Valecillos demo (GDC 2026)

17 min · public

Meta DevRel Dilmer Valecillos shipped a complete MR basketball game from scratch in roughly 17 video minutes using Unity AI Assistant + Claude Code (in-editor) + Meta MCP Extensions + HZDB. The whole flow is patterns 01, 02, 03, 05, 06 stacked.

How to Build Full VR/MR Games With Unity AI + Meta Agentic Tools Dilmer Valecillos · 17:01

Build sequence

1. Setup: Asset Store fetches Meta SDK, OpenXR Meta XR feature group, Oculus Touch profile, Android manifest.
2. Tool check (pattern 02): “What custom tools are available?” — confirms create_camera_rig et al are visible.
3. Plan injection (pattern 01): pre-iterated basketball-hoop.md passed as reference; backboard / bracket / rim / net produced consistently in one pass.
4. Build loop: ball launcher prefab (pattern 06 correction), trail renderer for fire effect, 30s game loop, scoreboard UI, audio manager, green grid line at score.
5. Domain doc fallback (pattern 03): for passthrough implementation, HZDB queries Meta’s docs directly instead of letting the model guess.
6. Visual loop (pattern 05): Unity AI screenshots the editor view after each action; corrections happen inside the same chat thread.

Takeaway: the agent isn’t a one-shot generator. The author iterated on every artifact (UI rebuilt twice, audio policy negotiated). The leverage is that each iteration takes minutes not hours — the same patterns that make a single agent fast also make iteration fast.

IWSDK · Virtual Makerspace — research-grade WebXR study apparatus

Phase 1 implemented · 2026-05

Author’s own project (github.com/Educatian/virtual-makerspace) — a research-grade WebXR prototype investigating embodied Productive Failure across four maker modules. The agentic workflow keeps the implementation aligned with a non-negotiable pedagogy charter and a precise telemetry schema.

Your First Steps with Immersive Web SDK Meta Developers

Anchoring artifacts (pass them as context every session)

• docs/pedagogy-charter.md — Productive Failure / embodied cognition / GBL-not-gamification commitments. Prevents agent feature creep.
• docs/mvp-scope.md — telemetry schema (event types + payloads) and Phase 1/2/3 boundaries.
• CLAUDE.md — IWSDK pitfalls (LocomotionEnvironment, 11–14 ms budget, Three.js import rule).

Tooling chain

• IWSDK-RAG MCP (pattern 03) — semantic search across @iwsdk/core + elics + super-three returns actual source code for API questions.
• IWER MCP (pattern 07) — controls a WebXR session from the chat window: xr_set_transform, xr_select, ecs_pause, ecs_step, ecs_snapshot, ecs_diff, browser_screenshot.
• HZDB MCP (pattern 10) — checks the connected Quest’s battery / Android version / browser logcat.
• Six sub-skills (pattern 09) — /iwsdk-planner · /iwsdk-grab · /iwsdk-ray · /iwsdk-ui · /iwsdk-debug · /iwsdk-physics.
• Reviewer agent (pattern 11) — iwsdk-project-code-reviewer declared in CLAUDE.md; reviews diffs against the pedagogy charter and IWSDK pitfall list.

Module 1 — Electronics Studio (representative slice)

• Procedural breadboard: 16 cols × 12 rows = 192 sockets as a single InstancedMesh.
• Eight grabbable parts (LEDs, resistors, wires, 9V battery) with magnetic snap-on-release within 27 mm.
• Live circuit evaluation: union-find on wires + resistors + LED leads vs battery terminals; LED emissiveIntensity jumps to 1.8 the moment a closed loop forms.
• Telemetry: every grasp / release / snap / state change is a typed event in IndexedDB; gzipped NDJSON export on session_end.

Placement guides — toggleable with G

CircuitEvalSystem closes the loop, LED glows

Spectator-style 3rd-person view via IWER

M3 — embodied data labeling prototype

Walkthrough — one feature, end to end (HoverPreview, ~30 min)

To make the patterns concrete, here is the compressed actual session that produced the green-socket hover-preview shown above. Five exchanges total; each maps to a pattern.

1. Tool check & context attach (02, 01):

   What custom tools do you have available?
   Then read these files into context:
   - docs/pedagogy-charter.md  (relevant section: §6 PF policy)
   - src/systems/snap-system.ts
   - src/components/snap.ts
   - CLAUDE.md  (relevant section: 11-14 ms budget, no-allocation rule)

   Agent reports IWER + IWSDK-RAG + HZDB live, summarises each file in one sentence to confirm load. Cost so far: ~3 min.
2. Plan file (01):

   Draft docs/plans/hover-preview.md.
   Spec: while a part is held within 27 mm of two valid sockets,
   render two small green markers at the snap target so the
   participant can see the snap before committing.
   Constraints from the charter: must be GBL-compatible
   (world-as-feedback, not score-like). 200 ms duration max.
   Telemetry: emit hover_target { entity_id, target_sockets[2] }
   on enter; hover_target_clear on leave.
   Performance: zero allocation in update(); reuse Vector3.

   Agent writes the plan in 12 lines. Author edits 2 lines (clarifies that markers are non-blocking visuals, not interactable entities). Plan committed to git before any code change. Cost so far: ~10 min.
3. Plan-mode build (04): the change touches a system (shared) and emits a new event type (schema), so plan mode wins both checklist questions.

   Implement docs/plans/hover-preview.md in plan mode.
   Use findBestSnap() from snap-helpers.ts so the markers
   share one source-of-truth with SnapSystem.

   Agent proposes: new HoverPreviewSystem, two new components (HoverPreviewMarker, HoverPreviewState), extends the telemetry event catalog. Author approves; agent applies. Cost so far: ~17 min.
4. Screenshot loop (05):

   browser_screenshot. In one paragraph, describe the
   markers. Then ecs_query_entity for the held part and
   verify HoverPreviewState.targets is the same pair as
   the visual markers.

   Agent reports markers visible at the predicted sockets, ECS state matches. Cost so far: ~22 min.
5. Reviewer agent (11):

   Use the iwsdk-project-code-reviewer agent. Review the
   hover-preview commit against docs/pedagogy-charter.md
   and CLAUDE.md. Flag charter violations and any update()
   allocations.

   Reviewer flags one issue: the markers fade at 200 ms but charter §6 specifies the neutral spatial-locator allowance is for fail markers, not preview markers — preview markers are continuously visible during hold (no time limit needed). Implementer agent removes the fade timer. Cost so far: ~28 min. Done.

The session shipped a feature in a third of an hour, but the substantive work happened in steps 2 and 5: the plan file forced the constraint set up front, and the reviewer caught a charter mis-read that would otherwise have shipped silently. Most of the leverage of agentic XR development is in the bookends, not in the implementation middle.

Takeaway: the patterns that make agents productive (plan files, custom-tool checks, ECS-aware debugging) double as the controls that keep a research-grade build defensible. The pedagogy charter is both an LLM context document and a reviewer-facing artifact.

Community wave · GPT-5.5 / vibe coding examples from X

external case · 2026-04 → 05

The author’s portfolio is one data point. The broader community wave around GPT-5.5 — OpenAI’s “new class of intelligence for agentic coding” rolling out across ChatGPT and Codex from late April 2026 — has produced a flood of single-prompt browser games and 3D demos in the weeks since. The headline finding for XR builders: game-feel is no longer the bottleneck for one-shot generation. Whether the rest of the loop (research-grade telemetry, sustained scope, multiplayer) makes sense to vibe-code is a separate question.

Headline example — Pietro Schirano (@skirano)

Featured on the GPT-5.5 release page. Schirano posted on X: “This is the game GPT-5.5 built for me in one shot, featured on the release page. What makes it different from previous AI-generated games is how fun it actually feels to play. The controls are smooth, and every shape is generated from scratch in Three.js.” Link: x.com/skirano/status/2047403025094905964. The prompt itself was not disclosed in the post.

Earlier in the GPT-5.2 cycle, Schirano shipped a single-file 3D graphics engine with interactive camera controls and 4K export — one of the wildest demos catalogued by The Neuron Daily’s round-up of the GPT-5.2 wave (theneurondaily.com/p/the-top-10-wildest-gpt-5-2-demos).

Other notable community examples

• Flavio Adamo (@flavioAd) — Hex Bounce — glowing balls ricocheting inside a rotating hexagon, generated by GPT-5.2 in three.js. Hosted as a public physics-stress playground at inferent.io/playground.
• Nicolas Zullo (@NicolasZu) — multiplayer 3D dogfighting game — vibe-coded entirely with Claude in Cursor, 0 hand-written code. 20 hours, 500 prompts, €20 in API spend; 1.5 M views on X, 45,000 testers. Link: x.com/NicolasZu/status/1899931187398979890.
• Clad3815 — Pokémon Crystal Hard-Mode agent — GPT-5.2 streaming live on Twitch, making real-time decisions to play through the game. Long-context + tool-use stress test.
• Bartosz Naskręcki — algebraic-geometry app — visualised the intersection of two quadratic surfaces and converted the resulting curve into a Weierstrass model, single prompt, 11 minutes. Featured on OpenAI’s GPT-5.5 release page.
• Ethan Mollick (Wharton) — procedural harbor-town evolution — “build me a procedurally generated 3D simulation showing the evolution of a harbor town from 3000 BCE to 3000 AD.” GPT-5.5 Pro completed in 20 minutes (down from 33 in the previous version) and was the only model that actually showed evolution rather than just replacing buildings. Write-up: oneusefulthing.org/p/sign-of-the-future-gpt-55.

OpenAI’s official prompt repository

openai/gpt-5-coding-examples (1,881 stars, MIT-leaning showcase) curates 60+ apps generated entirely from a single GPT-5 or GPT-5.2 prompt. Each example pairs a .yaml file (the exact prompt) with a built app and a screenshot on OpenAI’s CDN. Hosted demo: gpt5-coding-examples.vercel.app. Game-relevant entries include Asteroid Game, Escape the Maze, Falling Fruit Catcher, Fun Game, Tic-Tac-Toe, Solar System Explorer, Ocean Wave Simulation, Festival Lights Show.

asteroid-game (GPT-5.2)

ocean-wave-simulation (GPT-5.2)

solar-system-explorer (GPT-5.2)

festival-lights-show

escape-the-maze

falling-object-catcher

fun-game

tic-tac-toe-game

Three real prompts (verbatim from OpenAI’s repo)

Useful as starting points — copy and adapt rather than re-deriving from scratch. These are exact zero-shot prompts that produced the shipped demos above:

Asteroid Gameprompt

Make a 2d space game, in which I can fly a ship,
avoid and blow up asteroids, and dogfight with other
computer-controlled AI. Be creative with the design
of the ships. Ensure the gameplay works and is fun.
Output code in a single next.js page.tsx file, which
can be pasted directly into a next.js app created by
create-next-app, alongside any context or instructions
needed to run it.

Escape the Mazeprompt

Create a single-page app in a single HTML file with
following requirements:
- Name: Escape the Maze
- Goal: Navigate from start to finish in a randomly
  generated maze.
- Features: Arrow key controls, timer, shortest path
  bonus, replay button.
- The UI should be clear with visible maze walls and
  a movable avatar.

Fun Gameprompt

Create a single-page app in a single HTML file with
the following requirements:
- Name: Fun Game
- Goal: Jump over obstacles to survive as long as
  possible.
- Features: Increasing speed, high score tracking,
  retry button, and funny sounds for actions and
  events.
- The UI should be colorful, with parallax scrolling
  backgrounds.
- The characters should look cartoonish and be fun to
  watch.
- The game should be enjoyable for everyone.

The pattern across these prompts is consistent: a one-line goal, a bullet list of must-have features, a UI description, and a target output format (single HTML, single next.js page, single file). Notice the absence of pedagogical scaffolding, telemetry, or research framing — these are entertainment-first prompts. For research-grade XR work, the same prompt skeleton needs the constraints from the rest of this guide layered on top.

Where it stops working — honest framing

• One-shot is one slide. The viral examples are first-prompt outputs; the second iteration usually needs the patterns in this guide (plan files, screenshot loop, reviewer agent) for anything past 200 lines of code.
• Three.js is over-represented in the wave because it’s the path of least resistance for browser-deploy. If your project needs hand tracking, room-scale tracking, or passthrough, a one-shot demo is decoration; the IWSDK / Unity workflows in § 7b and § 7a are the durable paths.
• Multiplayer is still the ceiling. Zullo’s multiplayer dogfighter took 500 prompts and 20 hours, not one prompt — 10× the engineering of the single-shot demos. Read that as the realistic budget for any networked game built this way.
• Vibe coding is not vibe-research. A 500-prompt session leaves no audit trail by default. For a study apparatus, the researcher-specific concerns (reproducibility, pre-registration, participant-data privacy) need to be enforced from the first prompt; retrofitting them costs more than embedding them up front.

Takeaway: the GPT-5.5 wave is real and useful as a prototyping accelerator. For XR builders the value is mostly upstream — one-shot Three.js demos collapse the “is this concept worth building” question into a 10-minute experiment. Beyond that point, the patterns earlier in this guide do the heavy lifting; vibe-coding alone won’t hold up a research-grade study or a commercial release.

§ 8Common pitfalls

Things that consistently break agentic XR sessions, grouped by stack.

Cross-stack

Unity-side

IWSDK-side

§ 9Agentic failure modes

The pitfalls in § 7 are stack-specific. The failures below are intrinsic to the agent itself — the same patterns of error appear regardless of whether you’re on Unity, IWSDK, or anything else. Naming them is half the defence.

§ 10Researcher-specific concerns

Three concerns that don’t apply to consumer dev but are load-bearing for anyone building a research apparatus, study stimulus, or instructional material with agents in the loop.

Reproducibility

An agentic build six months from now should produce the same artifact — otherwise the apparatus drifted between Wave 1 and Wave 2 of your study. Defence:

• Commit the plan files (pattern 01) into the repo, not the chat log.
• Pin SDK versions explicitly (@iwsdk/core: 0.3.1, not ^0.3).
• Snapshot the agent build outputs (gltf hashes, ECS component lists) into docs/build-receipts/ per release.
• Record the model and revision used for each major build (claude-opus-4-7, date) in a release note. Model behaviour changes invisibly across revisions.

Pre-registration and IRB

If your study’s stimuli were partially shaped by an LLM, that fact is part of the methods section. For pre-registered studies:

• Pre-register the build artifact, not the build process — the participant interacts with a deterministic apparatus regardless of how it was made.
• Record the prompts and plans that produced experimental stimuli, in the same way you would record the design rationale of a paper instrument.
• If the agent helps tune sensitive parameters (difficulty curve, scaffolding density), treat those as pilot-derived parameters and lock them before main data collection.

Participant data near agent context

§ 12Downloads

Public-safe templates and starter scripts you can drop into a new project. Nothing here contains client secrets, participant data, or non-public IP.

§ 13References

Official sources

• IWSDK repo — github.com/facebook/immersive-web-sdk — “WebXR made simple. Full-featured framework with interactions, locomotion, and spatial UI. Powered by Three.js.”
• IWSDK docs — iwsdk.dev — tagline: “AI-native WebXR development.”
• Unity AI Assistant — unity.com/ai (verify current beta / pricing status before adopting)
• Meta Quest Developer Hub — developer.oculus.com
• Anthropic Claude Code — docs.claude.com/claude-code

The Dilmer demo

• YouTube: “How to Build Full VR/MR Games With Unity AI + Meta Agentic Tools!” (Dilmer Valecillos, Meta DevRel, 2026-05-03, 17:01).
• Channel: Dilmer Valecillos

Disclosures & verification

• This is a practical onboarding guide written by an external researcher; not affiliated with Meta, Unity, Anthropic, or any vendor mentioned.
• IWSDK repo URL, package name, and one-liner verified at github.com/facebook/immersive-web-sdk on 2026-05-04 (official description: “WebXR made simple. Full-featured framework with interactions, locomotion, and spatial UI. Powered by Three.js.”). Tagline at iwsdk.dev: “AI-native WebXR development.”
• Virtual Makerspace claims are from the author’s own working repo at Educatian/virtual-makerspace; the breadboard / 27 mm snap / 192 sockets / Phase 1–3 details are checked against README.md, docs/mvp-scope.md, and docs/pedagogy-charter.md.
• Unity-side claims trace to the Dilmer Valecillos video and the author’s study notes — not to direct hands-on Unity AI use. Take the Unity workflow as second-hand reporting, the IWSDK workflow as first-hand.
• WebXR cross-platform support details (visionOS feature flag, etc.) cross-checked against immersiveweb.dev as of 2026-05-04.

Author projects

• Educatian / virtual-makerspace — research-grade IWSDK study apparatus.
• Circuit Playground — sibling K-12 product on IWSDK, in PA_Research/circuit-playground/.

Pedagogy / theory

• Kapur, M. (2008, 2014). Productive failure in mathematical problem solving. Cognition and Instruction; Educational Psychologist.
• Schneider, B., Pea, R., Sharma, K. (2017–2024). Gaze coupling in CSCL. International Journal of Computer-Supported Collaborative Learning.
• Makransky, G., & Petersen, G. B. (2021). The Cognitive-Affective Model of Immersive Learning (CAMIL). Educational Psychology Review.

Further viewing — videos

Curated companion videos. Each is a click-to-load facade — no YouTube traffic until you start playback.

Your First Steps with Immersive Web SDK Meta Developers

Deploy Your WebXR App with IWSDK + Submit as PWA XR Bootcamp

World Model WebXR with Spark.js 2.0 + LOD + IWSDK XR Bootcamp

Build Your First VR/MR App w/ Dilmer Valecillos (Meta) Devpost · Builder Sessions

WebXR / VR / AR with Three.js (foundation series) Nik Lever

Full VR/MR Games with Unity AI + Meta Agentic Tools Dilmer Valecillos · 17:01

Further reading — documentation & tooling

• Meta IWSDK overview — developers.meta.com/horizon/documentation/web/iwsdk-overview
• IWSDK AI-Assisted Development Tooling (Meta official) — developers.meta.com/.../iwsdk-ai-assisted-dev-tooling
• “Accelerate VR Development with AI & Immersive Web SDK” (Meta blog) — developers.meta.com/horizon/blog/...
• meta-quest/agentic-tools (Apache-2.0, 13 skills + hzdb MCP) — github.com/meta-quest/agentic-tools
• Unity AI-powered Meta Quest setup (Meta tutorial) — developers.meta.com/.../unity-tutorial-ai-vr-setup
• HZDB MCP setup for Meta Horizon OS — developers.meta.com/.../ts-mqdh-mcp
• CoplayDev/unity-mcp (community Unity MCP, MIT, 9.1k stars) — github.com/CoplayDev/unity-mcp
• IvanMurzak/Unity-MCP (alt community Unity MCP, 2.5k stars) — github.com/IvanMurzak/Unity-MCP
• Anthropic — Build an MCP server (official tutorial) — modelcontextprotocol.io/docs/develop/build-server
• Claude Code — MCP integration — code.claude.com/docs/en/mcp
• felixtrz/elixr (IWSDK’s predecessor by the same author) — github.com/felixtrz/elixr

All references current as of 2026-05-04. Agent stacks change quickly; treat the specific tool versions as illustrative, the patterns as durable.

§ 14Glossary

Acronyms and shorthand used throughout this guide.

§ 15Adjacent work

This page sits inside a larger set of teaching, assessment, and workflow notes. The links below are the most useful adjacent reads if you want the same agentic pattern language applied to different problems.

Related guides in this site

External anchors

• iwsdk.dev for the current IWSDK framing and tool surface.
• meta-quest/agentic-tools for the broader Claude Code skill bundle and hzdb MCP pattern.
• ahujasid/blender-mcp for the asset-pipeline MCP this guide leans on.
• gpt5-coding-examples.vercel.app for one-shot browser-game references and prompt structure.

Playwright checks and page snapshots

The guide already has a screenshot-heavy structure. These page captures are kept in the repo so the layout can be sanity-checked without opening a browser each time.

Adjacent work · desktop

Adjacent work · mobile

Companion visuals

These are not game examples. They are the closest adjacent artifacts in the repository: workflow slides, oversight notes, and project-level coordination graphics that can be linked from the same article.

agentic-overview

claude-codex-mcp-workflow

weekly-lab-workflow

agentic-oversight