Four Billion Parameters of Three.js: Benchmarking a 4B 3D-Coder's Raw Output and Its Self-Repair Loop

Small coding models fine-tuned for 3D scene generation are having a moment. One of them — Qwen3.5-4B-3D-Coder, a 4-billion-parameter fine-tune that emits complete, self-contained Three.js HTML files — runs entirely on a laptop GPU and even ships a restored multi-token prediction (MTP) head for ~2× decode speedups. That combination makes it one of the more interesting sovereign-AI artifacts of the moment: a fully local pipeline that turns a text prompt (or an image) into an animated WebGL scene.

We ran it through a proper evaluation on an RTX 4080 Laptop (12 GB): three scene-generation tasks, headless-Chrome verification, and a benchmark suite. The headline numbers: 0 of 3 scenes ran unpatched — every single generation contained at least one hallucinated API — but a self-repair loop that feeds browser errors back to the same model fixed all three in one round. Here is exactly what broke, what worked, and how to build the loop yourself.

The setup

Everything ran locally on a single machine — no cloud, no API keys:

Component	Choice
GPU	RTX 4080 Laptop, 12 282 MiB VRAM
Runtime	llama.cpp b11538, CUDA 12.4
Model	SuperLogic/Qwen3.5-4B-3D-Coder-MTP-GGUF, Q8_0 (4.72 GB)
Vision	mmproj-Qwen3.5-4B-BF16.gguf (676 MB projector)
Context	28 672 tokens, q8_0 KV cache, flash-attention
Speculative decoding	--spec-type draft-mtp --spec-draft-n-max 4

The base ccore release ships the fine-tune without its MTP head; the SuperLogic variant restores it byte-identically and re-quantises to Q8_0. llama.cpp engages the draft head at load time (creating MTP draft context against the target model) and reports per-request acceptance statistics — the definitive proof that speculative decoding is actually working, no microbenchmarks required.

llama-server -m Qwen3.5-4B-3D-Coder-MTP-Q8_0.gguf \
  --mmproj mmproj-Qwen3.5-4B-BF16.gguf \
  -ngl 99 -fa on -c 28672 -b 2048 -ub 2048 --parallel 1 \
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-sampling probabilistic \
  -ctk q8_0 -ctv q8_0 --host 127.0.0.1 --port 18080

Benchmarks

Metric	Result
Prefill (text, 5 595-token prompt)	~2 560 tok/s
Image prompt processing (151 image tokens)	~988 tok/s
Generation, thinking disabled	84–89 tok/s
Generation, thinking enabled	68–69 tok/s
Full Three.js scene generation	47–55 tok/s
MTP draft acceptance (long runs)	0.43–0.50, mean draft length 2.0–2.7

Draft acceptance around 0.45 with a mean accepted length near 2.5 tokens per step is exactly what a single-layer MTP head delivers — roughly a 2× decode speedup over the same weights without the draft head. Notably, acceptance rises for repetitive content (city windows: acceptance 0.55) and falls for creative prose, which matches the theory: the draft head predicts locally-continuations well and novel content poorly.

One operational finding worth its weight: on a 12 GB laptop GPU, the 64 K context preallocation leaves only ~1.2 GiB of headroom for per-request compute buffers. Long repair requests (9 000-token prompt + 48 000-token generation budget) starved the decode loop to under 1 tok/s. Right-sizing the context to 28 672 with -b 2048 -ub 2048 restored full speed. And if the machine itself is hot — the same laptop throttles from 108 tok/s to 0.8 tok/s at 88 °C under sw_thermal_slowdown — no server configuration will save you. Benchmark on a cool machine.

The three scenes

Every claim below is grounded in three generations: a procedural night city from a one-line text prompt, an interactive solar system, and a voxel island produced from a single reference image via the vision projector. The embeds run the post-repair output verbatim — the raw generations never rendered anything, as the failure catalogue below shows — with one deployment concession: the importmap points at a locally vendored three r160, so the scenes also work without a CDN.

model output · post-repair

Run the interactive scene

Generated by Qwen3.5-4B-3D-Coder, repaired after one self-repair round. Served verbatim; only the importmap points at a locally vendored three r160.

model output · post-repair

Run the interactive scene

Generated by Qwen3.5-4B-3D-Coder, repaired after one self-repair round. Served verbatim; only the importmap points at a locally vendored three r160.

model output · post-repair

Run the interactive scene

Generated by Qwen3.5-4B-3D-Coder, repaired after one self-repair round. Served verbatim; only the importmap points at a locally vendored three r160.

For the vision path, the entire visual prompt was this reference image — no text description beyond "build this as a voxel scene":

What the model gets right

Credit where due — much of the generated output is genuinely impressive for 4B parameters:

• Complete, structurally valid HTML every time: DOCTYPE, meta tags, CSS styling, <script type="module">, animation loops. Roughly 20–28 KB of code per scene.
• Real Three.js fluency: procedural placement, InstancedMesh, ShaderMaterial with vertex/fragment shaders, OrbitControls, fog, emissive materials, HSL colour manipulation.
• Vision works end-to-end: given the model card's voxel-tree reference image, the vision projector produced a correct description ("blocky, voxel-style island... autumn tree with layered orange and yellow foliage") and then generated a complete voxel island in that palette.
• It is a reasoning model: every generation spends 10 000–13 000 tokens in reasoning_content before writing a line of code.

That last point is also the biggest usability trap. With max_tokens: 8000, all three of our scene requests returned empty content — the entire budget was consumed by thinking. For scene generation, budget max_tokens ≥ 15000, or disable thinking per request:

{
  "messages": [{"role": "user", "content": "Build an animated night city..."}],
  "max_tokens": 16000,
  "chat_template_kwargs": {"enable_thinking": false}
}

No-think mode is roughly 30 % faster and produces clean code for well-specified tasks; the thinking traces mainly pay off on the complex scene structure.

Where it fails: four hallucination classes

We evaluated the model's raw output — three scenes (procedural night city, interactive solar system, voxel island from an image reference), each verified in headless Chrome. None ran unpatched. The failures fall into four classes:

1. The importmap hallucination (all three scenes)

Every generation wrote the ES-module importmap as XML, not JSON — in three different dialects, no less:

<!-- scene 1 --><value>https://cdn.jsdelivr.net/npm/three@0.160.0/build/three.module.min.js</value>
<!-- scene 2 --><entry key="three" value="https://cdn.jsdelivr.net/npm/three@0.160.0/build/three.module.min.js"/>
<!-- scene 3 --><url>https://cdn.jsdelivr.net/npm/three@0.160.0/build/three.module.js</url>

An invalid importmap fails to register, so import * as THREE from "three" dies with Failed to resolve module specifier "three" and the page stays blank. This is the failure mode you would invent for a model that has seen far more XHTML than JSON-in-HTML, and it is a reminder that plausible-looking config syntax is where small models consistently misfire.

2. Invented APIs

The city scene constructed glows via new THREE.SpriteGeometry(1, 1) — a class that does not exist in Three.js. Fine-tuning on scene code apparently does not immunise against inventing constructor names.

3. Undeclared shader uniforms

The solar system's rim-glow ShaderMaterial passed uColor in the uniforms object but never declared it in the fragment shader source — a GLSL compile error, caught only at first render:

// fragment shader as generated: uses uColor, never declares it
gl_FragColor = vec4(uColor, rim * 0.48);
// error: 0:146: 'uColor' : undeclared identifier

4. Ghost identifiers and ordering bugs

The voxel island called isPond(x, z) — a helper the model never wrote — and declared const pathCells = new Set() after its first use, a temporal-dead-zone error queued behind the first crash.

The self-repair loop

Since every failure is caught deterministically by a browser, the natural architecture is a loop where the same 4B model debugs its own output — no larger model involved, which keeps the pipeline honest as an evaluation of what this model can do on its own.

// verify-render.mjs — headless render + error capture (Playwright, system Chrome)
const browser = await chromium.launch({ channel: "chrome", headless: true });
page.on("pageerror", (e) => errors.push("PAGEERROR: " + e.message));
page.on("console", (m) => { if (m.type() === "error") errors.push(m.text()); });
await page.goto(url, { waitUntil: "load" });
await page.waitForTimeout(9000); // CDN fetch + scene build + frames

The repair step posts the generated file plus the exact captured errors back to the same llama-server, with a system prompt that names the classic pitfalls (importmaps are JSON, every identifier must be declared, shader uniforms must appear in the shader source) and demands the complete corrected file.

Result: all three scenes rendered cleanly after a single repair round — zero page errors, zero console errors, canvas present. Pixel-level verification of the screenshots confirmed real scenes rather than black canvases: the night city shows 8 160 distinct colours with structured luminance variance across a 29.6 % lit area, and the voxel island's average pixel colour lands in the warm beige/orange palette of the reference image. The model even fixed its own temporal-dead-zone bug by hoisting the declaration — a fix many working developers get wrong.

Practical guidance

• Budget tokens like a reasoning model: max_tokens ≥ 15000 for scenes; expect 10–13k thinking tokens before any HTML appears. Empty content with finish_reason: "length" means the budget went to thinking — raise it, do not retry blindly.
• Right-size the context: 28 K tokens with modest batch sizes keeps a 12 GB GPU out of trouble. A 64 K preallocation on this class of hardware leaves too little headroom for large requests.
• Verify renders, do not trust them: static inspection ("it looks complete") missed nothing — but it also caught nothing. Headless Chrome with console capture found every failure deterministically.
• Acceptance stats beat microbenchmarks: llama.cpp prints draft acceptance per request. If you are not seeing mean accepted lengths above ~2, your speculative decoding is not engaged and you are benchmarking the wrong thing.
• Keep the raw output: storing the untouched generation next to the repaired one turns a demo into an evaluation. The delta between the two is the model's real bug profile.

Takeaway

A 4B model that writes 25 KB of structured Three.js from a sentence or an image — locally, at ~50 tok/s on a laptop GPU — would have been science fiction a couple of years ago. But the raw success rate is still zero, and that is the honest number to plan around. The failure classes are narrowly concentrated (config syntax, invented identifiers, declaration ordering), which makes them exactly the kind of thing a deterministic verifier plus a self-repair loop can mop up — as this one did, in a single round, with no model larger than the generator itself.

The next iteration of these fine-tunes will presumably train against executable verification rather than next-token plausibility, and the raw success rate should climb accordingly. Until then: generate, verify in a real browser, hand the errors back to the model, and let it fix itself. The loop is ~100 lines of code, and it converts a demo into a working pipeline.