# Astra / Argon visual comparison — 17 October 2026

Requested by the repository owner: emphasize images and 3D; use **Codex CLI and agy CLI**. Main was fast-forwarded to `e55d7b5` before work began.

## Primary protocol

- Codex CLI **0.162.1**: `gpt-6-astra`, reasoning effort `high`, fresh ephemeral invocation, read-only sandbox, JSONL events and final response saved.
- agy CLI **1.4.0**: `gemini-4-argon-high`, effort `high`, fresh invocation, terminal sandbox enabled, JSONL events saved. An isolated test project is used for locating the reference image. No permanent command allowlist was added.
- Five identical artifact briefs. The image input has a different absolute path in each task directory; it is the same PNG. Codex receives it as an attachment, while agy can read it through its image-reading tool.
- Models return complete source. No shell, browser, file-writing, web, native image generator or delegated agents in these five trials. The evaluator strips only a surrounding Markdown code fence, then saves the answer without repairing it.
- Supplied runtime: Three.js **0.180.0**, including OrbitControls and GLTFExporter. No remote fonts, images or libraries.
- One invocation per model per task, no evaluator feedback, no selective reruns. Timeout: 1,500 seconds. This small convenience sample does not establish a general model ranking.
- CLI system prompts, global skill discovery, tokenizer, internal reasoning policies and image attachment mechanisms differ. The same effort label is not a guarantee of identical reasoning budgets.

## Tasks and evidence

1. **Text → 3D:** miniature Thai canal market; exactly three stalls and one boat, curved bridge, day/night and water level controls, real GLB export.
2. **Image → 3D:** reconstruct a synthetic dimensioned room plan; preserve spatial positions and footprints, top/perspective view, dimension toggle and GLB export.
3. **Vector illustration:** a rainy forest observatory, three mountain ridges, bicycle and one amber window, paper-cut style, SVG 1200×630.
4. **Technical vector illustration:** fictional camera exploded into five separated assemblies along one optical axis; aperture has six blades, SVG 1200×630.
5. **Simulation:** projectile motion with gravity 9.81 m/s², adjustable speed/angle, analytic trajectory, range/height/time readouts and static GLB export. This is not a reproduction of Google's robot-control experiments.

An additional **native raster image availability trial** asks each CLI to use its existing image tool. It is reported separately: an image tool can call a different model, and language-model token pricing cannot stand in for undisclosed image-generation charges.

Evaluation captures the original output in a fresh Chrome profile with external requests blocked. Checks include JavaScript errors, real meshes, camera orbit, mobile overflow, controls, GLB header and round-trip loading. SVG structure is checked separately from visual review. A working export is not a certification of watertight topology, printability or engineering safety.

## Usage and money

Read actual CLI usage events. Do not estimate tokens from characters. Keep cache reads and reasoning subsets separate and never add reasoning twice. Preserve unavailable fields as unknown.

Pricing snapshot: `https://openrouter.ai/api/v1/models`, retrieved 2026-10-17. Standard USD rates per million tokens:

| Model | Uncached input | Cache read | Output |
|---|---:|---:|---:|
| openai/gpt-6-astra | 10 | 1 | 50 |
| google/gemini-4-argon | 2 | 0.10 | 10 |

Formula: `(uncached_input × input_rate + cache_read × cache_rate + output × output_rate) / 1,000,000`. Check each CLI's cache semantics before mapping its fields. Astra's long-context override is per request, not the sum of all requests. Cache-write charges, image tools, tax, FX, subscription allocation and unreported internal services are not silently treated as zero.

These are **OpenRouter-equivalent estimates**, not an OpenRouter invoice. The trials use existing CLI subscriptions. They do not establish incremental subscription charges or what a different API route would consume.

## Excluded setup and pilot attempts

An initial API pilot preceded the user's CLI requirement: an Argon readiness call plus one market task per model. Its raw responses are retained outside the repository. It is not used for the primary quality comparison. Provider-reported pilot charge: $1.272406 (including $0.000122 readiness).

An initial CLI agent workflow exposed agy's non-interactive permission behavior: `accept-edits` soft-denied commands and `auto` was unavailable behind a rollout flag. A subsequently proposed permanent command allowlist was rejected by automatic approval review; no such allowlist was installed. The final source-only protocol requires no command grants. The initial Codex agent artifact and any measurable setup usage are retained separately and excluded from the primary totals.

## Files and publication

Only sanitized metrics, prompts, original source artifacts, generated exports, screenshots and licensing information belong in the deliverable. Raw CLI logs can contain private environment details or internal reasoning and stay local. No credentials, private account identifiers, internal reasoning text or raw provider logs are committed. Video files and raw media stay outside the repository.

## Reproduce the primary trials

Use `scripts/benchmarks/astra-argon-cli-source.py`. Supply an output directory outside the repository and a fixture directory containing `reference-plan.png` and `vendor/` (the published demo includes both). For Argon also pass `--agy-project YOUR_EXISTING_PROJECT_ID`. The CLI uses your existing account; do not paste credentials into prompts. Omitting `--execute` writes prompts and invocation metadata without starting a model call.

```sh
python3 scripts/benchmarks/astra-argon-cli-source.py --model astra --output /tmp/visual-results --fixtures /tmp/visual-fixtures --execute
python3 scripts/benchmarks/astra-argon-cli-source.py --model argon --agy-project YOUR_EXISTING_PROJECT_ID --output /tmp/visual-results --fixtures /tmp/visual-fixtures --execute
python3 scripts/benchmarks/summarize-cli.py --root /tmp/visual-results --prices docs/benchmarks/astra-argon-20261017/pricing-snapshot.json --out /tmp/visual-costs
```

Serve the results over localhost, then run `inspect-visuals.mjs ROOT BASE_URL PLAYWRIGHT_MODULE_DIRECTORY`. It launches the installed Google Chrome on macOS. The evaluator uses GLTFLoader as well as the model's supplied OrbitControls/GLTFExporter; include that addon and its imports in your evaluation fixture. Keep raw logs private. The public `evaluation.json` files contain sanitized measurements, not hidden reasoning.

Public prompt files normalize the reference image's absolute path to `./reference-plan.png` for privacy. The original prompt hash in results.json hashes the prompt actually sent, before this path normalization. Artifact hashes are computed before the web-only importmap path change.