Video Synthesis ANNs
A coordinator-based approach to AI video: a small Director ANN plans the shot list, then Camera, Motion, Depth, FX, Audio, and Quality ANNs do their part. Each is a published, versioned, rateable module in the Aigarth marketplace. This page is the live view of the proposal, the architecture compatibility verdicts, the development roadmap, the cost model, and the risk register.
Director, Camera, Motion, Depth, FX, Audio, Quality
render, motion, depth, fx, audio, quality
ann_pipelines, ann_pipeline_runs, ann_feedback_events, compute_workers, ann_version_metrics
evaluation, blog, dashboard, tracker
ANN registry, versioning, reviews, marketplace listings, deploy hook — all ready in services/ann. We need a `role` enum, a `pipeline` table, and a `pipeline_runs` table to model Director/Camera/Motion as first-class roles.
Add `ann_role` enum, `ann_pipelines`, `ann_pipeline_runs`, `ann_feedback_events`. ~3 new tables, 6 new endpoints. No existing flow changes.
services/compute already does typed jobs, clusters, regions, reservations, and idempotency. The worker-side story is greenfield — we need a worker protocol, capability advertisement, and sandboxed execution.
Add `compute_workers` table, extend `jobs.kind` enum with 6 new kinds (render, motion, depth, fx, audio, quality), add `/v1/workers/*` namespace + long-poll `/v1/jobs/next`.
Nothing in the current system stores per-ANN quality signals, user feedback attribution, or version-over-version improvement. This is the largest single gap and the part that turns the use case from a demo into an evolutionary system.
Add `ann_feedback_events` table (kind, value, attributed to ann_version + pipeline_run) and a `ann_version_metrics` materialized view. Cron job to refresh. ~1 new table per service, 1 rollup job.
The shape is right — services/qubic already has wallets, stakes, treasury, validators, network. On-chain ANN ownership, worker registry, and reputation map cleanly to Qubic smart contracts. But K12 signature verification is format-only today, so on-chain writes are not yet safe.
Ship real K12 signature verification (precondition). Then deploy on-chain `ann_registry`, `worker_registry`, `reputation`, `rewards` contracts. Off-chain rollup commits to chain once per day.
Reads the prompt, plans the shot list. Returns a structured JSON describing scenes, durations, camera, audio, and style.
(LLM call, not a compute job)Turns each shot into a 3D camera path through space, with lens parameters. Knows 50 cinematic moves and when to use each.
(rule-based, no GPU)Generates in-between frames from a keyframe and a camera path. Pure interpolation, deterministic, fast on CPU.
motion.frame_interpolateProduces per-frame depth maps. Foreground / background separation for parallax and depth-of-field. Uses MiDaS on CPU.
depth.midas_smallComposites particles, steam, light glows, atmosphere. Domain-specific — a Steam FX ANN and a Rain FX ANN can coexist.
fx.compositeAligns narration, music, and sound effects to the timeline. Knows when to dip the music under a voice.
audio.alignWatches the result. Spots artifacts, weird motion, dropped frames. Decides whether to ship or to flag a stage for re-run.
quality.evaluateEvery stage emits a structured event into ann_feedback_events. The Quality ANN at the end can flag any stage for re-run; only that stage re-executes, not the whole pipeline.
Architecture evaluation, blog article, this dashboard. No code in production.
- Architecture compatibility evaluation (4 dimensions, color-coded)
- Blog article on the Aigarth Cloud website
- Dashboard /video page live with the architecture diagram
- Risk register + cost model
Stakeholders aligned. Decision: proceed to Phase 1.
Single Python process acting as Director + Camera + Motion + Render. A prompt in, an MP4 out, every stage recorded.
- `role` enum + `ann_pipelines` + `ann_pipeline_runs` tables
- Director ANN (LLM call)
- Camera ANN (rule-based)
- Motion ANN (OpenCV affine transforms)
- Render ANN (FFmpeg)
- Mock pipeline-run endpoint
- 30-second sample render: 'city at sunset'
30s MP4 from a 1-paragraph prompt in under 10 minutes. Decision gate.
Publish ANNs as real listings. Sell pre-composed pipelines. Wire up per-version metrics and reviews.
- Director / Camera / Motion / Render ANNs published as listings
- `kind = 'pipeline'` listing type
- `ann_version_metrics` materialized view + cron
- Per-ANN quality ranking in the marketplace
Users buy and run a 5-ANN pipeline end to end without help.
Worker registry, worker protocol, `aigarth/worker-video` image. Real per-stage cost.
- `compute_workers` table + `/v1/workers/*` namespace
- `POST /v1/jobs/next` long-poll + `POST /v1/jobs/:id/progress`
- `aigarth/worker-video` Docker image (Python + OpenCV + FFmpeg + PyTorch CPU)
- Per-worker reputation scoring from feedback events
- 3-worker local cluster for E2E testing
Workers earn reputation; misbehavior is slashed; per-stage cost is real money.
On-chain ANN ownership, worker registry, reputation, rewards. Off-chain rollups committed to chain.
- Real K12 signature verification (precondition)
- On-chain `ann_registry`, `worker_registry`, `reputation`, `rewards` contracts
- Daily off-chain rollup → on-chain state root commitment
- QUBIC reward distribution: 70% worker, 20% ANN author, 10% treasury
An ANN's ownership and a worker's reputation are verifiable on-chain.
| Stage | Cost (QU) | Latency | Note |
|---|---|---|---|
| Director (LLM call) | 0.05 | 2 s | One round-trip to gateway |
| Camera (rule-based) | 0.01 | < 1 s | Pure function |
| Motion (OpenCV, CPU) | 0.50 | 60 s | 900 frames × 65 ms/frame |
| Depth (MiDaS, CPU) | 0.30 | 45 s | 900 frames × 50 ms/frame |
| FX (compositing) | 0.10 | 10 s | 900 frames × 11 ms/frame |
| Audio (alignment) | 0.05 | 5 s | One short clip |
| Quality (model) | 0.10 | 15 s | One full pass |
| Render (FFmpeg) | 0.05 | 30 s | Encode h264 |
| Total | ~1.16 | ~3 min | per-stage cost visible to user before commit |
The cost is dominated by Motion and Depth (every frame). Per-stage pre-flight estimates let the user decide: pay for a faster worker, swap in a cheaper ANN, or skip the stage.
Mitigation: Per-stage pre-flight estimate, per-stage credit deduction, pre-purchased capacity via reservations.
Mitigation: Quality ANN at the end of every pipeline; per-stage retry; A/B two Director ANNs in shadow.
Mitigation: Docker sandbox; no network egress except Aigarth APIs; resource caps; reputation + slashing.
Mitigation: Per-stage `attribution` field on each feedback event; Quality ANN emits a 'weakest stage' verdict.
Mitigation: Streaming low-res preview during render; final-render handoff; 'cancel + partial refund'.
Mitigation: Precondition. Implement using @noble/curves (already a dep). Blocks Phase 4 entirely.
Mitigation: Seed with 5–8 production-quality ANNs before opening the marketplace.
Mitigation: Generation provenance attached to every artifact; takedown workflow via audit log.
Phase 0 is the documentation you just read. Phase 1 is the centralized prototype — a 30-second MP4 from a one-paragraph prompt, generated end to end by the Aigarth platform. Re-evaluation gate at the end of Phase 1.