Mixed reality live production becomes more dependable when every signal, tracking reference, cue, and fallback is planned before the crew enters the venue. This reusable checklist covers camera tracking, physical performers, virtual sets, AR graphics, compositing, audience-facing displays, rehearsals, and common failure modes so producers can build a workflow that is easier to test and recover.
Overview
A mixed reality event combines real camera footage with computer-generated environments or graphics. Depending on the show, the physical scene may include a presenter, musician, product, or set piece, while the virtual layer adds scenery, data graphics, a digital avatar, or an interactive effect. The audience may see the result through a broadcast, livestream, venue display, mobile device, headset, or a combination of formats.
The central production challenge is not simply creating attractive graphics. The camera, tracking system, render engine, keying or compositing system, audio chain, and final display must agree about time, position, scale, and output format. A small mismatch can make a virtual object appear to slide, float, or pass through a performer.
Before choosing equipment, define the audience output. A camera feed composited for a conventional livestream has different requirements from a spatial stream intended for a compatible device or a large venue display. For format planning, compare the options in Volumetric Video vs Spatial Video vs 360 Video and document the selected output in the technical plan.
Basic signal flow
Performer and set
↓
Camera + lens + tracking data ──→ Render engine / virtual scene
↓ ↓
Camera signal ───────────────→ Keying and compositing
↓
Program switcher / record
↓
Broadcast, livestream, venue display, archive
Keep a version of this diagram with the show file. Add the actual devices, connection types, operator positions, and backup paths. If a signal bypasses the render engine, reaches a confidence monitor, or feeds a separate audience display, show that explicitly.
Checklist by scenario
1. Physical performer with a virtual set
- Define the performer’s standing, walking, and gesture areas on the physical floor.
- Mark the usable camera zones and identify any angles that reveal tracking markers, set edges, or lighting equipment.
- Measure the real floor, key props, and camera height so the virtual set uses a consistent scale.
- Confirm the keying method, background treatment, spill control, and wardrobe restrictions during a camera test.
- Prepare a clean camera feed, composited program feed, and a backup view for the director or operator.
- Test shadows and reflections. A virtual floor may need a designed shadow treatment rather than relying on the physical stage.
2. Tracked camera with AR graphics
- Record the camera body, lens, sensor, lens profile, focus range, and zoom range in the technical sheet.
- Confirm that tracking data is routed to the intended render engine and that its coordinate system matches the scene.
- Decide which graphics are world-locked, screen-locked, or attached to a performer. These categories require different behavior when the camera moves.
- Set safe areas and readability rules for graphics that will also appear in a conventional livestream or venue feed.
- Assign an operator to monitor tracking quality rather than expecting the graphics or camera operator to notice every drift.
3. Virtual performer or digital avatar
- Specify whether the avatar is pre-rendered, driven by a performer, generated in real time, or controlled by a cue system.
- Test facial, body, hand, and lip-sync inputs separately before combining them.
- Prepare a neutral idle pose, a safe end state, and a static backup graphic if the live performance system fails.
- Check the avatar’s scale, floor contact, shadows, occlusion order, and interaction with physical presenters.
- Keep a clean version of the scene without optional effects so the operator can reduce render load quickly.
4. Audience-facing venue display
- Confirm the display’s aspect ratio, refresh behavior, brightness controls, viewing zones, and acceptable input formats with the venue team.
- Test the audience sightlines from the front, sides, rear, and any elevated seating areas.
- Separate the venue output from the broadcast output when each needs a different crop, delay, or graphic treatment.
- Plan cable paths, power distribution, signal conversion, and a local fallback source.
- Run the complete chain at show resolution and frame rate instead of approving it from a laptop preview.
5. Remote or spatial streaming audience
- Document the target devices, delivery format, expected latency, and whether the audience receives a flat video, spatial video, or a rendered viewpoint.
- Check encoding, transport, and playback compatibility as one workflow. A successful render does not guarantee a successful stream.
- Measure the available venue upload path during the same operating period as the event, if possible.
- Prepare a lower-complexity output for network congestion or unsupported playback.
- Use the spatial streaming bandwidth and latency checklist and the event Wi-Fi and network planning guide when documenting delivery assumptions.
Camera-tracking calibration and equipment checklist
Calibration should be treated as a repeatable procedure, not a one-time setup task. Recheck it after moving the camera, changing a lens, altering the tracking reference, rebuilding the physical set, or changing the virtual scene scale.
Equipment and software
- Camera body, lenses, lens data, tripod or dolly, and suitable support hardware.
- Camera-tracking system, reference markers or tracking environment, and a visible tracking-quality monitor.
- Video capture or input hardware, render workstation, compositing tools, graphics control, and program switcher.
- Genlock or another agreed timing method where the workflow requires synchronized devices.
- Audio mixer, timecode or timing reference if used, confidence monitors, intercom, and recording devices.
- Backup camera view, backup media, spare cables, power protection, and a tested fallback scene.
Calibration steps
- Lock the camera configuration. Record the lens, focal length, focus distance, aperture, sensor mode, resolution, and frame rate.
- Verify the physical reference. Ensure markers or tracking features are stable, visible, evenly distributed, and not blocked by crew or set pieces.
- Calibrate lens characteristics using the system’s supported process. Pay particular attention to distortion and zoom behavior.
- Align the tracking coordinate system with the physical floor, stage origin, and virtual scene origin.
- Place a known virtual object beside a known physical object. Move the camera slowly and check that their relative position remains stable.
- Test a pan, tilt, dolly, and any planned crane or handheld movement. Watch for lag, jumps, scale changes, and loss of tracking.
- Save the approved calibration with a clear name and protect it from accidental overwriting.
- Repeat the check after venue installation and again after the final lighting, set dressing, and cable runs are complete.
Rehearsal schedule and cue sheet
A practical rehearsal separates technical validation from creative pacing. A suggested sequence is:
- Prelight and systems check: verify power, network, camera inputs, tracking visibility, render nodes, audio, displays, and recording.
- Calibration rehearsal: lock the camera and lens configuration, align the scene, and approve a short movement test.
- Blocking rehearsal: walk performers through marks, sightlines, interaction zones, and safe positions if tracking or compositing is lost.
- Technical run: execute every cue with graphics, transitions, audio, camera moves, recording, and the intended output feeds.
- Failure run: deliberately test the fallback scene, backup camera, static graphic, lower-complexity render, and communication path.
- Full run: perform the show without stopping unless safety or equipment protection requires it. Record the program for review.
Reusable cue sheet fields
For each cue, record: cue number, time or trigger, camera, performer action, virtual scene, AR graphic, audio state, output destination, operator, confidence-monitor expectation, fallback action, and approval status. Add a notes column for latency, occlusion, tracking quality, or timing issues observed during rehearsal.
What to double-check
| Failure mode | Early warning | Immediate response |
|---|---|---|
| Tracking drift | Virtual objects slowly slide or lose alignment during movement. | Hold the camera, switch to a known safe angle, and recalibrate or use the fallback composition. |
| Occlusion errors | Graphics appear in front of a performer or prop when they should pass behind it. | Check depth masks, keying, scene order, and the performer’s position relative to the defined interaction zone. |
| Timing or sync mismatch | Audio, camera motion, graphics, or remote actions feel visibly late or early. | Identify the affected path, use the approved timing reference, and move to the output with the most stable sync. |
| Render overload | Frame rate drops, effects pop out, or operator controls become unresponsive. | Disable optional effects, reduce scene complexity, or load the tested lightweight scene. |
| Signal or network interruption | Frozen frames, dropped inputs, encoder warnings, or playback buffering. | Switch to the local backup source or lower-complexity delivery path and notify the director using the agreed call. |
Also check the human workflow. Every operator should know who can call a hold, who activates the fallback, and which feed the director should select. Technical recovery is slower when responsibility is ambiguous.
Common mistakes
- Designing before defining the output: A scene built for a venue screen may not translate cleanly to a spatial or conventional livestream. Set the delivery targets first.
- Approving a stationary test only: Alignment that looks correct on a locked shot can fail during a push-in, pan, or change of focal length.
- Ignoring the physical floor: Virtual objects need believable scale, contact, and interaction points. Mark these on the real stage.
- Leaving fallback decisions until show day: A backup is useful only when it has been loaded, tested, assigned, and explained to the crew.
- Adding effects without a render budget: Keep a simpler scene available and test it under the same conditions as the full show.
- Changing camera settings after calibration: Lens, frame-rate, resolution, or focus changes can invalidate assumptions in the tracking and render pipeline.
When to revisit
Use this checklist before seasonal planning cycles, each new venue installation, every major camera or lens change, and any event that adds a new display or streaming destination. Revisit it when the render engine, tracking system, capture hardware, plugins, codecs, streaming platform, or minimum workstation requirements change.
For a repeat production, keep a dated technical record containing the approved signal-flow diagram, calibration files, scene versions, cue sheet, network assumptions, rehearsal notes, and failure decisions. Review that record before the next event rather than assuming the previous setup still applies.
As a final action, schedule three short checks: one during preproduction, one after venue installation, and one immediately before the technical run. Confirm the output format, camera configuration, tracking alignment, sync, render headroom, fallback path, and operator assignments at each check. If the show includes volumetric assets or long-term recordings, pair this process with the volumetric video storage and archiving workflow. A disciplined review makes mixed reality live production more repeatable without limiting creative choices.