WorldCast: World-Consistent Character Video Generation with Large-Angle Camera Control

WorldCast. World-Consistent Character Video Generation with Large-Angle Camera Control. By Lei Cao and nine others, from Wuhan University and Xiaomi. Published on the twenty-third of October. Its code repository is worked out: about forty source files, with a readme and a licence.

Cinematic character video depends on precise camera control, and existing methods have trouble when the camera moves a lot. Large-angle trajectories that combine rotation, translation and a changing radius, starting from an arbitrary initial azimuth, are hard to follow. A big viewpoint change also exposes parts of the character that the initial frame never showed. Those unseen regions are where things go wrong: geometric distortion, degraded appearance, and identity that drifts away from the person you started with. The two difficulties are coupled. Following the camera reveals the unknown, and the unknown is what breaks consistency.

The paper attacks this from two directions. To study the problem systematically it introduces CLACT, a large-scale character-centric dataset and benchmark covering diverse large-angle camera trajectories, with accurate camera geometry and multi-view observations of each character for training and evaluation. On top of that sits WorldCast, a framework for world-consistent character video under those same motions. Its premise is to give the generator explicit geometry for what can be seen, and a memory of the character for what cannot.

The geometry comes first. WorldCast back-projects the initial frame into a 3D point cloud using estimated depth, aligns the scale of that point cloud with the target camera trajectory, and renders it along the path. The renders act as explicit geometric guidance, and they come with coverage masks that mark what the initial frame actually covers.

The character information that the point cloud cannot supply is filled in by Geometry-Aligned Memory Attention, or GAMA. Relative camera poses are estimated jointly from the initial frame and from multi-view character anchors, so the anchors and the target view can be expressed in one common coordinate frame. GAMA uses that alignment to decide which character features are relevant to the current view, retrieving them and injecting them during generation.

The evaluation spans diverse large-angle camera trajectories. The reported result is accurate trajectory following, achieved while character geometry, appearance and identity are preserved. The abstract gives no figures, so the claim stands in qualitative form: the camera goes where it is told, and the character stays the same character.

Each half of WorldCast answers one half of the coupled problem. A rendered point cloud handles the visible world and says where the camera is. Memory attention, aligned in the same coordinates, handles the hidden side of the character. The dataset, built around exactly these large-angle motions, gives both halves something to be trained and measured against.

WorldCast. World-Consistent Character Video Generation with Large-Angle Camera Control. By Lei Cao and nine others, from Wuhan University and Xiaomi. Published on the twenty-third of October. This is the long version: the whole paper rather than the abstract.

Picture a shot where the camera swings all the way around a character while pulling back, starting from whatever direction the character happens to face. Camera-controlled video models struggle with exactly this, and the paper names two failures that arrive together. The first is the path: big rotations combined with translation and a changing radius produce under-rotation, wrong translation, and drift. The second is the character. As the camera comes round it exposes regions the first frame never showed, so geometry, appearance and identity begin to wander. The authors argue this is one problem, because the target trajectory decides both the geometry the video must follow and which unseen parts of the character must be invented at each moment.

The generator underneath is Wan two point one, an image-to-video diffusion transformer trained with flow matching: it learns to carry random noise toward a clean video, guided by text and the first frame. Everything WorldCast adds is extra conditioning fed into that model.

The data did not exist, so the authors built CLACT. Existing camera-motion collections cover large combined trajectories thinly, and never pair them with other views of the same character. CLACT's training side holds fifty thousand sequences rendered in Unreal Engine, with arcs, full orbits, orbit-and-dolly and free-form paths. Another thirty thousand Blender samples add front, back, left and right views of each character, which the paper calls anchors. Evaluation uses forty-two scenes, synthetic and real, each under six trajectories.

The first half of the method is geometry. A pretrained estimator predicts depth for the first frame, every pixel is lifted into a coloured point cloud, and the cloud is rendered along the target trajectory. Each rendered frame gives a colour guide seen from the right viewpoint, plus a coverage mask marking pixels the cloud cannot fill, left for the generator. Both are stacked with the noisy video latent. Every pixel also gets a Plücker ray, its viewing direction combined with the camera's position, so the model is told the camera explicitly.

One detail the abstract compresses is scale. Predicted depth and camera translation may follow different conventions, and if they disagree the rendered parallax stops matching the real training video, so guidance contradicts supervision. During training the authors render the cloud at nine candidate scales, track image motion with optical flow in each render and in the real video, and keep the best match. At inference there is no reference video, so no calibration happens.

The second half is Geometry-Aligned Memory Attention, or GAMA. The character is segmented from the first frame and each anchor, and a pretrained multi-view geometry model estimates all their camera poses jointly. A similarity transform, fitted on matching camera centres with the first frame as the shared view, maps those poses into the target trajectory's frame. Each anchor then gets its own Plücker rays through the very same camera projection the video uses.

Twenty memory-attention blocks are inserted after every other backbone block. Video and memory tokens attend jointly, but only video tokens are updated, through an output projection initialised at zero, so training starts from the untouched pretrained model. Camera control is trained first; then only the memory bank and anchor adapter.

Against twelve baseline configurations spanning camera-controlled generation, novel-view generation and video re-rendering, WorldCast has the lowest error on all three camera metrics. Its absolute trajectory error is zero point zero six five, against zero point zero nine eight for the best baseline, Uni3C. MEt3R geometric inconsistency falls to zero point two two eight, and ArcFace identity similarity reaches zero point five five five, against zero point four three three for the best baseline. Visual quality is competitive rather than leading: Lyra and DepthDirector each edge it on one quality measure.

The comparison that best tests the thesis is quieter. Two novel-view baselines were rerun with the same extra character views. Their gains were limited or inconsistent, and MVGenMaster's trajectory error got worse. Extra views alone do not help. The ablations agree. Removing anchor alignment raises rotation error to one point three seven four, worse than removing the whole memory branch, at one point one four nine, while identity barely moves. And dropping the Unreal Engine sequences more than triples first-stage trajectory error.

The paper has no limitations section. Its conclusion lists future work instead: more flexible ways of getting anchors, more complex dynamic scenes, and longer or interactive trajectories. The first points at a real requirement: WorldCast is evaluated with four anchor views beside the first frame, not a single photograph. Results are single numbers with no error bars or repeated runs. Baselines run at their own native resolution and length, and re-rendering methods are fed a static video made by repeating the first image. Camera accuracy is measured by recovering poses with a monocular estimator, so it inherits that estimator's errors.

The idea to carry away is that extra views of a character only help when they live in the same coordinate frame as the camera you are asking for. A point cloud from the first frame covers what was seen. An aligned memory of anchors covers what was not. One shared camera encoding binds the two, so as the camera circles, the model knows not just what the character looks like, but which side of it should be showing now.