What’s new here

This repo controls video generation with three inputs: a reference image, a driving video for expressions, and a camera trajectory. The model is a small adapter (about 1.7 GB for the full version) on top of the frozen WAN2.2-Animate-14B, so the heavy base files are the public ones.

What it does

You give it a near-frontal portrait, a video of someone’s face, and a camera path. It generates the person in the photo performing the driver’s expressions from the requested viewpoints. Preprocessing fits a FLAME head shape to the photo, tracks the driver’s expression, jaw, gaze and blinks, and renders a proxy head plus a skeleton in the target camera. Those feed the model.

There are two models. The full one runs 20 sampling steps and makes 77-frame clips (3.2 s). The fast one is a distilled 4-step student that streams 16-frame blocks on a KV cache, so videos can be any length. The README reports about 3.5 fps at 288p on an A100. Output sizes are 288×512, 576×1024 (default) or 720×1280.

The README says the training cameras cover ±65° azimuth and ±17° elevation, and quality degrades gradually beyond that. It also notes the fast model can drift in colour on dark backgrounds after about 20 s.

Who it’s for

Graphics and vision researchers who want to reproduce the paper or build on it. It also suits anyone with a 48 GB or 80 GB GPU who wants to try controllable face animation. The code and weights are under a non-commercial license, so it is not for products.

Try it

You need Linux, an NVIDIA driver for CUDA 12.8, conda, ffmpeg and gcc 9-14. A GPU with 48 GB works for the full model, and the fast model at 288p. The fast model at 576p or 720p needs 80 GB. Plan for about 65 GB of disk. You also need a free FLAME account, because the installer downloads FLAME 2020 with your login.

git clone https://github.com/SimonGiebenhain/video-diffusion-avatars.git video-diffusion-avatars
cd video-diffusion-avatars
bash envs/setup_env.sh          # one conda env `vda` (VDA_ENV_NAME=<ENV> to rename it); ~30 min, needs a GPU
conda activate vda
 
python download.py weights      # our weights (~6.5 GB) from Hugging Face into weights/, md5-checked
python download.py base         # the public WAN2.2-Animate-14B files (~37 GB) into weights/base/
 
python download.py examples     # ~220 MB into examples/: the 25 examples + the example driving videos
python generate.py --cond examples/teaser_swimmer --out outputs/teaser_swimmer.mp4

The 25 examples ship with precomputed conditioning, so they run without any preprocessing. For your own photo, bash scripts/run_example.sh <PHOTO.jpg> <NAME> runs every step.

How mature is it

A new repo: created 2026-10-16, last pushed 2026-10-20, with 59 stars, 6 forks and no open issues or pull requests. There is one contributor and 3 commits in the last 90 days. No releases are tagged. Weights are on Hugging Face and the code ties to an arXiv paper. GitHub lists the license as Other. The README says the code and weights are CC BY-NC-SA 4.0, and several third-party parts carry their own, partly non-commercial, licenses.