In one paragraph
Running a model yourself means choosing where it runs and what runs it. Some projects schedule jobs across clouds, Kubernetes and Slurm. Some are serving layers that put an OpenAI-compatible endpoint on your own GPUs or Trainium boxes. Some are apps or model releases that run on a laptop or a single GPU. Some are notebooks that show how to call or deploy a model in your own AWS account. They differ in how much of the stack they take over.
What counts here: tools and samples for running models on your own hardware or in your own cloud account.
The main approaches
Compute schedulers
These tools decide where a job runs. You describe what you need, and the tool finds capacity, starts the machines and runs the job. skypilot takes a YAML task spec (GPU type and count, setup, run command) and sky launch provisions VMs or pods, syncs your working directory and streams logs. It covers 20+ clouds, Kubernetes and Slurm, with failover and bin-packing. On Kubernetes it adds gang scheduling and multi-node jobs. It also serves models on clusters you own.
Model serving
These projects run the inference server itself. vllmops manages bare-metal vLLM from one YAML file per model, with a CLI and a TUI. It regenerates a LiteLLM gateway on each start or stop, so clients see one OpenAI-compatible URL. kiln is an OpenAI-compatible engine for AWS Trainium. It has its own scheduler, paged KV cache, radix prefix cache and NKI kernels, and it compiles graphs ahead of time on CPU hosts.
Local AI apps
These run a model for a specific task on hardware you control. VoiceStudio is a desktop app for voice cloning, dubbing, dictation and transcription, with a local API and MCP support. vui is a voice assistant server built on its own small TTS model, with a pure-C CPU build. moondream is a small vision language model (2B and 0.5B) that answers questions about images. video-diffusion-avatars is research code that animates a portrait from a driving video and a camera path.
Cloud model samples
These are example code for using or deploying a model in your own cloud account. anthropic-on-aws has notebooks and demo apps for Claude on AWS, including tool use, a PDF knowledge base with citations and a Streamlit playground. sample-moonshotai-on-aws has one Kimi K3 notebook for Bedrock and links to Marketplace and SageMaker deployment examples.
Map of the theme
flowchart LR t["Model hosting and compute"] t --> f1["Compute schedulers"] t --> f2["Model serving"] t --> f3["Local AI apps"] t --> f4["Cloud model samples"] f1 --> r1["skypilot-org/skypilot"] f2 --> r2["Freim32/vllmops"] f2 --> r3["foxl-ai/kiln"] f3 --> r4["m87-labs/moondream"] f3 --> r5["fluxions-ai/vui"] f3 --> r6["debpalash/VoiceStudio"] f3 --> r7["SimonGiebenhain/video-diffusion-avatars"] f4 --> r8["aws-samples/anthropic-on-aws"] f4 --> r9["aws-samples/sample-moonshotai-on-aws"]
Where the new ideas are
- Scheduling: skypilot puts clouds, Kubernetes and Slurm behind one spec, and offers a Skill so coding agents can drive it.
- Serving on non-GPU silicon: kiln rebuilds vLLM and SGLang style techniques for ahead-of-time compiled graphs on Trainium. It can also split prefill and decode into separate engines.
- Config as code: vllmops keeps model and profile settings in Git-reviewable YAML.
- Context-aware speech: vui decodes each reply from a KV cache that holds the whole dialogue, including your audio.
- Streaming video: video-diffusion-avatars has a distilled 4-step student that streams 16-frame blocks, so video length is not capped.
Side by side
| Repo | Approach | Runs on | What you give it | Language |
|---|---|---|---|---|
| skypilot-org/skypilot | Compute schedulers | 20+ clouds, Kubernetes, Slurm | YAML task spec | Python |
| Freim32/vllmops | Model serving | Bare-metal vLLM | YAML per model, CLI and TUI | Python |
| foxl-ai/kiln | Model serving | AWS Trainium (trn1, trn2) | OpenAI-compatible requests | Python |
| debpalash/VoiceStudio | Local AI apps | Apple Silicon Mac, Windows, Linux (CUDA, Metal or CPU) | Desktop app, local API, MCP | Python |
| fluxions-ai/vui | Local AI apps | Streaming server on Linux with an NVIDIA GPU; the TTS model also has a C build for CPU | Browser mic over WebRTC | Python |
| m87-labs/moondream | Local AI apps | Wide range of devices; Modal example linked | An image and a question | Python |
| SimonGiebenhain/video-diffusion-avatars | Local AI apps | Linux with a 48 GB or 80 GB NVIDIA GPU | Portrait photo, driver video, camera path | Python |
| aws-samples/anthropic-on-aws | Cloud model samples | AWS | Notebooks and demo apps | Jupyter Notebook |
| aws-samples/sample-moonshotai-on-aws | Cloud model samples | Amazon Bedrock, with links to SageMaker | One Bedrock notebook plus links | Jupyter Notebook |
How the idea moved
flowchart LR n1["started Aug 2021<br/>skypilot-org/skypilot"] n2["started Dec 2023<br/>m87-labs/moondream"] n3["started May 2024<br/>aws-samples/anthropic-on-aws"] n4["started Jun 2025<br/>fluxions-ai/vui"] n5["started Apr 2026<br/>debpalash/VoiceStudio"] n6["started Jun 2026<br/>Freim32/vllmops"] n7["started Aug 2026<br/>aws-samples/sample-moonshotai-on-aws"] n8["started Oct 2026<br/>foxl-ai/kiln"] n9["started Oct 2026<br/>SimonGiebenhain/video-diffusion-avatars"] n1 --> n2 --> n3 --> n4 --> n5 --> n6 --> n7 --> n8 --> n9
- started Aug 2021 · skypilot · Adds: runs training and serving jobs on 20+ clouds, Kubernetes and Slurm from one YAML spec, with failover and bin-packing.
- started Dec 2023 · moondream · Adds: answers questions about images, captions them and detects objects with a 2B model and a 0.5B variant for constrained hardware.
- started May 2024 · anthropic-on-aws · Adds: shows Claude on AWS through notebooks and runnable demos for tool use, PDF knowledge bases with citations, classification and a multimodal playground.
- started Jun 2025 · vui · Adds: runs a real-time voice assistant on its own small TTS model, which decodes replies from a KV cache of the whole dialogue, with a pure-C CPU build.
- started Apr 2026 · VoiceStudio · Adds: bundles voice cloning, voice design, video dubbing, dictation, transcription and audiobook creation in one local desktop app with a local API and MCP support.
- started Jun 2026 · vllmops · Adds: manages bare-metal vLLM from Git-reviewable YAML through a CLI and a TUI, and keeps one OpenAI-compatible LiteLLM gateway in sync with the running models.
- started Aug 2026 · sample-moonshotai-on-aws · Adds: gives a Kimi K3 notebook for Bedrock and links to Marketplace and SageMaker deployment examples.
- started Oct 2026 · kiln · Adds: serves large open models on Trainium with its own scheduler, paged KV cache, radix prefix cache and NKI kernels, from graphs compiled ahead of time on CPU hosts.
- started Oct 2026 · video-diffusion-avatars · Adds: animates a portrait from a driving video and a chosen camera path through a small adapter on a video diffusion model, with a distilled 4-step streaming student.
Easily confused
- A hosted model API. The schedulers, serving engines and local apps run models on hardware or in a cloud account you control. The AWS sample collections show how to call managed models on AWS, such as Kimi on Bedrock, and the Kimi collection also links to a Marketplace listing for endpoints run by Moonshot AI. Those are guides to a service, not tools that host the model.
- Model training or fine-tuning tooling. A scheduler like SkyPilot can launch training jobs, but the theme is about where and how a model runs, not how it is trained.
- One layer of the stack. A scheduler places jobs, a serving engine answers requests, and an app wraps a model for a task. A serving tool like vllmops does not pick your cloud, and a scheduler does not replace an inference engine.
Gaps nobody has filled
- The cloud sample collections cover AWS only. Examples for deploying the same models in other cloud accounts are missing.
- The voice, vision and avatar projects give no path for placing their workloads on a shared GPU pool.