Model hosting and compute

In one paragraph

Running models yourself means choosing where the compute comes from and how the model gets called. Some tools schedule GPU jobs across clouds, Kubernetes and Slurm. Others manage a model server on one machine. Some ship a model or a finished app that runs on your own hardware. Others are notebooks that call hosted models from your cloud account. The split is mostly about what you take on: capacity, a serving process, or just API calls.

What counts here: tools and samples for running models on your own hardware or in your own cloud account.

The main approaches

Compute schedulers

You describe a job once and the tool finds the machines. skypilot takes a YAML task spec (GPU type and count, setup commands, run command). sky launch then finds capacity across 20+ clouds, Kubernetes and Slurm, provisions it, runs the job and streams logs back. It adds automatic failover, and on Kubernetes it adds gang scheduling and multi-node jobs. It can also serve models on clusters you already own.

Model serving

These tools keep a model server running and give clients one place to call it. vllmops is a CLI and TUI control plane for bare-metal vLLM. Each model is a YAML file, and a LiteLLM gateway is regenerated on every start or stop so clients see one OpenAI-compatible URL. The cost: a gateway respawn drops requests in flight.

Local AI apps

The model or app runs on a machine you control, and you use it directly. VoiceStudio is a desktop app for voice cloning, dubbing, dictation, transcription and audiobooks. It runs on Apple Silicon Macs, Windows and Linux, with CUDA, Metal or CPU. moondream is a small vision language model in 2B and 0.5B sizes that answers questions about images, captions them and detects objects. video-diffusion-avatars is research code built on WAN2.2-Animate-14B. It animates a portrait photo from a driving video along a chosen camera path.

Cloud model samples

These are notebooks and demos for models hosted in your own cloud account. anthropic-on-aws covers Claude on AWS: prompt engineering, tool use, a PDF knowledge base with citations and a multimodal Streamlit playground. sample-moonshotai-on-aws has one Kimi K3 notebook for Amazon Bedrock, plus links to AWS Marketplace and to SageMaker deployment examples.

Map of the theme

flowchart LR
  t["Model hosting and compute"]
  t --> f1["Compute schedulers"]
  t --> f2["Model serving"]
  t --> f3["Local AI apps"]
  t --> f4["Cloud model samples"]
  f1 --> r1["skypilot-org/skypilot"]
  f2 --> r2["Freim32/vllmops"]
  f3 --> r3["m87-labs/moondream"]
  f3 --> r4["debpalash/VoiceStudio"]
  f3 --> r5["SimonGiebenhain/video-diffusion-avatars"]
  f4 --> r6["aws-samples/anthropic-on-aws"]
  f4 --> r7["aws-samples/sample-moonshotai-on-aws"]

Where the new ideas are

Failover and bin-packing across many backends: SkyPilot treats clouds, Kubernetes and Slurm as one pool.
Git-reviewable serving config: vllmops keeps each model and its profiles in YAML, and drives them from a CLI or a live TUI.
Small models for small hardware: Moondream ships a 0.5B variant next to its 2B model for constrained devices.
Many voice jobs in one local app: VoiceStudio puts a GUI, a local API and MCP support around several voice engines.
Streaming video from a distilled student: video-diffusion-avatars uses a 4-step model on a KV cache so videos can be any length.

Side by side

Repo	Approach	Runs on	What it runs	Entry point
skypilot-org/skypilot	Compute schedulers	20+ clouds, Kubernetes, Slurm	Training and serving jobs (vLLM, SGLang, Ollama and others)	YAML task spec and `sky launch`
Freim32/vllmops	Model serving	Bare-metal vLLM	vLLM models behind a LiteLLM gateway	CLI and TUI
debpalash/VoiceStudio	Local AI apps	Apple Silicon Mac, Windows, Linux (CUDA, Metal, CPU)	Voice cloning, dubbing, transcription	Desktop GUI, local API, MCP
m87-labs/moondream	Local AI apps	A wide range of devices; Modal listed as a cloud runtime	2B and 0.5B vision language model	Quickstart on moondream.ai docs
SimonGiebenhain/video-diffusion-avatars	Local AI apps	Linux with an NVIDIA GPU (48 GB or 80 GB)	WAN2.2-Animate-14B avatars	`generate.py` and `scripts/run_example.sh`
aws-samples/anthropic-on-aws	Cloud model samples	AWS	Anthropic Claude	Notebooks and runnable demos
aws-samples/sample-moonshotai-on-aws	Cloud model samples	Amazon Bedrock, AWS Marketplace, SageMaker (links)	Kimi K3	One Bedrock notebook

How the idea moved

flowchart LR
  n1["started Aug 2021<br/>skypilot-org/skypilot"]
  n2["started Dec 2023<br/>m87-labs/moondream"]
  n3["started May 2024<br/>aws-samples/anthropic-on-aws"]
  n4["started Apr 2026<br/>debpalash/VoiceStudio"]
  n5["started Jun 2026<br/>Freim32/vllmops"]
  n6["started Aug 2026<br/>aws-samples/sample-moonshotai-on-aws"]
  n7["started Oct 2026<br/>SimonGiebenhain/video-diffusion-avatars"]
  n1 --> n2 --> n3 --> n4 --> n5 --> n6 --> n7

started Aug 2021 · skypilot · Adds: runs training and serving jobs across 20+ clouds, Kubernetes and Slurm from one YAML spec, with automatic failover and bin-packing.
started Dec 2023 · moondream · Adds: answers questions about images, captions them and detects objects, with a 0.5B variant next to the 2B model for constrained hardware.
started May 2024 · anthropic-on-aws · Adds: collects AWS notebooks and demos for Claude, covering prompt engineering, tool use, PDF knowledge bases with citations and a multimodal Streamlit playground.
started Apr 2026 · VoiceStudio · Adds: bundles voice cloning, voice design, video dubbing, dictation, transcription and audiobooks into one local desktop app with a GUI, local API and MCP support.
started Jun 2026 · vllmops · Adds: manages bare-metal vLLM from Git-friendly YAML model files, with a CLI, a live TUI and a LiteLLM gateway rebuilt on each start or stop.
started Aug 2026 · sample-moonshotai-on-aws · Adds: gives one Kimi K3 notebook for Amazon Bedrock and links to the AWS Marketplace listing and SageMaker deployment examples.
started Oct 2026 · video-diffusion-avatars · Adds: expression and camera control to a video diffusion model through a small adapter, with a distilled 4-step student that streams videos of any length.

Easily confused

Not a model hub or a model API. The schedulers, servers and local apps run a model on hardware or an account you control. The cloud samples are the closest to API use. Their notebooks and demos teach calling patterns for Claude and Kimi on AWS, and the Kimi repo only links out to SageMaker self-hosting examples.
Schedulers are not servers. SkyPilot decides where a job runs and provisions it. A serving tool such as vllmops manages the model process on machines you already have, and puts one URL in front of them.
A model is not an app. Moondream is a model you call. VoiceStudio is a finished desktop app. video-diffusion-avatars is research code you run yourself.
Not training or fine-tuning in general. SkyPilot can launch training jobs, but the theme is about where models run.

Gaps nobody has filled

No comparison of speed, memory or quality for the same model across hosting options.
No model gateway that regenerates its config without dropping requests in flight.
No deployment steps for small vision models on specific constrained devices.
No packaged app with a GUI or API around research video models.

RationalStuff v3