What’s new here

Kiln takes the serving techniques of vLLM and SGLang to a static-shape accelerator. It drops the managed Neuron inference layer (NxD Inference, vllm-neuron) and keeps only the driver, runtime, compiler and collectives. Because Trainium graphs are compiled ahead of time, a CPU compile farm builds every graph first. The Trainium box then starts serving from cache instead of compiling for hours.

What it does

Kiln is an OpenAI-compatible inference server for large open models on trn1 and trn2. It brings its own scheduler (continuous batching, chunked prefill, overlap scheduling), a paged KV cache with FP8, radix prefix caching and a host-memory tier. It also ships NKI kernels for MoE, linear attention and sparse attention, speculative decoding (MTP heads, EAGLE-3, n-gram), and tool-call and reasoning parsers.

It can also split prefill and decode into separate engines behind a router, and run one long prefill across several boxes with a layer pipeline. Both are opt-in. The README reports a 1,044,480-token prompt taking 233.6 s on one trn1.32xlarge and 68.62 s through four pipeline stages and a decode box over NIXL.

The README is open about the costs. On GLM-5.3-Flash it reports 35-80% lower cost per output token than vLLM on 8 x H200 at spot prices at concurrency 16 to 64, but it loses on Capacity Block prices at concurrency 64 and above, and on latency. A next token takes 131-363 ms on trn1 against 20-43 ms on the GPU. The win is cost per token on spot capacity, for throughput workloads.

Who it’s for

Teams that can get Trainium capacity and want cheap batch throughput from large open models, and who are fine running a compile farm. It is not for anyone who needs low latency or on-demand pricing. The README says plainly that on-demand trn1 is ten times its spot price and loses to SageMaker on-demand.

Try it

It needs the PyTorch inference venv of the AWS Deep Learning AMI for Neuron, SDK 2.32 (Ubuntu 24.04), on trn1 or trn2.

git clone https://github.com/foxl-ai/kiln && cd kiln
source /opt/aws_neuronx_venv_pytorch_inference_vllm_0_24_0_1_1_0/bin/activate
export PYTHONPATH=$PWD
 
# A small model on one NeuronCore
python -m kiln --model Qwen/Qwen3-0.6B --device neuron --port 8000
 
# The first request compiles the model's graphs into the local cache; later ones reuse them
curl -s localhost:8000/v1/completions -H 'content-type: application/json' \
  -d '{"model": "Qwen/Qwen3-0.6B", "prompt": "Trainium is", "max_tokens": 32}'

A container image is also published: docker pull public.ecr.aws/sanghwa/kiln:0.3.0. GLM-5.3-Flash graphs take hours to compile on the device, so compile them first on a CPU host with tools/compile_farm.py. The exact configuration is under “How to resume” in docs/price-performance.md.

How mature is it

Very new and small: 5 stars, 0 forks, 1 contributor, created 2026-10-05 and last pushed 2026-10-09, with 6 commits in the last 90 days. It has 3 releases, the latest v0.3.0 on 2026-10-09, and no open issues or PRs. The license is Apache-2.0. The README carries detailed, commit-referenced measurements and lists what it does not win.