Data systems

joladev/

kite

Elixir library for subscribing to AT Protocol's Jetstream V2 websocket stream with cursor persistence and failover.

What’s new here

Kite is a purpose-built AT Protocol Jetstream V2 consumer for Elixir. You get a use Kite behaviour that handles reconnection, backpressure, cursor flushing, and multi-endpoint failover as OTP concerns, so your code only needs to implement the event handler and optionally the cursor callbacks.

What it does

Kite connects to one or more Jetstream websocket endpoints and delivers a live stream of AT Protocol events (creates, updates, deletes) to a callback you implement. You filter by collection NSID (e.g. app.bsky.feed.post) and event kind at startup.

Two optional callbacks, get_cursor/0 and put_cursor/1, let you persist your position so restarts resume from the last known offset. Without them the subscriber starts from the latest available event. Cursors are flushed on a configurable interval (default 5 seconds).

Delivery is best effort and at least once under normal conditions. Failover rewinds the cursor, and a crash or a raising handler restarts from the last flushed cursor, so events can be replayed and handlers must tolerate duplicates. If the handler falls too far behind, events can be lost, because Jetstream instances keep only about 1-2 days of data and have had outages.

On the reliability side: websocket ping/pong keepalives, backoff on reconnect, automatic failover across the endpoint list with a configurable rewind window to handle clock skew, and backpressure when the handler falls behind. If you fall too far behind and the upstream rejects the connection, Kite requests the oldest data the server still holds. OTP 28 adds optional zstd compression.

Who it’s for

Elixir developers building applications that consume the AT Protocol firehose: feed aggregators, moderation tools, analytics, or services that need to react to Bluesky or other atproto network activity in near real-time. The README notes it is running in production at shelf.cafe.

Try it

Requires Elixir 1.18 and OTP 25 or higher. For zstd compression, OTP 28 or higher is needed.

Add to mix.exs:

def deps do
  [
    {:kite, "~> 0.3"}
  ]
end

To see it run with no project, save this script to a file and run it with elixir (requires Elixir installed):

Mix.install([:kite])
 
defmodule EverythingEverywhereAllAtOnce do
  use Kite,
    endpoints: ["wss://jetstream.us-west.bsky.network"],
    kinds: [:commit]
 
  def handle_event(payload, _context), do: IO.inspect(payload)
end
 
EverythingEverywhereAllAtOnce.start_link([])
Process.sleep(:timer.seconds(60))

Docs are on hexdocs.pm/kite.

How mature is it

Created September 17, 2026, last pushed September 21, 2026. 13 commits over 90 days, 1 contributor, 10 stars, 0 forks, 0 open issues and pull requests. No formal releases on GitHub; the package is published to Hex at ~> 0.3. MIT license. Early-stage but used in production according to the README.