Data systems

In one paragraph

Data systems covers software that stores, moves, encodes or produces data. Examples range from a distributed key-value store and object-storage databases, to an ELT platform and a websocket stream subscriber, to a protobuf codec, a PDF-to-fake-document generator, a legal corpus and a GPU geospatial library. They differ in what they touch: bytes on disk, bytes on the wire, or the rows themselves. Some are services you run, others libraries you import.

What counts here: databases, pipelines, encodings and generators that store, move or produce data.

The main approaches

Databases

These projects keep data and answer queries over it. They differ in how they lay data out.

tikv splits data into Regions, each replicated across a Raft group on RocksDB, with a Placement Driver handling sharding and migration. It offers raw key-value access and ACID transactions with snapshot isolation. opendata builds buffer, log, timeseries and vector databases on SlateDB, so the data lives in object storage.

Data movement

These projects carry data from where it is produced to where it is used.

airbyte is an ELT platform with 600+ connectors, a Connector Builder UI and a low-code CDK. kite is an Elixir library that subscribes to AT Protocol’s Jetstream V2 stream, filters by collection, and saves a cursor so restarts resume where they stopped.

open-us-law ships the data itself along with the code that builds it. It publishes US primary law as Parquet with a 27-column schema, plus the scrapers and ingestion pipeline behind it.

mlx-geospatial runs point-in-polygon, spatial joins and haversine distances on Apple silicon through MLX. Results can stay as MLX arrays and feed an MLX model directly.

Encodings

These projects define how data is written as bytes. pbf reads and writes protocol buffers in JavaScript at about 2.5KB gzipped, and its CLI compiles .proto schemas into plain JavaScript modules.

Synthetic data

These projects produce data that looks real but is not. sample-synthetic-document-generator rewrites a reference PDF as synthetic HTML through Amazon Bedrock, then scans the result for PII with Amazon Comprehend. A second pipeline generates seeded CSV or JSON rows offline.

Map of the theme

flowchart LR
  t["Data systems"]
  t --> f1["Databases"]
  t --> f2["Data movement"]
  t --> f3["Encodings"]
  t --> f4["Synthetic data"]
  f1 --> r1["tikv/tikv"]
  f1 --> r2["opendata-oss/opendata"]
  f2 --> r3["airbytehq/airbyte"]
  f2 --> r4["Vaquill-AI/open-us-law"]
  f2 --> r5["tabulai/mlx-geospatial"]
  f2 --> r6["joladev/kite"]
  f3 --> r7["mapbox/pbf"]
  f4 --> r8["aws-samples/sample-synthetic-document-generator"]

Where the new ideas are

Shared object-storage foundation: opendata puts several database types on one storage layer, SlateDB, with common tooling planned above it.
Connectors as agent tools: airbyte has a separate Agent SDK that exposes connectors as typed tools for LLM frameworks.
One store, two access modes: tikv offers raw key-value access and Percolator-style transactions together.
Paid step split from free step: sample-synthetic-document-generator extracts a schema with a model once, then generates rows offline with a seed.
GPU results as model input: mlx-geospatial can return results as MLX arrays.
Stream consumer as an OTP behaviour: kite lets you declare filters and cursor callbacks instead of managing websocket state.

Side by side

Repo	Approach	Language	Interface	Key fact
mapbox/pbf	Encodings	JavaScript	Library plus CLI that compiles `.proto`	About 2.5KB gzipped
tikv/tikv	Databases	Rust	Client drivers in Go, Java, Rust and C	Scales to 100+ TB, ACID transactions
airbytehq/airbyte	Data movement	Python	Self-hosted or Cloud, Connector Builder UI, low-code CDK	600+ connectors
opendata-oss/opendata	Databases	Rust	Buffer, log, timeseries and vector databases	Most of the shared layer is roadmap
aws-samples/sample-synthetic-document-generator	Synthetic data	Python	CLI (`pocsynth convert`)	PII audit CSV from Amazon Comprehend
Vaquill-AI/open-us-law	Data movement	Python	Parquet files plus scrapers	5,081,709 sections in snapshot v2026.09.1
tabulai/mlx-geospatial	Data movement	Python	NumPy, Shapely or GeoDataFrame inputs	No polygon overlays
joladev/kite	Data movement	Elixir	OTP behaviour module	Best effort, at least once delivery

How the idea moved

flowchart LR
  n1["started Jan 2014<br/>mapbox/pbf"]
  n2["started Dec 2015<br/>tikv/tikv"]
  n3["started Jul 2020<br/>airbytehq/airbyte"]
  n4["started Nov 2025<br/>opendata-oss/opendata"]
  n5["started May 2026<br/>aws-samples/sample-synthetic-document-generator"]
  n6["started Jul 2026<br/>Vaquill-AI/open-us-law"]
  n7["started Aug 2026<br/>tabulai/mlx-geospatial"]
  n8["started Sep 2026<br/>joladev/kite"]
  n1 --> n2 --> n3 --> n4 --> n5 --> n6 --> n7 --> n8

started Jan 2014 · pbf · Adds: decodes and encodes protocol buffers in JavaScript with separate tree-shakeable classes, lazy decoding and a CLI that compiles .proto files into editable code.
started Dec 2015 · tikv · Adds: a distributed key-value store on Raft and RocksDB that serves both raw access and ACID transactions, with sharding handled by a Placement Driver.
started Jul 2020 · airbyte · Adds: an ELT platform with 600+ connectors, a no-code Connector Builder and an Agent SDK that turns connectors into typed LLM tools.
started Nov 2025 · opendata · Adds: buffer, log, timeseries and vector databases that share one object-storage layer, SlateDB.
started May 2026 · sample-synthetic-document-generator · Adds: rewrites a reference PDF as synthetic HTML with fake data, audits it for PII, and generates seeded tabular rows offline from an extracted schema.
started Jul 2026 · open-us-law · Adds: US statutes, regulations, court rules and constitutions in one Parquet schema with dated snapshots, plus the scrapers that build it.
started Aug 2026 · mlx-geospatial · Adds: point-in-polygon, spatial joins and haversine distances on the Apple silicon GPU, with results that can be returned as MLX arrays.
started Sep 2026 · kite · Adds: subscribes to AT Protocol’s Jetstream V2 stream as an Elixir behaviour module with collection filters and pluggable cursor persistence.

Easily confused

A database versus a pipeline. A database such as TiKV keeps data and answers reads and writes. A pipeline such as Airbyte or Kite carries data from one place to another and does not own it. OpenData blurs the line a little, because its Log and Buffer databases are meant for data movement but still store what they hold.
Synthetic data versus redaction. The synthetic document generator does not mask a document in place. It writes a new document with the same layout and fake content, then checks that for PII.
A dataset versus a database. open-us-law is a published corpus with the code that builds it. It is files you download, not a server you query.
An encoding versus a transport. pbf defines how a message becomes bytes. It does not send them anywhere.

Gaps nobody has filled

No exactly-once delivery for live event streams. The Jetstream subscriber is at least once and rewinds on failover.
No shipped shared operations layer sits above the object-storage databases. The service catalog and admin tooling are on a roadmap.
No polygon overlays or polygon-to-polygon predicates on the GPU.

RationalStuff v3