Theme

Data systems

9 repos reviewed · since Oct 10, 2026 · updated Nov 19, 2026 · rewritten each time a repo joins

In one paragraph

Data systems covers software that stores, moves, encodes or produces data. Some projects are databases, such as a Raft-based key-value store and a set of databases that keep their data in object storage. Others stream events between systems, pack bytes for the wire, generate fake documents, publish structured datasets or run geospatial queries on a GPU. The split is by what the data is doing: resting, travelling, being packed, or being made. What counts here: databases, pipelines, encodings and generators that store, move or produce data.

The main approaches

Databases

These projects hold data and answer queries over it. tikv spreads RocksDB instances across nodes, replicates each Region through Raft, and adds ACID transactions modeled after Google Percolator. opendata builds buffer, log, timeseries and vector databases on SlateDB, so the data lives in object storage. global-law keeps each body of law as a git repository with one commit per law taking effect, and answers by in-force date.

Data movement

These projects carry data from where it is produced to where it is used. airbyte is an ELT platform with 600+ connectors from sources to warehouses, lakes and AI applications. kite is a narrower Elixir library that subscribes to the AT Protocol Jetstream V2 websocket stream and persists a cursor so restarts resume. open-us-law ships the scrapers and ingesters that pull US primary law into one 27-column Parquet schema, and publishes quarterly snapshots. mlx-geospatial runs point-in-polygon, spatial joins and haversine distances on Apple silicon GPUs through MLX, and can pass results as MLX arrays to a model.

Encodings

These projects turn structured data into bytes and back. pbf reads and writes protocol buffers in JavaScript, with separate decode and encode classes and a CLI that compiles .proto files into plain JavaScript.

Synthetic data

These projects produce data that looks like real data without carrying it. sample-synthetic-document-generator rewrites a reference PDF as a synthetic HTML copy through Amazon Bedrock, then scans the result for PII with Amazon Comprehend. A second pipeline generates seeded tabular rows offline.

Map of the theme

flowchart LR
  t["Data systems"]
  t --> f1["Databases"]
  t --> f2["Data movement"]
  t --> f3["Encodings"]
  t --> f4["Synthetic data"]
  f1 --> r1["tikv/tikv"]
  f1 --> r2["opendata-oss/opendata"]
  f1 --> r3["lexifina/global-law"]
  f2 --> r4["airbytehq/airbyte"]
  f2 --> r5["Vaquill-AI/open-us-law"]
  f2 --> r6["tabulai/mlx-geospatial"]
  f2 --> r7["joladev/kite"]
  f3 --> r8["mapbox/pbf"]
  f4 --> r9["aws-samples/sample-synthetic-document-generator"]

Where the new ideas are

  • Shared object-store foundation: opendata puts several database types on one storage layer and one set of operational tooling.
  • Connectors as LLM tools: airbyte ships a separate Agent SDK that exposes connectors to pydantic-ai, LangChain, OpenAI Agents and FastMCP.
  • Law as version history: global-law uses git commits and in-force dates so a provision can be read as it stood on a given day, through Python tools and an MCP server.
  • Split of paid and free generation: sample-synthetic-document-generator uses a model for schema extraction, then generates rows offline from a seed.
  • GPU results as model input: mlx-geospatial can return spatial results as MLX arrays.
  • Stream subscriber as an OTP behaviour: kite wraps a websocket stream as an OTP behaviour with compile-time collection filters.

Side by side

RepoApproachLanguageInterfaceStated detail
mapbox/pbfEncodingsJavaScriptLibrary plus CLI that compiles .proto2.5KB gzipped
tikv/tikvDatabasesRustClient drivers in Go, Java, Rust and CACID transactions, scales to 100+ TB
opendata-oss/opendataDatabasesRustFour databases: buffer, log, timeseries, vectorShared service layer is mostly roadmap
airbytehq/airbyteData movementPythonConnector Builder UI, low-code CDK, Airbyte API600+ connectors, self-hosted or Cloud
joladev/kiteData movementElixirOTP behaviour module with optional cursor callbacksAt least once under normal conditions
aws-samples/sample-synthetic-document-generatorSynthetic dataPythonCLI (pocsynth convert)PII audit CSV from Amazon Comprehend
Vaquill-AI/open-us-lawData movementPythonParquet files on Hugging Face plus scrapers5,081,709 sections in 308 files
lexifina/global-lawDatabasesHTMLPython tools, MCP server, hosted MCP connectorOne commit per law taking effect
tabulai/mlx-geospatialData movementPythonNumPy, Shapely and GeoDataFrame inputs, sjoinFloat32 on GPU, polygon overlays not supported

How the idea moved

flowchart LR
  n1["started Jan 2014<br/>mapbox/pbf"]
  n2["started Dec 2015<br/>tikv/tikv"]
  n3["started Jul 2020<br/>airbytehq/airbyte"]
  n4["started Nov 2025<br/>opendata-oss/opendata"]
  n5["started May 2026<br/>aws-samples/sample-synthetic-document-generator"]
  n6["started Jul 2026<br/>Vaquill-AI/open-us-law"]
  n7["started Aug 2026<br/>tabulai/mlx-geospatial"]
  n8["started Sep 2026<br/>joladev/kite"]
  n9["started Oct 2026<br/>lexifina/global-law"]
  n1 --> n2 --> n3 --> n4 --> n5 --> n6 --> n7 --> n8 --> n9
  • started Jan 2014 · pbf · Adds: reads and writes protocol buffers in a 2.5KB gzipped JavaScript library, with separate tree-shakeable decode and encode classes and a CLI that compiles .proto schemas into editable JavaScript.
  • started Dec 2015 · tikv · Adds: one distributed store that serves raw key-value access and ACID transactions with snapshot isolation, sharded and replicated by Raft and a Placement Driver.
  • started Jul 2020 · airbyte · Adds: moves data from APIs, databases and files to warehouses, lakes and AI applications through 600+ connectors, with an Agent SDK that turns connectors into typed LLM tools.
  • started Nov 2025 · opendata · Adds: buffer, log, timeseries and vector databases that share one object-store storage layer, SlateDB.
  • started May 2026 · sample-synthetic-document-generator · Adds: rewrites a reference PDF as a synthetic HTML copy with fake data, audits it for PII, and generates seeded tabular rows offline from an extracted schema.
  • started Jul 2026 · open-us-law · Adds: US primary law in one 27-column Parquet schema with stable IDs, human citations, in-force status and quarterly dated snapshots, plus the scrapers that build it.
  • started Aug 2026 · mlx-geospatial · Adds: point-in-polygon, spatial joins and haversine distances on the Apple silicon GPU through MLX, with results that can feed a model directly.
  • started Sep 2026 · kite · Adds: an OTP-native subscriber for the Jetstream V2 stream, with collection filters set at compile time and pluggable cursor callbacks for resuming after restarts.
  • started Oct 2026 · global-law · Adds: each body of law as a git repository with one commit per law taking effect, queried by in-force date through Python tools and an MCP server.

Easily confused

  • A dataset is not a database. The law corpora publish files and history to read. A database such as TiKV or the OpenData databases serves live reads and writes.
  • A pipeline is not a store. Kite delivers events to a handler and only remembers a cursor so it can resume.
  • An encoding is not a database format. pbf packs messages into bytes. It does not index, query or persist anything.
  • Synthetic data is not anonymised data. The synthetic document generator writes new text and values with a model, then checks the output for PII, rather than masking fields in the original.

Gaps nobody has filled

  • No exactly-once delivery for event streams. The Jetstream subscriber is at least once, and a crash restarts from the last flushed cursor.
  • No measure of how closely synthetic output matches the statistics of its source. The synthetic generator audits for PII only.
  • No polygon overlays or polygon-to-polygon predicates on the GPU.
  • The shared service catalog, admin tooling and distributed state for object-storage databases are roadmap items, not shipped.