Platform ·Part 2 · May 7, 2026

KubeTrader: Data Platform

A visual tour of the running platform: distribution, storage & serving, orchestration, research, and delivery.

natsclickhousedagstergitopsdata-platform

Part 1 covered the why and the ingest path; this part walks the rest of the system, mostly through short clips and diagrams.

3. Distribution — NATS / JetStream

  • Core NATS for messaging on the hot path.
  • Test environments mirror only the data they need via leaf nodes.
  • The data warehouse pulls at its own pace via JetStream.
[distribution] kubetrader · nats / jetstream data path
nats · jetstream · trading · storage
PROD ACCOUNT · VPC-A TEST ACCOUNT · VPC-B async append log selected data Feed handler Features & Signals Trading services Data warehouse CORE NATS cluster low-latency fan-out JetStream durable append log Leaf node mirror test services
NATS + JetStream sustaining ~50K msg/s in (~30K out), ~1 TB/day, billions of messages
NATS + JetStream sustaining ~50K msg/s in (~30K out), ~1 TB/day, billions of messages

4. Storage & Serving

Storage

  • Tiered ClickHouse cluster backed by S3 as the OLAP store.
  • Idempotency via ReplacingMergeTree.
  • Ingest straight off the NATS engine (JetStream pull-consumer configuration).
  • Aurora Serverless v2 as unified storage for internal app state.
  • Migrations via Flyway.
[storage] kubetrader · storage layer
clickhouse · dagster/dbt · aurora · flyway
CLICKHOUSE · ALTINITY OPERATOR MV's Backfills MV Dagster · dbt TTL NATS engine jetstream pull consumers LANDING ZONE ReplacingMergeTree idempotency MARTS Agg / sum merge trees Merge tree TIERED STORAGE EBS / NVMe · HOT DATA S3 BUCKET · COLD STORAGE Flyway · schema migrations SUPPORTING STORES Aurora Serverless v2 · PostgreSQL grafana · dagster · service state S3 buckets · users-experiments · Loki logs · Dagster logs

Serving

  • ClickHouse client in Grafana (Business Charts plugin).

A couple of examples of the serving surface in use:

  • Basis analytics dashboards
  • Order drill-down
Pick a clip

5. Orchestration — Dagster

  • Backfill operations.
  • SCD2 data collection & insertion.
  • Triggers dbt data transformations.
  • Visual lineage.
  • Backtest & reporting.

The example clips walk a Dagster tour, then backfill jobs that trigger spot-node provisioning and schedule onto it.

Dagster1 / 2
Dagster tour
Dagster tour
Backfill jobs triggering spot-node provisioning and scheduling onto it
Backfill jobs triggering spot-node provisioning and scheduling onto it

6. Research — On-Demand Compute

On-demand provisioned compute environments (S / M / L), launched via Jupyter Enterprise Gateway, running as remote kernels on Kubernetes, with cluster capacity dynamically scaled by Karpenter, and controlled access to ClickHouse and AWS services (S3, Bedrock).

Jupyter Enterprise Gateway · on-demand kernels on Karpenter-scaled nodes
Jupyter Enterprise Gateway · on-demand kernels on Karpenter-scaled nodes

7. Delivery — GitOps CI/CD

Two main delivery paths:

  • Internal-built services — continuous delivery without babysitting tags, via Argo Image Updater.
  • External services — manifests picked up directly by Argo CD.
[delivery] kubetrader · delivery layer
ci path · cd path · oidc · aws · eks
GITHUB · CI AWS EKS build required build & push image OIDC poll registry · new tag write tag → Git watch Git · detect drift sync apply manifests only 1 2 3 4 5 6 7 USER commit · PR GitHub Actions workflow trigger CI JOB · GITHUB-HOSTED RUNNERS test build tag push ECR container registry Git · manifests Kubernetes resources GitOps source of truth Argo Image Updater polls registry Argo CD GitOps controller RENDER Kustomize Helm CLUSTER WORKLOADS deployments · services · pods
Argo CD: the platform reconciled to Git — applications synced and healthy
Argo CD: the platform reconciled to Git — applications synced and healthy