01 / Reuse unchanged components
Reuse the platform.
Deploy the change.
Build experimental data products on the existing AWS infrastructure. Run separate versions of the services a feature changes, and connect them to unchanged shared dependencies.
Existing cluster, networking, login, and compatible storage and build services.
Branch versions of the compiler, control plane, runtime, or deployment controller.
Fresh products, scoped credentials, private state, and distinct build outputs.
Edit code and run local scripts on the Mac. Use the external drive for heavy builds and AWS for the running experiment.
Share the service, separate the data
Reuse Kafka brokers with experiment topics and access rules. Each synthetic product gets its own Postgres instance through the existing deployment controller.
Read existing dependencies. Publish experiment images and packages to separate destinations.
Duplicate only what changes
A runtime change may need only a new product build. Changes to compiler services or the controller get their own instances, connected to unchanged dependencies.
Use a separate dependency instance when an experiment needs to modify or restart that service itself.
Keep the shared platform stable
Route experiment work explicitly and limit its permissions and resource use. Changes to cluster-wide controllers, networking, or resource definitions need a separate cluster boundary.
02 / How development will work
Carry a product
through the change.
Keep a small, populated data product in the sandbox. Change Dataspine, deploy the affected components, and use that product to observe the new behavior.
Work on the language, runtime, platform services, or infrastructure in a local branch.
Deploy private instances of changed services. Build product code from the checkout through the existing pipeline.
Ingest events, query through its generated client, and inspect its logs, storage, and processing behavior.
Publish a new revision over experiment data. Exercise replay and restart experiment processes before resetting it.
What this enables
| Experiment | Use the sandbox to |
|---|---|
| A language feature | Introduce a feature such as compaction, build a product that uses it, and observe its effect on real event processing. Check results across updates, replay, and restarts. |
| A cache redesign | Deploy the changed runtime and storage configuration. Compare cold and warm queries, check invalidation after upgrades, and verify correctness when cached data disappears. |
| Delta schema changes | Upgrade a product with existing records in its own database. Ingest the new schema, query old and new records, and exercise client compatibility and migrations. |
Make platform changes part of the workflow
Choose replacement components per experiment. A cache change can reuse login, Kafka, and builds. A schema migration gets a private database. Restart a shared dependency only after giving the experiment its own instance.
Keep experiments repeatable
Record source and deployed versions, inputs, and expected results. Preserve the product's state while studying an upgrade; reset and reseed it for a different experiment.
03 / How we will deliver it
Start with one experiment.
Choose one feature, identify the components it changes, and deploy the smallest combination that can build and operate a real product.
| Step | Deliverable |
|---|---|
| 1. Map dependencies | A clear list of shared services, changed components, and experiment-owned data. Verify existing capacity, access, and incremental cost. |
| 2. Add experiment routing | Separate service instances where needed, explicit build and deployment destinations, scoped credentials, and private controller state. Shared resources keep their existing owners. |
| 3. Build and use a product | A synthetic product built from the checkout through the real pipeline. Ingest and query through its generated client, then upgrade it while retaining its data. |
| 4. Recover and clean up | Verify replay and experiment-process recovery. Remove only owned workloads, data, and artifacts. Keep unchanged shared services running. |
The first engineering work
Make every build and deployment target the experiment explicitly. A private control-plane database needs a matching build-finalizer configuration. A changed controller needs its own watch namespace, state, and restricted permissions.
Spend on the experiment
Keep all additional spending within €100 total. Include extra service instances, shared-service usage, build workers, storage, and cleanup. Set an end time and limit concurrent builds.
Check capacity before sharing it. Use separate worker capacity when needed, and give retained experiment data a deletion date. Shared infrastructure still has usage and performance costs.
Definition of ready
A developer can deploy changed code, build a real product, inspect its behavior, and repeat an upgrade without replacing unrelated services.
Expand when necessary
Add a private database server for disruptive database work, or a separate cluster for cluster-wide changes. Keep all other compatible dependencies shared.
Source material remains in the private repository.