Open source tools. You keep what we build.
dbt, Spark, Delta Lake, MLflow, Unity Catalog, Python. The open layer of the stack, running inside the platforms you already pay for. Your logic stays in your repository and your data stays in a format more than one engine can read, so changing your mind later costs a decision instead of a rebuild.
BEFORE WE START
Sound familiar?
Why teams come to us asking for the open parts of the stack, before any work starts. Each one has a way out, and the rest of this page is how we get there.
How we approach it
Transformation logic living inside a vendor's proprietary UI, tables in a format only that vendor reads, orchestration defined in a console nobody can export. The exit cost is the real price of the licence, and it never shows up on the quote.
How we approach it
Experiments run in notebooks, parameters remembered rather than recorded, the winning model copied to production by hand. When the regulator or the business asks what changed, the honest answer is that nobody knows.
How we approach it
Years of history locked in a proprietary table format. Every new tool needs an export, every export needs a reconciliation, and the copies drift apart. Open formats make the storage layer a decision you can revisit.
How we approach it
Pipelines that only made sense to one engineer, no tests, no docs, no lineage. Open tooling with real communities means the next hire has probably used it before, and the answer to most questions is already written down somewhere public.
What we actually use
Not a list of everything that exists. These are the tools we have written conventions for and run in production, which is a much shorter list than the one on most consultancy websites.
dbt
Our transformation layer, whichever platform sits underneath. Layered project structure, tests saturated on sources and gold, docs next to the models, dbt build wired into CI. It has its own page because we have that many opinions about it.
Fast track to dbt→Apache Spark
The engine under the lakehouse work on both Databricks and Fabric. PySpark for the data engineering and data science workflows, notebooks for exploration, incremental pipelines for everything that has to run every night.
Delta Lake
The open table format we default to. ACID transactions and time travel are the practical part. The strategic part is that your history sits in Parquet files you own, readable by more than one engine.
MLflow
Experiment tracking and the Model Registry. Every run recorded with its parameters, every promotion to production going through the registry rather than a copy-paste. It is how you answer what changed and when, months later.
Unity Catalog
Databricks open sourced it, which matters: a governance layer you can run enterprise-wide across a multi-cloud or hybrid estate. Permissions, lineage and the catalog in one place, not three.
Python
Python 3.11 is the default for pipelines, models and the glue in between. Boring, on purpose. Everyone we hire knows it and every platform we work on runs it.
SQLFluff and dbt-utils
SQLFluff lints the SQL and the Jinja so reviews argue about logic instead of formatting. dbt-utils covers the macros and tests every project would otherwise rewrite. Pinned versions, small set, evaluated before adoption.
Model Context Protocol
The open standard for connecting AI assistants to real systems. We build MCP servers so an assistant reads your actual data and tools rather than guessing, and so the same integration works with more than one model vendor.
How we decide what goes in.
Open source is a means, not a badge. Four rules that keep it that way.
Open formats first. The exit is part of the design.
Data in Delta and Parquet, transformations in SQL and Python in your Git repository, models in the MLflow registry. If you replace us or replace the platform, the assets come with you. That is not a favour, it is how the thing should be built.
Open source where it wins. Managed where it earns its price.
We are Microsoft and Databricks partners and we say so. The honest position is that open tooling runs inside those platforms, not against them. dbt on Databricks SQL, Spark on Fabric capacity. You pay a vendor for the compute and operations, not for the right to read your own tables.
A small, pinned set beats a big, floating one.
Every dependency is a maintenance promise. We evaluate maintenance activity, adapter support and performance before adopting anything, pin the versions, and keep the list short enough that upgrading is a normal Tuesday rather than a project.
Our own playbook is open too.
The conventions we apply on client work are published on GitHub, not kept as a competitive secret. Read them before you talk to us, lift what is useful, fork the rest. Knowledge sharing across organisations is the point.
Runs inside your platform
Open tooling is not an alternative to Databricks or Fabric. It is the layer we build on top of them, which is what keeps the work portable if the platform decision ever changes.
FAST TRACK →Spark, Delta and Unity Catalog
Most of this list started at Databricks and was open sourced from there. We run it with Unity Catalog for governance and Asset Bundles to ship it.
FAST TRACK →Open formats on OneLake
Fabric stores in Delta on OneLake and runs Spark natively, so the same conventions travel. Native dbt jobs land there too.
Open tooling is how we build, across every domain.
Strategy decides which parts of the stack should be open. We build it that way. Experts keep it running once your team owns it.
Strategy
From business questions to a data, AI and software direction your organisation can follow.
Explore →Build
Data and analytics, AI, and custom software. Built with adoption and ownership from day one.
Explore →Experts
Specialists who think like teammates. Keep your engine running and your team growing.
Explore →Worried about what happens when you want out?
Half-hour call. Tell us your stack and we'll be straight about which parts lock you in and which parts you would actually keep.
Talk to an expert