Open source tools. You keep what we build.
dbt, Spark, Delta Lake, MLflow, Unity Catalog, Python. The open layer of the stack, running inside the platforms you already pay for. Your logic stays in your repository and your data stays in a format more than one engine can read, so changing your mind later costs a decision instead of a rebuild.
Sound familiar?
The reasons teams come to us asking for the open parts of the stack.
What we actually use
Not a list of everything that exists. These are the tools we have written conventions for and run in production, which is a much shorter list than the one on most consultancy websites.
dbt
Our transformation layer, whichever platform sits underneath. Layered project structure, tests saturated on sources and gold, docs next to the models, dbt build wired into CI. It has its own page because we have that many opinions about it.
Fast track to dbt→Apache Spark
The engine under the lakehouse work on both Databricks and Fabric. PySpark for the data engineering and data science workflows, notebooks for exploration, incremental pipelines for everything that has to run every night.
Delta Lake
The open table format we default to. ACID transactions and time travel are the practical part. The strategic part is that your history sits in Parquet files you own, readable by more than one engine.
MLflow
Experiment tracking and the Model Registry. Every run recorded with its parameters, every promotion to production going through the registry rather than a copy-paste. It is how you answer what changed and when, months later.
Unity Catalog
Databricks open sourced it, which matters: a governance layer you can run enterprise-wide across a multi-cloud or hybrid estate. Permissions, lineage and the catalog in one place, not three.
Python
Python 3.11 is the default for pipelines, models and the glue in between. Boring, on purpose. Everyone we hire knows it and every platform we work on runs it.
SQLFluff and dbt-utils
SQLFluff lints the SQL and the Jinja so reviews argue about logic instead of formatting. dbt-utils covers the macros and tests every project would otherwise rewrite. Pinned versions, small set, evaluated before adoption.
Model Context Protocol
The open standard for connecting AI assistants to real systems. We build MCP servers so an assistant reads your actual data and tools rather than guessing, and so the same integration works with more than one model vendor.
How we decide what goes in.
Open source is a means, not a badge. Four rules that keep it that way.
Open formats first. The exit is part of the design.
Data in Delta and Parquet, transformations in SQL and Python in your Git repository, models in the MLflow registry. If you replace us or replace the platform, the assets come with you. That is not a favour, it is how the thing should be built.
Open source where it wins. Managed where it earns its price.
We are Microsoft and Databricks partners and we say so. The honest position is that open tooling runs inside those platforms, not against them. dbt on Databricks SQL, Spark on Fabric capacity. You pay a vendor for the compute and operations, not for the right to read your own tables.
A small, pinned set beats a big, floating one.
Every dependency is a maintenance promise. We evaluate maintenance activity, adapter support and performance before adopting anything, pin the versions, and keep the list short enough that upgrading is a normal Tuesday rather than a project.
Our own playbook is open too.
The conventions we apply on client work are published on GitHub, not kept as a competitive secret. Read them before you talk to us, lift what is useful, fork the rest. Knowledge sharing across organisations is the point.
Runs inside your platform
Open tooling is not an alternative to Databricks or Fabric. It is the layer we build on top of them, which is what keeps the work portable if the platform decision ever changes.
FAST TRACK →Spark, Delta and Unity Catalog
Most of this list started at Databricks and was open sourced from there. We run it with Unity Catalog for governance and Asset Bundles to ship it.
FAST TRACK →Open formats on OneLake
Fabric stores in Delta on OneLake and runs Spark natively, so the same conventions travel. Native dbt jobs land there too.
Open tooling is how we build, across every domain.
Strategy decides which parts of the stack should be open. We build it that way. Experts keep it running once your team owns it.
Strategy
From business questions to a data, AI and software direction your organisation can follow.
Explore →Build
Data and analytics, AI, and custom software. Built with adoption and ownership from day one.
Explore →Experts
Specialists who think like teammates. Keep your engine running and your team growing.
Explore →Worried about what happens when you want out?
Half-hour call. Tell us your stack and we'll be straight about which parts lock you in and which parts you would actually keep.
Talk to an expert