Data engineering and DevOps articles
Hands-on articles from the Plainsight team on data engineering and DevOps: CI/CD for data and BI, database projects and automated deployment, data architecture that holds up as you grow, and the step from DevOps to MLOps and LLMOps.
Start here
Data EngineeringCI/CD for dbt on Microsoft Fabric
How we set up CI/CD for dbt on Microsoft Fabric: one workspace per environment, isolated schemas, slim CI with deferral, quality gates and scheduled runs.
DevOpsStreamline Your SQL Deployment Using Azure DevOps and Database Projects
First things first Before lift-off, make sure the following prerequisites are met: - The previous two blog posts in this series were read and the steps were…
DevOpsOps I Did It Again: From DevOps to MLOps to LLMOps
You might have heard of DevOps, MLOps or even LLMOps, but what are they really and how do they relate to one another? Each emerged to solve a distinct set…
More on Data engineering and DevOps
DevOpsFrom Technician Notes to PDF: Automating Incident Reporting
A smarter way to automatically turn field technician input into clean, branded PDF reports: from ticket data to professional documents in seconds.
Power BIEfficiently Managing Version History and Deployment in Power BI
How to choose a version history and deployment setup for Power BI that fits your team: PBIP and TMDL in Git, deployment pipelines, quality gates and shared models.
Data EngineeringImplementing Data Mesh with dbt: From Principles to Practice
Most traditional organizations rely on a centralized data team to own, build, and maintain everything data related. This team supports every business domain…
How to Simplify Deployment Pipelines: One Parameter to Rule Them All
Replace a parameter per data source with one environment parameter and a small Power Query function, so a Power BI deployment pipeline needs one rule per stage.
Microsoft FabricSQL Databases in Fabric: Because Life's Too Short for Database Configuration Nightmares
At the end of 2024, databases were introduced to Microsoft Fabric, marking a significant milestone in the evolution of Fabric as an end-to-end data…
Microsoft FabricAutomating Data Pipelines with Microsoft Fabric for Faster Insights
How to automate ETL with Data Pipelines in Microsoft Fabric: ingestion, transformation, storage, and orchestration, all in a low/no-code environment.
How to Deploy and Configure 12 Microsoft Fabric Capacities in a matter of hours
THE CHALLENGE - Deploy 12 Fabric Capacities - Across 4 business Units - With a dev, tst & prd capacity - With a Bronze, Silver, Gold & Control workspace for…
Data EngineeringStruggling with Slow Data Product Delivery? Here's Where to Automate
In today's data-driven economy, the speed at which you deliver data products directly affects your organization's ability to make timely decisions.
Data EngineeringHow to Analyse Data Across Multiple Regions with Confidence and Consistency
Multi-region analytics breaks on formats, latency and local data laws. How Azure, Microsoft Fabric and Power BI get you one trusted global view.
Data EngineeringHow to Improve Collaboration Between Your Data Tools: From Silos to Synergy
DBT builds the logic, Fabric runs it, Purview tracks it. How to connect transformation, governance, and reporting so every tool shows the same truth.
Data EngineeringHow to Build a Scalable Data Architecture That Supports Long-Term Growth
Legacy pipelines break every time the business changes. What scalable architecture looks like on Azure, Fabric, and OneLake, and how Plainsight builds it.
Data EngineeringHow Do You Use Azure Synapse and Databricks for Seamless Data Integration?
Synapse for warehousing and SQL analytics, Databricks for Spark and AI. How the two fit together and what to watch out for on cost, security, and performance.
Microsoft FabricHow Microsoft Fabric Simplifies Data Integration Across Your Organisation
OneLake, pipelines, Dataflow Gen2, Notebooks, Eventstream: how Fabric puts ingestion, storage, and reporting on one platform instead of five tools.
Data EngineeringTriggering and Monitoring a dbt Cloud Job with Azure Data Factory
Ready to master your data pipelines? Dive into our latest blog to discover how Azure Data Factory can trigger and monitor dbt Cloud jobs. Learn how to…
DevOpsOne-Click Workspaces for Power BI CI/CD: Is It Possible?
How we deploy one Power BI report to many workspaces across tenants: service principals, PowerShell and the Power BI APIs in an almost one-click setup.
Data EngineeringStreamline Data Operations: Integrate Database Projects with Data Factory
First things first In this blog post we are going to show a practical use case of how you should use database projects for SQL together with Azure Data…
DevOpsStreamline Your SQL Development with Database Projects and Git
Turn a SQL database into a database project you can track in Git. A hands-on walkthrough in Azure Data Studio, from bacpac import to your first push.
Microsoft FabricWhat's in Your Fabric: Data Warehouse or Data Engineering?
we get a lot of interesting questions on Microsoft Fabric. One of the recurring topics is which ETL engine to choose? Can we use multiple? And when to use…
Data Engineering1st things 1st, Data Lake Planning!
In today's world, data is king. Businesses are generating more data than ever before, and they need a way to store, manage, and analyse it all.