The problem
The client ran Azure Databricks at enterprise scale across four environments. Every change started as a ticketed request. Every environment lived on its own long-running git branch, so drift between dev, test, stage, and prod was constant. CI authentication depended on static service principal credentials, which made rotation and auditability harder than they needed to be. Onboarding new engineers took days because the model for “what gets deployed where” lived in a spreadsheet, not the repo.
Data engineers feeling the same pain from the other side: a new medallion schema meant a ticket, a wait, and a pile of clicks in the Databricks UI executed by someone else. New data products were paced by platform availability, not data-engineering throughput.
The approach
I rebuilt the repo around a single-branch, directory-per-environment layout. Each
environment owns its own state and variable files. PRs target main. The diff is the
change.
Unity Catalog moved into Terraform end-to-end: catalogs, medallion-tier schemas (bronze, silver, gold, copper), external locations on ADLS Gen2, cluster policies, and per-schema Entra ID group grants. Long-lived service principal credentials in CI got replaced with Azure OIDC federated credentials in GitHub Actions, which removed the rotation problem entirely.
The network got rebuilt at the same time: per-workspace VNets, NSGs, NAT gateway with static egress IPs, and managed-identity-only access to Unity Catalog storage.
Identity through Entra ID
Identity for the platform runs end to end through Microsoft Entra ID. Azure Databricks workspace SSO is wired into the same Entra tenant the rest of the organization uses, so sign-in to Databricks goes through the same Conditional Access policies, including MFA, device compliance, and named-location guards, that gate every other Microsoft 365 workload.
Authorization moves the same direction. Unity Catalog grants are issued to Entra security groups, never named users. Cluster policies, workspace permissions, and per-schema access all resolve through those groups. The Databricks-side picture mirrors the Entra-side picture by construction.
Provisioning is automated. Sailpoint pushes group membership into Entra via SCIM; the Databricks account-level SCIM connector mirrors users and groups from Entra into the workspace. When a hire joins their business unit’s Entra group, the next morning they have Databricks access across workspace, catalog, and schema without anyone touching the Databricks UI. When someone leaves, the same path revokes it.
Self-service schemas for data engineers
A new medallion schema is now a pull request. The platform team owns a Terraform module
that wraps the Unity Catalog catalog binding, the schema itself, its bronze / silver /
gold / copper layers, the ADLS Gen2 external location, the Entra ID group grants, and
the cluster-policy bindings into one call. A data engineer adds a schemas.tf entry,
opens a PR, gets the plan rendered into the review, and merges. The schema lands across
all four environments without anyone outside the team being involved.
The result is that new data products move at the speed of code review rather than the speed of ticket queues. Platform engineers stop being a bottleneck for routine provisioning and spend their time where it matters: module design, drift control, and cost guardrails, instead of clicking through the Databricks UI on someone else’s behalf.
Change control and supply chain
Change control runs through PRs now. Plan output renders into the PR as a comment so reviewers can see the actual diff. Each environment has its own CODEOWNERS. Applies are concurrency-locked so two PRs can’t fight. Plan artifacts are kept for 90 days.
Supply chain got tightened in passing. Every GitHub Action is SHA-pinned. Dependabot is configured with grouped provider and action updates on a 7-day cooldown so the team isn’t reviewing a new patch PR every morning.
The outcome
The migration ran with zero workload interruption. Engineers ship Databricks changes through PR review now instead of ticket queues. Data engineers stand up new medallion schemas through Terraform without leaving their normal workflow. There is nothing to rotate on the secrets side. RBAC is finally readable: every grant maps to a group, every group is owned by a team, every change is in git.