Data Analyst, data engineers, data specialist

About the role We are hiring a Senior Azure Databricks Data Engineer to design, build and own an enterprise lakehouse on Azure Databricks — from the first workspace to production-grade Bronze, Silver and Gold layers. You have built a Medallion architecture from the ground up, not just worked inside one. You know how to set up the platform so it scales, performs well and stays cost-efficient, with governance, security and data quality built in from day one. Key responsibilities Medallion architecture (Bronze / Silver / Gold) • Design and build the Medallion lakehouse from scratch on Azure Databricks and Delta Lake. • Define layer contracts: raw, append-only ingestion in Bronze; cleansed, conformed and deduplicated data in Silver; business-ready, modelled data (star schemas, data marts, aggregates) in Gold. • Build batch and streaming ingestion with Auto Loader, Structured Streaming, Delta Live Tables (Lakeflow Declarative Pipelines) and Azure Data Factory. • Implement incremental processing, CDC, SCD Type 1/2, MERGE patterns, schema evolution and late-arriving data handling. • Set data quality rules and expectations at each layer, with quarantine and alerting for bad records. Platform architecture and scalability • Architect the Azure Databricks environment: workspaces, dev/test/prod separation, VNet injection, Private Link, ADLS Gen2 storage layout and naming standards. • Design for scale across data volume, number of pipelines, teams and concurrent users. • Define cluster policies, compute types (job, all-purpose, serverless, SQL warehouses) and orchestration with Databricks Workflows. • Build reusable frameworks: metadata-driven ingestion, common libraries, templates and logging. Performance optimisation • Tune Spark and Delta workloads: partitioning, liquid clustering, Z-ORDER, OPTIMIZE/VACUUM, file sizing, caching, AQE and Photon. • Diagnose and fix skew, shuffle, spill and small-file problems using the Spark UI and query profiles. • Set and meet SLAs for pipeline run time and data freshness. Cost optimisation (FinOps) • Right-size clusters, use autoscaling, auto-termination, spot instances and serverless where it lowers cost. • Enforce cost controls through cluster policies, tagging and budgets. • Track DBU and Azure spend with system tables and dashboards, and report savings. Governance, security and standards • Implement Unity Catalog: metastore, catalogs/schemas per layer and domain, external locations, storage credentials. • Set up role-based access, row- and column-level security, data masking and PII handling. • Enable lineage, auditing, data classification and integration with Microsoft Purview. • Manage secrets with Azure Key Vault and identities with Entra ID service principals and managed identities. • Define and document engineering standards: coding guidelines, naming conventions, review process and data contracts. DevOps and delivery • Build CI/CD with Azure DevOps or GitHub Actions and Databricks Asset Bundles. • Provision infrastructure as code with Terraform or Bicep. • Add unit and integration tests, monitoring and alerting for pipelines. Leadership • Act as technical lead: mentor engineers, review code and design, and set best practices. • Work with architects, analysts, data scientists and business stakeholders to turn requirements into data products. Required skills and experience • 7+ years in data engineering, including 3+ years building production workloads on Azure Databricks. • Proven track record of building a Medallion architecture from the ground up — be ready to walk through one you designed end to end. • Expert in PySpark and Spark SQL; strong Python and SQL. • Deep knowledge of Delta Lake: ACID transactions, time travel, MERGE, CDF, OPTIMIZE, liquid clustering. • Hands-on with Unity Catalog and data governance: access control, lineage, auditing, PII protection. • Strong Azure stack: ADLS Gen2, Azure Data Factory, Key Vault, Entra ID, Event Hubs, Azure Monitor, networking basics (VNet, Private Endpoints). • Proven results in Spark performance tuning and cloud cost optimisation, with numbers to show for it. • Data modelling: dimensional modelling (Kimball), Data Vault awareness, SCD patterns. • CI/CD and IaC: Git, Azure DevOps or GitHub Actions, Databricks Asset Bundles, Terraform. • Clear communicator who can explain design trade-offs to both engineers and business stakeholders. Nice to have • Certifications: Databricks Certified Data Engineer Professional, Microsoft Certified: Azure Data Engineer Associate (DP-203) or Fabric Data Engineer (DP-700), Databricks Lakehouse Platform Architect. • Experience migrating from on-prem (SQL Server, Hadoop, Informatica, SSIS) or Synapse to Databricks. • Databricks SQL, Lakehouse Monitoring, MLflow or Feature Store exposure. • dbt on Databricks, Great Expectations, or similar data quality tools. • Power BI integration and semantic model design on Gold data. • Experience with data mesh or domain-oriented data products. •