Open roleExternal

Databricks Platform Engineer

Job Description

Design and implement scalable Databricks Lakehouse platforms on AWS and/or Azure aligned to client requirements
Architect end-to-end data platforms including ingestion, storage (Delta Lake), processing, and consumption layers
Build and configure cloud infrastructure using infrastructure-as-code (e.g. Terraform & Declarative Automation Bundles (DAB's))
Establish secure, compliant environments including networking (VNet/VPC, Private Link), identity (IAM/Entra ID), data governance (Unity Catalog), and access controls
Define environment strategies (dev/test/prod), CI/CD pipelines, and release processes for Databricks deployments
Implement monitoring, logging, cost optimisation, and performance tuning across the platform
Design and implement data pipelines using Delta Live Tables, Auto Loader, and Databricks Workflows for both batch and streaming workloads

Work directly with clients to translate business and technical requirements into scalable platform designs
Lead technical workshops, architecture sessions, and whiteboarding engagements with client stakeholders
Support rapid prototyping and proof‑of‑concept builds within Databricks to demonstrate platform capabilities and accelerate client adoption
Provide best practice guidance on Lakehouse architecture, data modelling, workload optimisation, and cost management
Produce high‑quality technical documentation including architecture diagrams, architecture decision records (ADRs), runbooks, and deployment guides
Enable client teams through structured knowledge transfer, training, and platform handover
Collaborate with data engineers, data scientists, and product teams to ensure successful delivery outcomes

Implement Unity Catalog for centralised data governance, including access control (RBAC/ABAC), data lineage, audit logging, and compliance enforcement
Apply security best practices across platform design: network isolation, secret management, encryption at rest and in transit, and identity federation
Ensure platform designs meet client regulatory and compliance requirements (e.g. GDPR, ISO 27001, sector‑specific standards)

Delivery of robust, secure, and scalable Databricks platforms that meet client performance and cost expectations
Clear, well‑architected solutions that balance flexibility, governance, and operational efficiency
Strong client relationships built on trust, technical credibility, and effective communication
Accelerated client adoption of the Lakehouse platform through well‑designed enablement and documentation
Reduced deployment time through reusable infrastructure patterns and automation
Proactive identification of risks, trade‑offs, and optimisation opportunities across platform design and delivery
Contribution to the organisation's growing body of reusable platform accelerators, reference architectures, and internal knowledge

3+ years experience in data platform engineering, cloud engineering, or similar roles
Strong hands‑on experience with Databricks, including Apache Spark, Delta Lake, Workflows
Proven experience designing and deploying data platforms on AWS and/or Azure (e.g. ADLS, S3, VNet/VPC, IAM)
Experience with infrastructure-as-code tools (e.g. Terraform preferred) and CI/CD pipelines (e.g. Azure DevOps, GitHub Actions)
Solid understanding of data architecture concepts including Lakehouse medallion architecture and dimensional modelling
Familiarity with security and governance frameworks (e.g. RBAC, ABAC, data masking, audit, compliance standards)
Excellent communication skills with the ability to explain complex technical concepts to non‑technical stakeholders
Comfortable working in a client‑facing consultancy environment with multiple concurrent engagements
Proactive, self‑driven, and able to take ownership of end‑to‑end platform delivery
Willingness to travel within the UK as required
Right to work in the UK

#J-18808-Ljbffr