Data Engineer - Tech Lead (Databricks, Pyspark)

We're looking for a Senior Data Engineer – Tech Lead (Databricks, PySpark) to join our team in London, UK, in a hybrid working mode.In this role, you will lead the design, development and optimization of scalable cloud-native data architectures, focusing on Azure Databricks, PySpark and Lakehouse principles. You will work hands-on to deliver performant data solutions for high-volume workloads, ensuring governance, reliability and best practices for enterprise-grade platforms.As a technical leader, you will define data strategies, drive modernization initiatives and mentor engineers, fostering excellence and innovation throughout the team. This position offers the opportunity to shape large-scale data ecosystems, implement modern engineering practices and enable next-generation analytics and AI-driven solutions.ResponsibilitiesLead the architecture, design and build of large-scale data platforms using Azure Databricks and modern cloud technologiesImplement and optimize ETL workflows and streaming pipelines with PySpark and Delta Live Tables following Lakehouse principlesEnhance performance, manage cloud costs and ensure platform reliability for structured streaming workloadsDefine data governance, security and quality standards to maintain consistency across the platformCollaborate with stakeholders to translate complex business requirements into actionable technical solutionsDevelop integration approaches using Azure-native services such as Data Factory, Synapse and Blob StorageMentor data engineers, promote modern engineering practices and perform technical reviewsDrive adoption of CI/CD, Infrastructure as Code and automated testing in data engineering environmentsImplement observability and monitoring using tools like Databricks Workflows and related frameworksContribute to AI-driven initiatives by leveraging Databricks ML/MosaicML to integrate Generative AI and LLM-based solutionsRequirementsBachelor’s or Master’s degree in Computer Science, Software Engineering or related fieldExtensive experience designing and implementing production-grade platforms using Azure DatabricksExpertise in PySpark, including advanced optimization, data skew mitigation and query tuningStrong programming skills in Python with knowledge of modern software design principlesPractical experience with structured streaming, Delta Lake and Delta Live TablesProven experience in Lakehouse migration and modernization using open table formats such as Delta Lake or Apache IcebergProficiency with cloud-native services on Azure and knowledge of multi-cloud environments (AWS or GCP)Hands-on experience with CI/CD and Infrastructure as Code tools (Terraform, GitHub Actions, Jenkins)Strong leadership ability to guide teams, define epics/user stories and ensure delivery in agile environmentsExcellent communication and stakeholder management skills for both technical and non-technical audiencesNice to haveExperience operationalizing LLM or Generative AI workflows in Databricks pipelinesFamiliarity with frameworks like LangChain, LlamaIndex or Databricks ML/MosaicMLKnowledge of AI governance, security practices and enterprise integration controlsBackground in financial trading data or related domainsOfficial Databricks certifications such as Certified Data Engineer Professional or Apache Spark DeveloperWe offerEPAM Employee Stock Purchase Plan (ESPP)Protection benefits including life assurance, income protection and critical illness coverPrivate medical insurance and dental careEmployee Assistance ProgramCompetitive group pension planCyclescheme, Techscheme and season ticket loansVarious perks such as free Wednesday lunch in-office, on-site massages and regular social eventsLearning and development opportunities including in-house training and coaching, professional certifications, and coursesIf otherwise eligible, participation in the discretionary annual bonus programIf otherwise eligible and hired into a qualifying level, participation in the discretionary Long-Term Incentive (LTI) Program #J-18808-Ljbffr