Senior Data and AI Engineer

Description

What You'll Do

• Design, build, and operate reliable, secure, and observable data pipelines and curated datasets that power enterprise reporting, analytics, and AI/ML use cases.
• Lead AI/ML engineering as a core workstream, designing feature-ready datasets, model pipelines, and AI-ready data products, and applying engineering rigour (testing, versioning, observability) to ML pipelines.
• Engineer data products and pipelines that support LLM and generative AI use cases, including retrieval-ready data structures and pipelines feeding AI applications.
• Drive engineering automation as a standing discipline, evaluating and adopting AI-assisted code generation and testing tools to reduce manual engineering effort, and building internal tooling and patterns the wider team can use to build faster.
• Own engineering quality, performance, and cost optimisation across the platform, implementing data quality controls, testing frameworks, monitoring, and observability.
• Build and maintain production-grade data infrastructure on Azure / Microsoft Fabric, including data lakes, lakehouses, and modern data warehouse patterns.
• Define and implement CI/CD pipelines for data and ML engineering workflows, applying infrastructure-as-code and automated quality gates as standard practice.
• Apply containerisation and orchestration tooling (e.g. Docker, Airflow, or equivalent) to production data and ML workflows.
• Partner across architecture, governance, data modelling, and reporting to deliver coherent, end-to-end data and AI products.
• Mentor and support engineers, setting the standard for quality, craft, and engineering rigour, including how the team uses AI-assisted automation.
• Contribute engineering expertise to RES's Synapse-to-Fabric migration programme, working alongside the platform architect to convert pipelines and warehouse objects at scale using AI-assisted tooling.

What You'll Bring

• Previous experience as a data engineer and senior engineer. Typically 10+ years.
• Azure Fabric data platform — expertise across Azure Data Factory, Synapse, Microsoft Fabric, Purview, Unity Catalogue, and Data Lake / Lakehouse architectures.
• Python — advanced proficiency including open-source data and ML libraries, frameworks, and production pipeline development.
• SQL — expert-level for data modelling, transformation, and complex query optimisation.
• AI/ML engineering — building data infrastructure for machine learning and AI use cases, feature engineering, model pipeline support, and production ML pipeline engineering.
• AI-assisted engineering automation — experience using AI coding and conversion tools (e.g. Copilot, Claude, or equivalent) to accelerate engineering work at scale, with a clear approach to validating their output.
• MLOps — CI/CD for data and ML pipelines, infrastructure as code, containerisation, and orchestration tools such as Airflow or equivalent.
• Data quality & observability — hands-on experience with testing frameworks, monitoring, and quality controls in production environments.
• LLMs and generative AI — practical understanding of how to engineer data products and pipelines that support LLM and GenAI use cases.
• Technical leadership — track record of engineering and architectural decision-making across data and AI/ML disciplines, setting standards, and delivering automated engineering work while contributing to strategy and roadmap thinking.

Your Background

Essential

• Degree in computer science, data engineering, software engineering, or a related field — or equivalent hands-on experience.
• Significant experience (typically 10+ years) delivering enterprise-grade data engineering solutions in production environments, with meaningful experience in ML/AI engineering.
• Proven track record as a Senior Data Engineer or Senior Data & AI/ML Engineer, including building large-scale data and ML systems.
• Deep expertise in the Microsoft Azure data ecosystem — ADF, Synapse, Fabric, Purview, Unity Catalogue.
• Advanced Python skills including open-source data and ML libraries, frameworks, and messaging systems.
• Strong experience building and maintaining production data infrastructure for AI and ML consumption, including model pipelines and feature engineering.
• Experience with MLOps practices: CI/CD for data and ML pipelines, automated testing, and infrastructure as code.
• Experience with modern data stack tooling — dbt, Airflow, Prefect, or equivalent orchestration and transformation frameworks.
• Experience with automation tooling such as Power Automate, Power Platform, or equivalent, and practical use of AI-assisted engineering tools in production settings.
• Relevant certifications in Microsoft Azure, data engineering, or AI/ML.
• Exposure to working alongside data scientists and AI engineers in a shared platform model.

Desirable

• Experience with a platform migration at scale — such as Synapse to Fabric or an equivalent large data platform transition — using automation or AI tooling to accelerate conversion work.

#J-18808-Ljbffr