Open roleExternal
Senior Data Engineer – Databricks & DLT
- london, england, United Kingdom
- Permanent·On-site
- Full time
- £90 - £130 Per Hour
Job Description
Senior Data Engineer – Databricks & DLT (Delta Live Tables)
Certain Advantage are recruiting on behalf of our trading client for a Senior Data Engineerwith strong analytical capabilities and hands‑on experience in building scalable data solutions using a programming language like Python or Scala as well as experience in Databricks and Delta Live Tables - This experience is essential.
Suitable candidates will have expertise inDatabricks,streaming architectures, data ingestion frameworks, anddashboarding, along with experience working withSAP data sources.
You will be responsible for designing and implementing scalable data pipelines and enabling real‑time and batch analytics across the organization.
Your Responsibilities:
- Design and implement scalable data pipelines usingDatabricks and Delta Live Tables (DLT).
- Develop and managestreaming data solutionsfor real‑time analytics and decision‑making.
- Ingest data from various sources—includingSAP systems using Qlik ReplicateandADFto ingest data from various sources ensuring data quality and consistency.
- Performdata modellingand optimize data structures for analytics and reporting.
- Collaborate with cross‑functional teams to understand data requirements and deliver robust solutions.
- Monitor, troubleshoot, and enhance data workflows for performance and reliability.
- Maintain technical documentation and contribute to data engineering best practices.
Required Skills & Qualifications:
- Strong experience withDatabricks- DLT, Delta Lake concepts, UC governance.
- Solid understanding ofstreaming technologies(e.g., Spark Structured Streaming, Autoloader).
- Programming skills inPython,SQL, or
- Proficiency indata modelling, ETL/ELT processes, and data architecture.
- Experience in MS Copilot, AI Foundry , Databricks (MosiacAI, MLflow, Agentbricks, Genie)
- Strong analytical background with problem‑solving skills.
- Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing.
- Hands‑on expertise inADF, and Qlik Replicatefor data ingestion and replication.
- Experience working inAzure cloud environments.
- Bachelor's or Master’s degree in Computer Science, Engineering, or a related field.