Senior Consultant II – Software Engineer

• Design, develop, and maintain scalable batch and streaming data pipelines using Apache Spark, cloud-native technologies, and modern data platforms
• Build and optimize ETL/ELT frameworks to ingest, transform, validate, and curate data from diverse enterprise source systems
• Develop and implement data solutions leveraging Microsoft Fabric, OneLake, Lakehouse, Data Warehouse, and Semantic Model capabilities
• Design data models, storage strategies, and data processing architectures supporting enterprise analytics and reporting
• Partner with Analytics Engineers, BI teams, Data Scientists, Product Managers, Platform Engineers, Security teams, and business stakeholders
• Deliver trusted, reusable, scalable, and analytics-ready datasets and data products
• Drive engineering best practices including code quality, peer reviews, automated testing, CI/CD, and documentation
• Ensure data quality, governance, lineage, metadata management, security, privacy, and compliance requirements
• Monitor and optimize performance, scalability, reliability, and cost efficiency of distributed data processing workloads
• Develop reusable frameworks, libraries, templates, and engineering patterns
• Implement and support CI/CD pipelines, Infrastructure-as-Code principles, and deployment automation
• Troubleshoot complex data integration, performance, and operational issues
• Create and maintain technical documentation, architecture diagrams, operational runbooks, and implementation guides
• Participate in technical design sessions, architecture reviews, and engineering discussions
• Mentor and guide junior and mid-level data engineers through coaching, code reviews, design reviews, and knowledge sharing
• Evaluate and adopt emerging technologies, engineering practices, and AI-assisted development capabilities
• Support production operations through monitoring, root cause analysis, incident resolution, and continuous improvement
• Contribute to Agile delivery processes including sprint planning, estimation, backlog refinement, and execution
• Promote DataOps practices, automation, observability, and engineering excellence
• Serve as a technical subject matter expert for modern data platforms, Spark processing, Microsoft Fabric, and lakehouse architectures

Requirements

• Legal right to work in the UK for employment at Allstate
• Allstate is not providing sponsorship for this vacancy
• Minimum of 2 years of experience in Data Engineering and Big Data technologies
• Deep expertise in Apache Spark (PySpark/Spark SQL)
• Experience with Microsoft Fabric, OneLake, Lakehouse Architecture, Data Warehouse, and building Semantic Models
• Strong SQL and data modeling skills
• Experience building and optimizing ETL/ELT pipelines
• Strong proficiency in Java and experience building distributed applications and services
• Experience with Kafka or similar event-streaming platforms
• Understanding of Data Governance, Data Quality, Security, and Privacy controls
• Ability to mentor junior engineers and provide technical leadership without direct people management responsibilities
• Knowledge of streaming and batch data processing patterns
• Experience with observability solutions including logging, monitoring, tracing, and performance analysis
• Experience working in cloud-based analytics and data platform environments
• Experience with CI/CD, Git-based development, testing, and deployment automation
• Experience working in Agile delivery environments

Core Competencies

Demonstrates expertise in designing and developing scalable data pipelines using Apache Spark and Microsoft Fabric, with a strong focus on ETL/ELT processes, data governance, and cloud-based analytics. Proven ability to mentor junior engineers and drive engineering best practices in Agile environments.

Highest-signal resume keywords

• Apache Spark (PySpark/Spark SQL)
• Microsoft Fabric
• ETL/ELT Pipeline Optimization
• Data Governance and Quality
• CI/CD and Deployment Automation

Hard Skills

• Data Engineering
• Big Data Technologies
• SQL
• Data Modeling
• Java
• Kafka
• Streaming Data Processing
• Observability Solutions
• Cloud-Based Analytics
• Agile Delivery

Soft Skills

• Mentoring
• Technical Leadership
• Collaboration

Industry Keywords

• Data Governance
• Data Quality
• Security
• Privacy Controls
• DataOps

Tools & Technologies

• Apache Spark
• Microsoft Fabric
• OneLake
• Lakehouse Architecture
• Data Warehouse
• CI/CD
• Git

#J-18808-Ljbffr