Senior Databricks Data Engineer

Overview

The Senior Databricks Data Engineer role involves building, developing, and managing the enterprise data platform for Stack Infrastructure Emea. This production-focused position requires designing reliable pipelines that integrate Finance, Delivery, Operations, HR, and PMO systems while establishing governed and high-quality datasets to support secure, scalable analytics and reporting. The successful candidate will take ownership of data engineering solutions, improve platform reliability and performance, and work alongside both technical and business stakeholders.

Requirements

Applicants must have strong hands-on experience building and supporting production data pipelines directly in Databricks, as experience with other cloud data platforms alone is not sufficient. The role requires practical experience with PySpark, Python, SQL, and Delta Lake, including data transformation and performance optimization. Candidates need experience orchestrating and troubleshooting Databricks data pipelines using Workflows, Delta Live Tables, or Lakeflow, along with essential working knowledge of Unity Catalog, data governance, access controls, and data quality practices.

Additional necessary experience includes integrating data from APIs, business applications, databases, and files, supporting Databricks workspaces, compute resources, or SQL Warehouses in a production environment, and utilizing Azure data services, Power BI, Git, and CI/CD practices. Candidates should also understand environment management, monitoring, troubleshooting, security, and cost control, while demonstrating the ability to communicate data issues clearly to non-technical stakeholders and document solutions effectively. Preferred qualifications include holding a Databricks Certified Data Engineer Professional or Databricks Certified Platform Administrator certification.