Job Summary:
We are seeking a highly skilled and experienced Senior Data Engineer with deep expertise in Databricks and Azure Data Factory. The candidate will be responsible for designing and implementing robust data pipelines that ingest, transform, and model data from our SAP systems into scalable and reusable data products. This role requires strong proficiency in PySpark and SQL, a solid understanding of data modelling principles, and hands-on experience with Databricks.
Responsibilities:
- Design, develop, and maintain scalable data pipelines using Databricks and Azure Data Factory.
- Ingest and process data from SAP systems, ensuring data quality, consistency, and reliability.
- Model raw data into structured, reusable data products that support business intelligence and analytics use cases.
- Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements and deliver solutions.
- Optimize data workflows for performance and cost-efficiency in a cloud-native environment.
- Implement best practices for data engineering, including version control, testing, and documentation.
- Monitor and troubleshoot data pipelines to ensure high availability and performance.
- Communicate proactively and clearly with stakeholders, demonstrating a can-do attitude and a collaborative mindset.
Primary Skills (Must Have):
- Databricks Extensive experience building and managing data pipelines and notebooks.
- Azure Data Factory Strong expertise in orchestrating data workflows and integrating with various data sources.
- Data Modelling Proficient in designing logical and physical data models for analytics and reporting.
- PySpark Advanced programming skills for large-scale data processing.
- SQL Expert-level skills in writing complex queries and optimizing performance.
Skills (Nice to Have):
- Azure Synapse Analytics Experience with data warehousing and analytics solutions.
- CI/CD for Data Pipelines Familiarity with DevOps practices in data engineering.
- Delta Lake Knowledge of ACID transactions and time travel in Databricks.
Experience:
- Total Experience: 5+ years in data engineering and analytics.
- Relevant Experience: 3+ years working with Databricks and Azure Data Factory
