Senior Data Engineer

Role Summary

We are seeking a technically strong and delivery‑focused Senior Data Engineer to contribute to the design, development, and optimization of scalable, secure, and governed data pipelines on Azure and Databricks. You will work closely with data analysts, data scientists, business analysts, and architects to enable high‑performance data flows that power analytics, AI initiatives, and business innovation.

This role requires deep engineering expertise, strong collaboration skills, and a product‑oriented mindset. You will also partner with the Data Architect to align with enterprise architecture and data modeling standards, and with Business Owners/Business Analysts to translate business requirements into reusable and well‑designed data assets. Active participation in Data Governance initiatives is essential to support consistent and robust governance across data domains.

Key Responsibilities

1. Data Pipeline Development

• Design and implement robust data pipelines using Spark, Delta Lake, and Azure Data Factory
• Build scalable ingestion, transformation, and enrichment workflows across structured and semi-structured data
• Optimize performance, cost-efficiency, and reliability of data jobs

2. Platform Integration & Maintenance

• Operate within Databricks and Azure environments to support lakehouse architecture
• Collaborate with Cloud Platform Engineers to implement CI/CD, monitoring, infrastructure‑as‑code, and policy‑as‑code practices
• Work with Data Architect to embed data lineage, metadata tagging, and classification into all pipelines

3. Collaboration & Product Enablement

• Partner with Data Analysts and Scientists to deliver curated datasets and feature stores
• Translate business requirements into technical solutions with the Business Analyst and/or Data Analysis Manager
• Collaborate with the Governance team to implement privacy, consent, and retention policies

4. Data Quality & Compliance

• Implement validation, anomaly detection, and exception handling mechanisms
• Participate in data profiling and quality monitoring to ensure trusted data delivery
• Maintain auditability and traceability across all data flows
• Ensure compliance with PDPA and internal governance standards

Qualifications & Preferred Skills

Required

• 3+ years of experience in data engineering or related field
• Strong hands-on experience with Azure Data Services, Databricks, Spark (PySpark or Scala)
• Proficiency in SQL, Delta Lake, and data modeling for analytics
• Experience with CI/CD tools, version control, and orchestration frameworks (e.g., Airflow, ADF)
• Familiarity with data governance concepts including classification, lineage, and access control
• Strong collaboration skills and ability to work in cross-functional teams
• Product-oriented mindset with focus on reusability, scalability, and business value

Preferred

• Experience in banking, fintech, or regulated industries
• Exposure to Unity Catalog, Azure Purview, and Power BI integration
• Knowledge of privacy-by-design, consent enforcement, and ethical data use
• Passion for building resilient, scalable, and governance-ready data platforms