Azure Data Engineer - Data Pipeline

Key Responsibilities:

- Design, develop, and maintain scalable data pipelines on Azure to support various data integration and analytics needs.

- Implement data ingestion, transformation, and loading processes using Azure Data Factory, Databricks, and other relevant Azure services.

- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and ensure data quality and integrity.

- Optimize and troubleshoot existing data pipelines to improve performance and reliability.

- Monitor and manage data pipeline performance, ensuring data is processed in a timely and efficient manner.

- Implement data security and compliance measures in line with industry standards and best practices.

- Document data processes, workflows, and pipeline architectures.

Qualifications:

- Bachelor's degree in Computer Science, Information Technology, or a related field.

- Proven experience as a Data Engineer with a focus on Azure technologies.

- Proficiency in Azure Data Factory, Azure Databricks, Azure SQL Database, and other relevant Azure services.

- Strong SQL skills and experience with data modeling and ETL processes.

- Familiarity with programming languages such as Python, Scala, or Java.

- Experience with big data technologies such as Spark, Hadoop, or similar.

- Excellent problem-solving skills and attention to detail.

- Strong communication and collaboration skills.

Preferred Qualifications:

- Azure certifications such as Azure Data Engineer Associate or Azure Solutions Architect.

- Experience with data visualization tools like Power BI.

- Knowledge of machine learning and data science concepts.

- Familiarity with DevOps practices and tools for CI/CD in data engineering.