Job Description:
We are seeking a highly skilled Data Engineer / Data Architect to design, build, and optimize scalable data architectures and frameworks. The ideal candidate will have expertise in Kafka, ETL processes, Databricks, and data platforms, ensuring efficient data ingestion, transformation, and processing.
Key Responsibilities:
- Design and implement data architectures for scalable, high-performance data pipelines.
- Develop and maintain ETL/ELT workflows using modern data engineering tools.
- Implement real-time and batch data processing using Apache Kafka and other streaming technologies.
- Work with Databricks, Spark, and cloud-based data platforms to optimize big data processing.
- Build data frameworks and automation pipelines for efficient data engineering practices.
- Ensure data governance, security, and compliance best practices.
- Collaborate with cross-functional teams (Data Scientists, Analysts, Software Engineers) to ensure smooth data operations.
Required Skills & Experience:
- 15 years of experience in Data Engineering / Data Architecture.
- Strong hands-on experience with Apache Kafka (data ingestion, streaming).
- Expertise in ETL tools, data modelling, and pipeline development.
- Experience with Databricks, Apache Spark, or similar big data frameworks.
- Knowledge of cloud platforms (AWS, Azure, GCP) for data storage and processing.
- Experience with SQL, Python or Scala for data engineering tasks.
- Strong understanding of data governance, security, and best practices.
Preferred Qualifications:
- Experience with modern data stack technologies ( Delta Lake, Airflow).
- Hands-on experience with containerization (Docker, Kubernetes) for data workloads.
- Familiarity with CI/CD for data pipelines.
