Responsibilities:
The Senior Data Engineer is a key contributor responsible for collecting, transforming, and managing data from various sources, ensuring its quality, security, and accessibility. They design data pipelines, optimize storage solutions, and collaborate with data stakeholders to support informed decision-making within an organization, and have the opportunity to work with pre-built machine learning models to support informed decision-making within an organization.
Managing and maintaining data pipeline
- End-to-end pipeline orchestration, data schedule and automate pipeline runs, ensuring data is processed at the right intervals, whether it's batch processing or real-time streaming.
- Set up mechanisms to collect data pipelines that outline the flow of data from source to destination
- Set up mechanisms to collect data from variety of sources, which can include databases, external APIs, log files, IoT devices, and more.
- Understand data extraction processes such as ETL, ELT, CT, and CDC to retrieve data.
- Understand dbt (data build tool) to manage data quality, data documents, and dependencies.
- Strongly understand Airflow and dbt (data build tool) to manage data quality, data documents, and dependencies.
- Monitoring and error handling to ensuring that data flows smoothly and efficiently.
- Strongly understand data modern stack trends.
- Understand how to use Confluent or Kafka for data streaming pipeline.
- Understand CI/CD pipelines.
Data Optimization
- Performance optimization for SQL Script.
- Strong programming language skills in SQL and Python are required
- Web scrapping analysis.
- DBT (data build tool) for data transformation and data modelling.
- Logging and Monitoring.
Data Integrity and Compliance
- Ensuring the quality, accuracy and consistency of data by implementing data validation checks and monitoring data sources for anomalies.
- Managing data retention and archiving policies to determine how long data should be retained and when it should be archived.
- Managing data governance including compliance to local country regulations e.g. PDPA and data security.
Design Documents
- Clarify and understand the functionality of each data job.
- Document data processes, configurations, and best practices and communicate effectively with team members and stakeholders.
- Data lineage and dbt to represent dataflow.
Requirements:
- Bachelor's degree in Computer Science, Computer Engineering, Quant/Maths
- Experience in SAP is a plus.
- Experience in Machine Learning is a plus.
- Experience with the Cloud services especially on Google Cloud services.
- Good communication both Thai & English.
- Understand MLOps is a plus
- Highly motivated and proactive with ability to work under pressure to meet demanding deadlines and able to work independently.
- Can do spirit.
- Team Player.
- Eager to learn new services.
