AWS Data Engineer, SSE

• Provide technical and development support to clients to build and maintain data pipelines.
• Develop data mapping documents listing business and transformational rules.
• Develop, unit test, deploy and maintain data pipelines.
• Design a Storage Layer for storing tabular/semi-structured/unstructured data.
• Design pipelines for batch/real-time processing of large data volumes.
• Analyze source specifications and build data mapping documents.
• Identify and document applicable non-functional code sets and reference data across insurance domains.
• Understand profiling results and validate data quality rules.
• Utilize data analysis tools to construct and manipulate datasets to support analyses.
• Collaborate with and support Quality Assurance (QA) in building functional scenarios and validating results.
• 
  

Requirements

• 4+ years experience developing and maintaining modern ingestion pipeline using technologies like (AWS pipelines, Lamda, Spark, Apache Nifi etc).
• Basic understanding of the MLOPs lifecycle (Data prep -> model training -> model deployment -> model inference -> model re-training).
• Should be able to design data pipelines for batch/real time using Lambda, Step Functions, API Gateway, SNS, S3.
• Hands on experience on AWS Cloud and its Native components like S3, Athena, Redshift & Jupyter Notebooks.
• Requirements Gathering - Active involvement during requirements discussions with project sponsors, defining the project scope and delivery timelines, Design & Development.
• Strong in Spark Scala & Python pipelines (ETL & Streaming).
• Strong experience in metadata management tools like AWS Glue.
• Strong experience in coding with languages like Java, Python.
• Good-to-have AWS Developer certified.
• Good-to-have Postman-API and Apache Airflow or similar schedulers experience.
• Working with cross-functional teams to meet strategic goals.
• Experience in high volume data environments.
• Critical thinking and excellent verbal and written communication skills.
• Strong problem-solving and analytical abilities should be able to work and deliver individually.
• Good knowledge of data warehousing concepts.