AWS Data Engineer

Key Responsibilities

Architecture & Design

• Design and architect the end-to-end AWS Data Lake and Lakehouse solution, including Landing Zone, Transformed Zone, and Curated/Consumption Zone layers
• Define and govern data architecture standards, patterns, and best practices across the platform
• Architect reusable data ingestion pipelines supporting REST APIs, JDBC databases, S3 file uploads, and SaaS connectors (e.g. Salesforce via AWS AppFlow)
• Design data storage strategies including hot, warm, and cold storage tiers, encryption, and data lifecycle policies

Development & Deployment

• Develop and deploy data ingestion pipelines using AWS Glue, Lambda, Step Functions, EventBridge, and API Gateway
• Build and maintain data transformation workflows (batch and stream processing) using AWS Glue and Amazon Redshift
• Implement orchestration, monitoring, logging, and notification frameworks for pipeline operations
• Develop and maintain the AWS Glue Data Catalogue, including schema evolution tracking and metadata tagging

Security & Governance

• Configure and enforce data security policies using AWS Lake Formation, IAM, and Secrets Manager
• Implement granular access controls at database, table, and column levels
• Ensure compliance with data classification, retention, and audit requirements
• Support data quality frameworks and observability monitoring

Maintenance & Operations

• Monitor platform health, performance, and pipeline reliability
• Troubleshoot and resolve data pipeline failures and data quality issues
• Maintain documentation for architecture decisions, pipeline configurations, and operational runbooks
• Continuously optimise platform performance and cost efficiency on AWS

Requirements

Essential

• Minimum 3 to 5 years of experience in data engineering, data architecture, or cloud infrastructure roles
• Hands-on expertise with core AWS data services: Amazon S3, AWS Glue, Amazon Redshift, AWS Lambda, Amazon Kinesis, AWS Step Functions, Amazon EventBridge, AWS AppFlow, AWS Lake Formation
• Strong proficiency in SQL and at least one scripting language (Python or Scala)
• Experience designing and implementing Data Lake or Lakehouse architectures
• Solid understanding of data governance, data cataloguing, and metadata management
• Experience with batch and streaming data processing patterns
• AWS Certified Data Engineer - Associate or AWS Certified Solutions Architect certification (or equivalent)

Preferred

• Experience integrating with Tableau or similar BI visualisation tools via Amazon Redshift or S3
• Familiarity with MLOps frameworks and AI/ML model deployment on AWS SageMaker
• Experience with Salesforce data integration using AWS AppFlow
• Knowledge of Change Data Capture (CDC) and incremental data load patterns
• Prior experience in a government or public sector data environment