Key Responsibilities
Infrastructure Design and Management:
- Design and provision scalable, secure, and cost-efficient infrastructure on AWS using Terraform (mandatory).
- Deploy and manage services such as Databricks (on AWS) , Amazon Redshift , EMR , and RDS using Infrastructure-as-Code.
- Maintain version-controlled, reusable infrastructure modules in alignment with IaC best practices.
DevOps and CI/CD Automation:
- Develop and maintain CI/CD pipelines using Jenkins (mandatory) integrated with Git-based workflows.
- Automate infrastructure provisioning, testing, and deployment processes.
- Support CI/CD for data orchestration tools such as Airflow , Databricks , and Redshift .
Data Pipeline Platform Management:
- Collaborate with data engineers to build and deploy pipelines using AWS Glue , Kinesis , Lambda , S3 , and Redshift .
- Implement orchestration using Apache Airflow (open-source and AWS MWAA).
- Ensure reliability, scalability, and security throughout the data lifecycle.
Monitoring, Logging Optimization:
- Set up monitoring, logging, and alerting tools using CloudWatch , Prometheus , Grafana , and CloudTrail .
- Perform root-cause analysis, troubleshoot issues, and optimize infrastructure and pipeline performance.
Security and Compliance:
- Apply AWS security best practices across environments, including IAM , VPC , KMS , encryption, and network segmentation.
- Conduct audits and implement governance policies.
Documentation Collaboration:
- Work cross-functionally with DevOps, data engineering, and cloud security teams.
- Document architecture, configurations, workflows, and reusables to drive team efficiency and knowledge sharing.
Required Skills Qualifications
Cloud Infrastructure-as-Code (IaC):
- Strong hands-on experience with AWS (EC2, S3, Lambda, RDS, Glue, Redshift, IAM, VPC, Kinesis).
- Expertise in Terraform for multi-account, multi-service AWS infrastructure.
DevOps CI/CD:
- Proficiency with Jenkins (mandatory).
- Experience with GitLab CI/CD, GitHub Actions, or AWS CodePipeline is a plus.
- Familiarity with Docker and Amazon EKS is desirable.
Data Platform Experience:
- Experience provisioning and operating platforms like Databricks , Amazon Redshift , and optionally Snowflake .
- Experience building and managing data pipelines using Glue , Airflow , and Step Functions .
Monitoring Troubleshooting:
- Strong understanding of observability tools and frameworks (e.g., CloudWatch, Grafana, Prometheus).
Scripting Automation:
- Proficient in Python , Bash , or similar scripting languages.
Security Governance:
- Hands-on with IAM roles/policies, VPC security, KMS encryption , and compliance controls.
Preferred Qualifications
- Certifications: AWS Solutions Architect, DevOps Engineer, or Data Analytics Specialty.
- Databricks, Redshift, or Snowflake certifications are a plus.
- Experience with other IaC tools (Ansible, CloudFormation) and workflow engines (Apache Airflow, MWAA).
- Exposure to big data platforms like EMR, Spark, or Hadoop.
