Project Role : Custom Software Engineer

Project Role Description : Develop custom software solutions to design, code, and enhance components across systems or applications. Use modern frameworks and agile practices to deliver scalable, high-performing solutions tailored to specific business needs.

Must have skills : Python (Programming Language)

Good to have skills : Oracle Procedural Language Extensions to SQL (PLSQL), Apache Airflow, Data Build Tool

Minimum 5 Year(s) Of Experience Is Required

Educational Qualification : 15 years full time education

Summary:

We are seeking a skilled and motivated Data Engineer to design, develop, and automate scalable data pipelines that support enterprise analytics and business intelligence. The ideal candidate will collaborate with business stakeholders to gather requirements, perform data profiling, implement data quality frameworks, and build robust ELT solutions using Python, SQL, Apache Airflow, DBT Cloud, and Snowflake. This role involves extracting data from multiple source systems, developing automated data pipelines, transforming data using modern data engineering practices, and ensuring high-quality, reliable, and governed data assets. The successful candidate should also have experience with CI/CD, version control, and AI-assisted development practices to improve engineering productivity.

Roles & Responsibilities:

Expected to be an SME, collaborate and manage the team to perform.
Responsible for team decisions.
Engage with multiple teams and contribute on key decisions.
Provide solutions to problems for their immediate team and across multiple teams.
Lead the planning and execution of software development activities to meet project goals and timelines.
Collaborate with business stakeholders to gather, analyze, and document business and technical data requirements.
Perform data profiling to understand source data, identify anomalies, and assess data readiness.
Define, implement, and maintain Data Quality (DQ) rules and automated validation checks to ensure data accuracy, completeness, consistency, uniqueness, and integrity.
Design, develop, and automate end-to-end ELT pipelines using Python, SQL, and Apache Airflow.
Extract data from various source systems including REST APIs, relational databases, cloud applications, files, and third-party platforms.
Load, optimize, and manage data in Snowflake while ensuring scalability, performance, and reliability.
Develop reusable data transformation models using DBT Cloud following Medallion Architecture (Bronze, Silver, and Gold layers).
Build metadata-driven and reusable ingestion frameworks with proper logging, monitoring, and error handling.
Validate source-to-target mappings and ensure end-to-end data integrity throughout the pipeline.
Optimize SQL queries, Python code, Airflow workflows, and DBT models for performance and maintainability.
Develop and maintain Airflow DAGs for workflow orchestration, scheduling, dependency management, and monitoring.
Build and maintain CI/CD pipelines using Azure DevOps and GitLab CI/CD for automated deployment of data engineering solutions.
Manage source code using GitHub, Azure DevOps Repos, and GitLab while following Git best practices, branching strategies, pull requests, and code review processes.
Leverage GitHub Copilot, Large Language Models (LLMs), and prompt engineering techniques to accelerate software development, testing, documentation, and troubleshooting.
Collaborate with data analysts, architects, DevOps engineers, and business teams to deliver scalable and reliable data solutions.
Document technical designs, pipeline logic, transformation rules, and operational procedures.
Follow data governance, security, coding standards, and engineering best practices.
Mentor junior team members to support their professional growth and skill development.
Facilitate effective communication between stakeholders to ensure alignment on project objectives and deliverables.

Professional & Technical Skills:

Must To Have Skills: Proficiency in Python (Programming Language).
Good To Have Skills: Experience with Apache Airflow, Oracle Procedural Language Extensions to SQL (PLSQL), Data Build Tool.
Strong knowledge of software development lifecycle and agile methodologies.
Strong proficiency in Python programming.
Advanced SQL development and query optimization.
Hands-on experience with Apache Airflow.
Experience with DBT Cloud for data transformation and modeling.
Strong knowledge of Snowflake data warehouse.
Experience building ETL/ELT pipelines.
Experience integrating data from REST APIs, relational databases, and external data sources.
Good understanding of data warehousing concepts and dimensional data modeling.
Experience implementing Medallion Architecture (Bronze, Silver, Gold).
Experience performing data profiling and implementing Data Quality (DQ) frameworks and validation rules.
Knowledge of source-to-target mapping, metadata management, and data lineage.
Proficiency with GitHub for source code management.
Familiarity with GitHub Copilot, LLM-based development tools, and prompt engineering.
Strong analytical, problem-solving, and debugging skills.
Excellent communication and stakeholder management skills.
Experience in designing scalable and maintainable software architectures.
Ability to write clean, efficient, and well-documented code.
Familiarity with version control systems and collaborative development tools.

Additional Information:

The candidate should have minimum 5 years of experience in Python (Programming Language).
This position is based at our Bengaluru office.
A 15 years full time education is required.