Lead Data Engineer
Primary Skills
- ETL Fundamentals, SQL, SQL (Basic + Advanced), Python, Data Warehousing, Modern Data Platform Fundamentals, Data Modelling Fundamentals, PLSQL, T-SQL, Stored Procedures
Specialization
- Databricks Engineering: Data Engineer
Job requirements
Job Description - Data Engineer (Databricks)
Company: Brillio Technologies
Role: Data Engineer - Databricks
Experience: 5-10 Years
Location: India (Hybrid/3 days)
Employment Type: Full-time
About Brillio
Brillio is a leading digital transformation partner delivering innovative solutions across Data, Cloud, AI, and Digital Engineering. We help global enterprises unlock value through cutting-edge technology and data-driven insights.
Role Overview
We are looking for a
hands-on Data Engineer with strong Databricks expertise to design, build, and scale modern data platforms. The ideal candidate will have experience in
PySpark, Delta Lake, and cloud-native data engineering and will work closely with analytics, product, and business teams to deliver high-performance data pipelines.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark
- Build and optimize data lakehouse architectures (Bronze-Silver-Gold layers)
- Develop data ingestion frameworks from multiple sources (batch & real-time)
- Implement Delta Lake features including schema evolution, ACID transactions, and optimization
- Create and manage Databricks notebooks, workflows, and jobs for production pipelines
- Ensure data quality, governance, and lineage across pipelines
- Optimize Spark jobs for performance (partitioning, caching, tuning)
- Collaborate with cross-functional teams (Data Scientists, Analysts, Architects)
- Support production deployments, monitoring, and troubleshooting of data pipelines
- Contribute to migration of legacy data platforms to modern cloud-based architecture
Technical Skills (Must-Have)
- Strong experience in:
- Databricks (Workflows, Notebooks, Jobs)
- PySpark / Spark SQL
- SQL & Data Warehousing concepts
- Hands-on experience with:
- Delta Lake / Lakehouse architecture
- Data pipeline development & orchestration
- Good understanding of:
- Data modeling, ETL/ELT processes, SCD concepts
Cloud & Tools
- Experience in at least one:
- AWS (S3, Glue, EMR) / Azure (ADF, ADLS, Synapse)
- Workflow tools:
- Airflow / Databricks Workflows
- Version control:
Good to Have
- Experience with:
- Streaming frameworks (Kafka / Spark Streaming)
- DBT / Snowflake / Redshift / Fabric
- Exposure to:
- Unity Catalog, Autoloader, Delta Live Tables
Experience Required
- 5-10 years of experience in Data Engineering
- Minimum 2-4 years of hands-on Databricks experience
- Proven experience building enterprise-scale data platforms and pipelines
Soft Skills
- Strong problem-solving and analytical skills
- Ability to work in a fast-paced, agile environment
- Good communication and stakeholder management
Why Join Brillio
- Work on cutting-edge data & AI programs
- Opportunity to build large-scale data platforms
- Collaborative and innovation-driven culture
