DataOps Technical Lead – Databricks/Azure
Experience : 6-10yrs
Work Location: Bangalore / Chennai / Gurgaon/ Pune/ Kolkata / Hyderabad
Job Description, Essential Duties and Responsibilities:
- DataOps Lead is responsible for leading technical team and managing end-to-end project deliverables.
- Should have strong experience and knowledge in DataOps support deliverables, KPIs and platform administration for Azure-Databricks DataOps project
- Data Ops & Support roles
- Pipeline Monitoring & Incident Response
- Ticket/Incident Management
- Data Anomaly Investigations & Root Cause Analysis
- Hotfixes, Enhancements & Release Coordination
- Creating Runbooks and Knowledge Base documents
- Data Quality Tracking
- Platform Administration
- Databricks Platform Administration
- Azure Infrastructure Management
- Identity, Access & Governance
- Data Pipeline Architecture & Platform Standards
- Tools & Technologies
- Databricks
- Azure data services
- SQL Server
- Azure Synapse
- Azure Data Factory (ADF)
- Python and PySpark scriptingusing DLTs
- Github
- AI based tools (Claude, Co-Pilotetc)
- Linear (task management)
- ServiceNow[AG1]
- Cosmos
- ADO
- Data pipelines: Monitor, identify issue, Debugging and Bug fixing of data pipelines. Should understand complex data pipelines and data engineering concepts. Mentoring and guiding junior team members.
- Data Quality & Data Observability: Identify and fix the Data quality issues by performing RCA. Should have a good understanding of data observability tools like Unravel etc.
- Data orchestration: Orchestration of jobs using Airflow.
- DevOps: Should have a good understanding of the Dev Ops process.
- Collaboration: Work closely with data engineering, data science, and IT teams to support data pipeline development, troubleshoot issues, and provide guidance on best practices.
- Incident Management: Lead troubleshooting and resolution of issues related to Databricks clusters and jobs, ensuring minimal downtime and impact on business operations.
- KPI: Responsible for managing MTTA, MTTR, IRR, response & resolution SLA and other key delivery matrix.
- Upgrades & Patches: Manage upgrades, patches, and new releases of Databricks, ensuring minimal disruption to ongoing operations.
- Documentation: Create and maintain detailed documentation of the Databricks environment, configurations, and procedures.
- SOP: Able to build SOPs and set up DataOps best practices.
- Continuous improvement and Innovation: Bringing continuous improvement and innovation for cost optimization, effort reduction, incidents reduction, platform stability and discuss with client, Development team for implementation.
- Team Leading: Lead the technical team and manage day-to-day technical deliverables.
- Escalation: Manage client's day-to-day technical escalation
- Stakeholder management: Able to handle key stakeholders, including client leadership.
Skills, Education and Experience:
- Having 8+ years of Azure-Databricks DataOps experience.
- Team handling experience: Able to handle a team of 5-7 engineers
- Open to work in UK & US Shift.
- Working in agile methodology and Sprint framework
- Must have education degree of B. Tech/B.E/MCA
