Kafka Data Engineer

ITS A CONTRACT TO HIRE (C2H) ROLE

We are seeking an experienced Kafka Data Engineer with strong expertise in Kafka Topics Migration, Data Streaming, and Distributed Systems. The candidate will be responsible for planning and executing Kafka cluster migrations, topic replication, data validation, performance tuning, and ensuring zero/minimal downtime during migration activities.

Key Responsibilities

• Design and execute Kafka topic migration across environments, clusters, or cloud platforms.

• Perform Kafka cluster upgrades, broker migrations, and data replication using tools such as MirrorMaker 2.0, Confluent Replicator, Cluster Linking, etc.

• Ensure seamless migration of topics, schemas, consumer groups, offsets, ACLs, and configurations.

• Monitor Kafka performance, throughput, latency, and optimize cluster health.

• Develop automation scripts using Python, Shell scripting, or similar technologies.

• Collaborate with application teams to validate producers, consumers, and streaming pipelines post-migration.

• Implement security controls including SSL, SASL, Kerberos, RBAC, ACLs.

• Troubleshoot Kafka connectivity, replication, partition balancing, and consumer lag issues.

• Work closely with DevOps, Cloud, and Data Engineering teams for deployment and infrastructure management.

• Prepare migration documentation, rollback plans, and post-migration reports.

Required Skills

• Strong experience with Apache Kafka Administration and Development.

• Hands-on experience in Kafka Topics Migration and cluster-to-cluster replication.

• Experience with Kafka Connect, Kafka Streams, Schema Registry, and Confluent Platform.

• Strong knowledge of MirrorMaker 2.0, Replicator, Cluster Linking.

• Experience with Linux/Unix environments.

• Proficiency in Python, Java, or Scala.

• Good understanding of distributed systems, event-driven architecture, and messaging systems.

• Experience with cloud platforms such as AWS MSK, Azure Event Hubs/Kafka, or GCP Pub/Sub.

• Exposure to CI/CD pipelines and Infrastructure as Code tools like Terraform.

Preferred Skills

• Experience with Databricks, Spark Streaming, Flink, or other real-time processing frameworks.

• Knowledge of containerization technologies such as Docker and Kubernetes.

• Exposure to monitoring tools like Prometheus, Grafana, Splunk, ELK.

• Kafka Certification or Confluent Certification is a plus.