Python-Spark Scala

Key Responsibilities:

• 5 9 years of experience in Data Engineering Big Data development
• Strong hands on experience with Python and Apache Spark Scala PySpark
• Deep understanding of Spark architecture RDD DataFrames Spark SQL
• Experience in distributed computing and big data processing
• Strong knowledge of SQL and data modeling concepts
• Experience with data pipeline development ETL ELT
• Familiarity with Linux Unix environments
• Experience with version control tools Git

Technical Requirements:

• Primary skills Technology Big Data Data Processing Spark Technology Functional Programming Scala Technology Machine Learning Python

Additional Responsibilities:

• Experience with cloud platforms AWS Azure or GCP
• Hands on with Databricks EMR Spark clusters
• Knowledge of streaming technologies Kafka Spark Streaming Structured Streaming
• Experience with workflow orchestration tools Airflow Oozie
• Familiarity with Delta Lake Lakehouse architecture
• Exposure to NoSQL databases MongoDB Cassandra
• Knowledge of CI CD and DevOps practices

Preferred Skills:

Technology->AI-Data science->PYTHON,Technology->Big Data - Data Processing->Spark->SparkSQL,Technology->Functional Programming->Scala