Job Summary
Design, develop, and maintain scalable data infrastructure and pipelines to support data-driven products. Collaborate cross-functionally to ensure data accuracy, integration, and compliance in a cloud-based big data environment.
Responsibilities
- Design, develop, and deploy data tables, views, and marts across data warehouses, operational data stores, data lakes, and data virtualization platforms to support business needs
- Extract, clean, transform, and manage data flows, including web scraping techniques, to ensure high data quality and availability
- Build, launch, and maintain efficient large-scale batch and real-time data pipelines using data processing frameworks for reliable data delivery
- Integrate and consolidate disparate data silos in scalable and compliant ways to enable unified data access
- Collaborate with project managers, data architects, business analysts, frontend developers, designers, and data analysts to build scalable, data-driven products
- Develop backend APIs and manage databases to support application functionality and performance
- Apply Agile methodologies with continuous integration and delivery to accelerate development cycles
- Engage in pair programming and code reviews to maintain code quality and knowledge sharing
Required competencies and certifications
- Proficient in data cleaning and transformation using tools such as SQL, pandas, or R to ensure data accuracy and consistency
- Skilled in building ETL pipelines with technologies like SQL Server Integration Services (SSIS), AWS Database Migration Services (DMS), Python, AWS Lambda, ECS Container tasks, Event bridge, AWS Glue, or Spring
- Experienced in database design and management across various systems including SQL, PostgreSQL, AWS S3, Athena, MongoDB, Postgres/GIS, MySQL, SQLite, Volt DB, and Cassandra
- Knowledgeable in cloud platforms such as AWS, Azure, or Google Cloud for big data engineering
- Experienced in building production-grade data pipelines and ETL/ELT data integration workflows
- Understanding of system design, data structures, and algorithms to optimize data processing
- Familiar with data modeling, data access methods, and storage infrastructures like Data Mart, Data Lake, Data Virtualization, and Data Warehouse for efficient data retrieval
- Familiar with REST APIs and web protocols to support data integration and application communication
- Familiar with big data frameworks and tools such as Hadoop, Spark, Kafka, and RabbitMQ
- Experienced with web scraping technologies including Beautiful Soup, Casper.JS, Phantom.JS, Selenium, and Node.js for customized data extraction
- Knowledgeable about data governance policies, access control, and security best practices to ensure data compliance
- Comfortable scripting in languages such as SQL and Python for automation and data manipulation
- Proficient in both Windows and Linux development environments for versatile deployment and testing
- Demonstrates interest in bridging engineering and analytics teams to enhance data-driven decision-making
