Data Engineer

Job Description:

We are having an urgent opening for a reputed company for the designation of Data Engineer for the Mumbai location.

Responsibilities:

1. Collect and process data
   1. Maintain existing ETL programs to ingest data from AWS, HDFS, SQL, and MySQL
   2. Implement API from existing sources
   3. Design and adapt ETL tools for new data sources, including streaming data
   4. Develop API for internal and external data exchanges
   5. Optimize data pipeline and storage requirements with creative designs
   6. Create an industry-leading process to efficiently collect and process data
2. Team collaboration
   1. Coordinate with multiple teams (data architect, data science, analytics, reporting, and technology) from different locations/time zone for ETL operations, Service Level Agreements
   2. Resolve ETL/API issues with other developers, or users
   3. Support Quality Assurance team for data provisioning, storage, and calculation routes
3. Reporting and documentation
   1. Provide quality checks to detect/report data abnormalities
   2. Document processes and applications
   3. Contribute to codebase and documentation (Confluence)
   4. Communicate changes and feedback for downstream activities
   5. Create best practice instructions
   6. Implement, and enforce best practices

Requirements:

1. At least two or three years (various levels of position) in data management, including API and ETL development, data pipeline designs, data warehousing
2. Strong skills in Python ETL. Preferably experience in PySpark working with big data.
3. Data transformation techniques appropriate for real-time streaming data
4. ANSI SQL
5. Linux environment programming
6. Familiarity with modern data architecture, data collection methods, data warehousing, database management, and tools
7. Education in computer science, industrial engineering, information technology

Tools:

1. Python
2. Spark
3. Databricks
4. SQL Server, including SSIS, MySQL, MSSQL
5. Big Query, Github, OLAP (SSAS)
6. Redshift

Others:

1. Demonstrate a career of learning and creativity
2. Techniques that show high focus, organization, communication, and presentation skills