Offer summary

Qualifications:

Bachelor's Degree in Computer Science or related field., Minimum of 5 years of experience in tech lead data engineering or architecture roles., Proficiency in Python and PySpark for ETL development and data processing., Practical experience with performance optimization of Spark jobs and knowledge of SOLID and OOD principles..

Key responsabilities:

Design, develop, and maintain ETL pipelines for data ingestion and transformation.

Collaborate with cross-functional teams for seamless deployment and integration of data solutions.

Lead performance tuning and query optimization efforts to enhance data processing efficiency.

Contribute to the development of best practices and standards for data engineering processes.

Commit is a global tech services company with offices in New York, Israel, and Europe. The company was founded in 2005 and has over 700 multi-disciplinary innovation experts who serve a broad range of companies, from small startups to large enterprises in multiple business sectors. Commit specializes in advanced technologies and applications with dedicated practices in Software, IoT, Big Data, Cloud, Cyber, Collaboration, Data center migration projects, and more. Commit offers innovative, end-to-end technology solutions by developing custom software and IoT platforms for clients looking to build their next-gen products within the modern ICT world. Commit’s complete and comprehensive engineering powerhouse of resources, and proprietary Flexible R&D methodology helps transform its clients’ technology visions into high-quality products while reducing costs and improving time-to-market.

Job description

Description

As a Senior/Tech Lead Data Engineer, you will play a pivotal role in designing, implementing, and optimizing data platforms for our clients. Your primary responsibilities will revolve around data modeling, ETL development, and platform optimization, leveraging technologies such as EMR/Glue, Air Flow, Spark, using Python and various cloud-based solutions.

Key Responsibilities:

Design, develop, and maintain ETL pipelines for ingesting and transforming data from diverse sources.
Collaborate with cross-functional teams to ensure seamless deployment and integration of data solutions.
Lead efforts in performance tuning and query optimization to enhance data processing efficiency.
Provide expertise in data modeling and database design to ensure scalability and reliability of data platforms.
Contribute to the development of best practices and standards for data engineering processes.
Stay updated on emerging technologies and trends in the data engineering landscape.

Requirements

Bachelor's Degree in Computer Science or related field.
Minimum of 5 years of experience in tech lead data engineering or architecture roles.
Proficiency in Python and PySpark for ETL development and data processing.
AWS CLOUD at least 2 years
Extensive experience with cloud-based data platforms, particularly EMR.
Must have knowledge with Spark.
Excellent problem-solving skills and ability to work effectively in a collaborative team environment.
Leadership experience, with a proven track record of leading data engineering teams.
The first point - it would be great if the candidate would confirm, that he/she is:
Practical experience with performance optimization of the Spark job (MUST)
Knowing what is SOLID and knowledge of OOD (MUST)

Required profile