Requirements
The Big Data Engineer is responsible for collecting, storing, processing, and analyzing large sets of data. The primary focus is on selecting optimal solutions for these tasks, as well as maintaining, implementing, and monitoring them. The engineer will also be responsible for integrating these solutions with the company's broader architecture.
Overall Responsibilities:
• Data analysis and preparation
• Coding
• Code testing
• Adherence to Agile practices
Detailed Responsibilities:
• Gather and process raw data at scale.
• Design and develop data applications using selected tools and frameworks.
• Read, extract, transform, stage, and load data using various tools.
• Execute tasks such as writing scripts, web scraping, calling APIs, and writing SQL queries.
• Process unstructured data for analysis.
• Analyze processed data and support business decisions with ad hoc analysis.
• Monitor data performance and infrastructure, modifying as needed.
• Define data retention policies.
• Collaborate with the engineering team to integrate work into production systems.
• Share knowledge to reduce bottlenecks and surface problems that slow the team down.
• Cooperate with team members to define rules and processes that ensure team coherence.
• Participate actively in Agile events such as daily scrum, planning, and retrospectives.
Requirements:
• Bachelor’s degree in Engineering, Information Technology, Computer Science, or a related field.
• At least 3 years of relevant experience.
• Strong knowledge of and experience with statistics.
• Programming experience (Python, Kafka, or Java) and a willingness to learn new languages.
• Proficiency with Hadoop v2, MapReduce, HDFS.
• Good knowledge of Big Data querying tools.
• Experience with Spark.
• Experience processing structured and unstructured data.
• Experience with NoSQL databases (Cassandra, MongoDB).
• Familiarity with messaging systems (Kafka, RabbitMQ).
