Match score not available

Azure Data Engineer

Remote:

Full Remote

Contract:

Full time

Experience:

Mid-level (2-5 years)

Work from:

South Africa

Offer summary

Qualifications:

Bachelor's degree in Computer Science or related field., 5+ years experience with Azure Databricks., Proficient in PySpark and Spark SQL., Understanding of Azure data services., Experience with ETL, data modeling and architecture..

Key responsabilities:

Design scalable data pipelines on Azure Databricks.
Develop and optimize ETL processes.
Collaborate with stakeholders for data solutions.
Ensure quality, performance, and scalability.
Monitor and troubleshoot data pipelines.

Advancio SME https://www.advancio.com/

51 - 200 Employees

See more Advancio offers

Job description

This is a remote position.

Job Overview:
We are seeking a highly skilled Azure Data Engineer with extensive experience in Databricks, PySpark, and Spark SQL to join our team. The ideal candidate will be responsible for designing, developing, and maintaining large-scale data solutions on Azure, with a focus on real-time and batch data processing using Databricks.

Requirements

Key Responsibilities:

Design, build, and maintain scalable data pipelines on Azure Databricks using PySpark and Spark SQL.
Develop and optimize ETL processes for handling large data sets.
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and implement solutions.
Ensure data quality, performance, and scalability by implementing best practices for coding and data architecture.
Create and manage data models, databases, and data lakes on Azure.
Monitor and troubleshoot data pipelines, ensuring high availability and reliability.
Implement security and compliance best practices in the data pipeline, following Azure standards.
Work with Azure services like Azure Data Factory, Azure Synapse, and Azure Blob Storage to orchestrate and manage data workflows.

Qualifications:

Bachelor's degree in Computer Science, Information Technology, or a related field.
5+ years of hands-on experience with Azure Databricks.
Proficiency in PySpark and Spark SQL for building scalable data pipelines.
Solid understanding of Azure data ecosystem, including Azure Data Lake, Azure Data Factory, and Azure Synapse Analytics.
Experience with ETL processes, data modeling, and data architecture.
Familiarity with cloud security best practices and data governance.
Strong problem-solving and troubleshooting skills.
Excellent communication and collaboration skills.