Job description

Job Title: Python Developer with PySpark Expertise

Location: Remote

Experience: 4-6 Years

Employment Type: Full-time

Job Description:

We are looking for a skilled Python Developer with PySpark expertise to join our team. The ideal candidate should have a strong foundation in Python, Big Data Processing, and Spark (PySpark) and be experienced in handling large-scale data processing workflows.

Key Responsibilities:

Develop, optimize, and maintain data processing pipelines using PySpark and Python.
Work with large datasets and ensure efficient data transformation, cleaning, and processing.
Design and implement ETL workflows for structured and unstructured data.
Collaborate with data engineers, data scientists, and cloud architects to build scalable data solutions.
Optimize Spark jobs for performance and scalability on distributed systems.
Integrate with cloud platforms like AWS, Azure, or GCP for data storage and processing.
Troubleshoot and resolve issues related to Spark job execution and cluster performance.

Required Skills & Qualifications:

3-5 years of experience in Python development with expertise in PySpark.
Strong understanding of Spark architecture, RDDs, DataFrames, and Spark SQL.
Hands-on experience with big data technologies such as Hadoop, HDFS, Hive, Kafka, etc.
Experience in writing optimized and efficient PySpark scripts for batch and real-time data processing.
Strong knowledge of SQL and database technologies (PostgreSQL, MySQL, NoSQL, etc.).
Experience with cloud services (AWS, Azure, or GCP) and data lake/data warehouse concepts.
Familiarity with CI/CD pipelines and version control (Git, Bitbucket, etc.).
Knowledge of containerization tools like Docker/Kubernetes is a plus.

Preferred Qualifications: