Description:
We are seeking an experienced **Python - PySpark Developer** with **Jupyter Notebook** expertise to join our dynamic team in **Dubai, UAE**. The ideal candidate should have at least 10 years of experience in software development, with a strong focus on big data technologies and Python programming.
### **Key Responsibilities**:
- Design, develop, and maintain large-scale **ETL pipelines** using **PySpark** and **Python**.
- Develop, test, and maintain efficient, reusable, and reliable code in **Jupyter Notebooks** for data analysis and visualization.
- Collaborate with data scientists and analysts to support their data processing and analysis needs.
- Optimize performance of PySpark jobs and workflows to handle large datasets.
- Ensure data accuracy and integrity by implementing and maintaining robust data validation procedures.
- Write well-designed, efficient, and testable code following best practices.
- Work with cross-functional teams to understand and translate business requirements into technical solutions.
- Perform troubleshooting, debugging, and performance tuning of applications.
- Stay updated with the latest trends and technologies in big data and cloud environments.
### **Key Requirements**:
- Minimum 10+ years of experience in software development, with expertise in **Python** and **PySpark**.
- Extensive experience with **Jupyter Notebooks** for data analysis and development.
- Proven experience in developing and optimizing large-scale **data processing pipelines**.
- Strong knowledge of **distributed computing** concepts and experience with **Apache Spark**.
- Familiarity with **cloud platforms** like **AWS**, **Azure**, or **Google Cloud**.
- Proficiency in working with **Hadoop**, **HDFS**, and other big data technologies is a plus.
- Strong problem-solving skills with a proactive and results-driven approach.
- Excellent communication skills and the ability to work in a fast-paced environment.