We are seeking an experienced Query Engine Architect to accelerate Apache Spark and related frameworks on GPUs. As Nvidia is leading the world in accelerated computing, we are building the next generation data processing ecosystem. Apache Spark is the most popular distributed data processing engine in data centers. It is used for a wide variety of workloads, from data preparation, feature generation, reporting, analytics, and more. Data scientists spend a considerable amount of time exploring data and iterating over machine learning (ML) experiments. Every hour of compute required to sort through datasets, extract features and fit ML algorithms impedes an efficient business workflow.
At NVIDIA, we are passionate about working on hard problems that have an impact. You will work with the open source community to enable Apache Spark data processing with GPUs. Data workflows can benefit tremendously from being accelerated, enabling data scientists to explore many more and larger datasets to achieve their business goals, faster and more efficiently.
What you'll be doing:
Lead the query optimization effort on the RAPIDS Spark team.
Review each stage of query processing, and identify areas for logical plan and physical plan optimization. Construct optimization of plans taking into account CPU / GPU hardware resources.
Find opportunities for adaptive query execution that are resource aware, for example, adapting based on CPU or GPU characteristics
Identify where operator fusion could drive better performance
Review columnar processing engine practices and see how they might apply to GPU based columnar processing
Engage open source communities, including Apache Spark and RAPIDS, for technical discussions and contributions
Work with Nvidia strategic partners on deploying accelerated data processing solutions in public cloud or on-premise clusters
Present technical solutions in industry conferences and meetups
What we need to see:
BS, MS, or PhD in Computer Science, Computer Engineering, or equivalent experience
15+ years of work or research experience in software development
5+ years working with key open source big-data projects as a contributor or committer including Apache Spark, Apache Hadoop, Apache Hive, Apache Flink, Apache Impala, Apache Drill, Apache Calcite, and Substrait
Outstanding technical skills in crafting and implementing high-quality distributed systems
Deep expertise in database query engines and query optimization
Excellent programming skills in C++, Java, and/or Scala
Knowledge of distributed system schedulers: Kubernetes, Hadoop YARN, Spark standalone, and/or Mesos
Ability to work with multi-functional teams across boundaries and geographies
Highly motivated with strong interpersonal skills
Ways to stand out from the crowd:
Contributions to major open source projects such as Apache Spark, Apache Hive, Apache Impala, Apache Drill, Substrait, Apache Calcite.
Working experience with acceleration libraries (CUDA, RAPIDS, UCX)
Basic ML/DL experience with Spark ML and XGBoost
We are widely considered to be one of the technology world’s most desirable employers, and as a result have some of the most forward-thinking and hardworking people in the world working for us. If you're passionate, creative, and driven, we'd love to have you join the team. With competitive salaries and a generous benefits package, we are widely considered to be one of the technology world’s most desirable employers. We have some of the most forward-thinking and hardworking people in the world working for us and, due to unprecedented growth, our exclusive engineering teams are rapidly growing. If you're a creative and autonomous engineer with a real passion for technology, we want to hear from you.
The base salary range is 272,000 USD - 419,750 USD. Your base salary will be determined based on your location, experience, and the pay of employees in similar positions.
You will also be eligible for equity and benefits. NVIDIA accepts applications on an ongoing basis.
NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.
