Sr. Staff Software Engineering

Overview:

DDN Storage is seeking great candidates to join our dynamic team of passionate customer-enabling technologists!

This is an incredible opportunity to be part of a company that has been at the forefront of AI and high-performance data storage innovation for over two decades. DDN Storage is a global market leader renowned for powering many of the world's most demanding AI data centers, in industries ranging from life sciences and healthcare to financial services, autonomous cars, Government, academia, research and manufacturing.

"DDN's A3I solutions are transforming the landscape of AI infrastructure." – IDC

“The real differentiator is DDN. I never hesitate to recommend DDN. DDN is the de facto name for AI Storage in high performance environments” - ~ Marc Hamilton VP, Solutions Architecture & Engineering | NVIDIA

DDN Storage is the global leader in AI and multi-cloud data management at scale. Our cutting-edge data intelligence platform is designed to accelerate AI workloads, enabling organizations to extract maximum value from their data. With a proven track record of performance, reliability, and scalability, DDN Storage empowers businesses to tackle the most challenging AI and data-intensive workloads with confidence. 

Our success is driven by our unwavering commitment to innovation, customer-centricity, and a team of passionate professionals who bring their expertise and dedication to every project. This is a chance to make a significant impact at a company that is shaping the future of AI and data management. 

Our commitment to innovation, customer success, and market leadership makes this an exciting and rewarding role for a driven professional looking to make a lasting impact in the world of AI and data storage. 

We are seeking an experienced Sr. Staff Software Engineering – Spark ecosystem specializing in Apache Spark ecosystem core development and distributed computing systems. In this role, you will focus on the implementation of Apache Spark connectors to the highest-performing data system in the world, DDN Infinia, working to improve its performance, scalability, and reliability. You will collaborate with a team of engineers to drive innovation in large-scale data processing frameworks and play a key role in advancing DDN’s Intelligent Data engineering solution.

This is a unique opportunity for engineers passionate about big data processing and high-performance distributed systems.

Key Responsibilities:

• Contribute to the development of Apache Spark ecosystem connectors to Infinia highly-distributed storage system, optimizing for performance, scalability, and resource efficiency.
• Design, implement, and improve distributed execution engine, focusing on low-latency and high-throughput data retrieval and processing.
• Troubleshoot and resolve complex issues related to distributed computing, fault tolerance, and data parallelism.
• Ensure robustness and efficiency in the handling of large-scale distributed workloads.
• Work with cross-functional teams including platform and high-performance engineers, and other stakeholders to ensure Spark integrates seamlessly with the broader ecosystem.
• Write and maintain clear, comprehensive documentation for internal systems and the broader open-source community.
• Conduct code reviews to maintain high-quality standards across the team.
• Participate in technical discussions, design reviews, and provide expertise in distributed data processing.

Requirements:

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• 12+ years of software engineering experience with a focus on distributed systems and large-scale data processing.
• In-depth knowledge of Apache Spark's architecture and components such as RDDs, DAGScheduler, TaskScheduler, and execution engines.
• Strong proficiency in Java, with significant experience in distributed system development.
• Experience with core Hadoop components and related big data technologies (HDFS, YARN HBase).
• Proficiency in solving low-level performance challenges in distributed computing, including task parallelism, memory management, and network optimization.
• Solid understanding of data partitioning, shuffling, fault-tolerance mechanisms, and optimization techniques in large, distributed clusters.
• Knowledge of distributed algorithms and techniques such as MapReduce and stream processing.
• Proven ability to debug and profile large-scale systems, identifying and resolving bottlenecks.

Preferred Qualifications:

• Experience with Kubernetes or container orchestration in a cloud environment.
• Proficiency in storage, distributed database systems, and high-availability architectures.
• Hands-on experience with performance tuning, profiling, and benchmarking large data pipelines.

Our team is highly motivated and focused on engineering excellence.

We look for individuals who appreciate challenging themselves and thrive on curiosity.

Engineers are encouraged to work across multiple areas of the company.

We operate with a flat organizational structure.

All employees are expected to be hands-on and to contribute directly to the company’s mission.

Leadership is given to those who show initiative and consistently deliver excellence. Work ethic and strong prioritization skills are important.

All engineers and researchers are expected to have strong communication skills.

They should be able to concisely and accurately share knowledge with their teammates.

Interview Process: After submitting your application, the team reviews your CV and statement of exceptional work. If your application passes this stage, you will be invited to a 30-minute interview (“phone interview”) during which a member of our team will ask some basic questions. If you clear the initial phone interview, you will enter the main process, which consists of four technical interviews:

• Coding assessment in a language of your choice.
• Systems design: Translate high-level requirements into a scalable, fault-tolerant service.
• Systems hands-on: Demonstrate practical skills in a live problem-solving session.
• Project deep-dive: Present your past exceptional work to a small audience.
• Meet and greet with the wider team.
• Our goal is to finish the main process within one week.
• We don’t rely on recruiters for assessments.
• Every application is reviewed by a member of our technical team.

DataDirect Networks, Inc. is an Equal Opportunity/Affirmative Action employer.  All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity, gender expression, transgender, sex stereotyping, sexual orientation, national origin, disability, protected Veteran Status, or any other characteristic protected by applicable federal, state, or local law.

#LI-Remote