Forward Deployed Engineer - SRE

About this role

FORWARD DEPLOYED ENGINEER - SRE

LOCATION: NORTH AMERICA REMOTE/SF-HYBRID · FULL-TIME

About Andromeda

Andromeda Cluster was founded by Nat Friedman and Daniel Gross to give early-stage startups access to the kind of scaled AI infrastructure once reserved only for hyperscalers.

We began with a single managed cluster — but it filled almost instantly. Since then, we’ve been quietly building the systems, network, and orchestration layer that makes the world’s AI infrastructure more accessible.

Today, Andromeda works with leading AI labs, data centers, and cloud providers to deliver compute when and where it’s needed most. Our platform routes training and inference jobs across global supply, unlocking flexibility and efficiency in one of the fastest-growing markets on earth.

Our long-term vision is to build the liquidity layer for global AI compute. We are expanding to new frontiers to find the brightest that work in AI infrastructure, research and engineering.

The Role

This is not a generalist SRE role, and it is not a support role. You will embed directly with the teams running large-scale training and inference on our clusters. You are responsible for onboarding them, tuning their jobs, and debugging their failures alongside them, while owning the infrastructure and automation that makes those clusters reliable in the first place.

Forward deployed means you spend real time inside customer environments: reading their training code, sitting in their Slack channels, watching their runs, and shipping fixes that land in our platform. When a multi-hundred-GPU run stalls, you are the person who figures out whether it's the fabric, the driver, the scheduler, or their dataloader, then you make sure it can't happen the same way twice.

We're looking for engineers who have personally run GPU clusters in production, understand the failure modes of distributed training, and can reason about performance from network fabric → kernel → framework. Equally important: you can explain what you found to someone else's engineering team without condescension, and turn that conversation into a product improvement.

What You’ll Do

- Serve as the primary technical point of contact for teams running large-scale training and inference workloads. Own onboarding end to end; environment setup, orchestration choice (Slurm, Kubernetes, or direct SSH), storage layout, first successful run at scale. You will continue to stay engaged as their workloads grow.

- Work inside customer environments to diagnose real failures: NCCL timeouts, stragglers, checkpoint I/O stalls, degraded links, OOM patterns, container and driver mismatches. Read their code when you need to. Reproduce, isolate, fix, and write it down.

- Profile and improve distributed training performance on live workloads. Improving MFU, cutting idle GPU time, and reducing time-to-first-successful-run for new deployments.

- Own reliability outcomes for the accounts you're deployed on.

- Ensure the health and performance of high-speed interconnects (InfiniBand, RoCE, NVLink) that underpin distributed training. Diagnose and resolve fabric-level issues that degrade collective operations.

- Build deep visibility into GPU utilization, memory pressure, interconnect throughput, job performance, and hardware health.

- Turn every repeated deployment problem into automation: cluster provisioning, GPU health checks and burn-in, preflight validation, self-healing, firmware/driver lifecycle management, and reusable reference configurations for common training and serving stacks.

- Lead incident response for complex, multi-layer failures spanning hardware, networking, orchestration, and ML frameworks. Own the customer-facing communication during the incident and the blameless postmortem and systemic fix after it.

- You will see our rough edges before anyone else does. Bring that signal back to influence the roadmap, file the hard bugs, and build the missing pieces yourself when that's the fastest path.

What We’re Looking For

- Hands-on experience operating GPU clusters in production (NVIDIA A100/H100/H200/B200 or equivalent). You understand GPU memory hierarchies, ECC behavior, thermal throttling, and hardware failure modes from direct experience.

- Production experience with InfiniBand, RoCE, or NVLink fabrics in the context of distributed training. You can diagnose why an all-reduce is slow, identify a degraded link in a fat-tree topology, and reason about congestion control at scale.

- Working knowledge of how large training and inference jobs actually run. You don't need to design models, but you need to understand what's happening at the systems level when a large run stalls.

- Expert-level Linux experience: kernel tuning, driver management (NVIDIA drivers, CUDA toolkit), cgroup/namespace internals, container runtimes, and performance profiling at the syscall and hardware level.

- Strong experience running Kubernetes in production with GPU workloads. Experience with device plugins, topology-aware scheduling, multi-cluster, custom operators. Experience with Slurm or other HPC schedulers is equally valued.

- Strong engineering skills in Python, Go, or Bash. You build production-grade tools and services, not just scripts.

- Infrastructure-as-Code proficiency (Terraform, Helm, Ansible, or equivalent).

- Hands-on experience building monitoring and alerting for GPU-specific telemetry (DCGM, nvidia-smi, fabric manager metrics) integrated into actionable dashboards.

- You can go deep on architecture with a customer's infra team and clearly articulate tradeoffs to their leadership. You're comfortable being the only one in the room who knows the answer, and equally comfortable saying you don't yet.

- Proven track record leading incident response for complex distributed systems.

Strong Candidates May Have

- Experience with high-performance parallel file systems (VAST, WEKA, Lustre, GPFS) and the checkpoint I/O a

Similar jobs

• Forward Deployed Engineer - Software Engineer — Elevenlabs · Remote
• Forward Deployed Security Engineer — Openai · Washington, DC
• Forward Deployed Software Engineer — Palantir · New York, NY
• Forward Deployed Software Engineer — Sea12 · New York City
• Forward Deployed Software Engineer — Candidhealth · San Francisco
• Forward Deployed Software Engineer — Northslope-technologies · New York
• Forward Deployed Software Engineer — Pareto-ai · Remote
• Forward Deployed Engineer (Senior) — Infinitus · California