Senior Software Engineer, AI Benchmarking

About this role

SENIOR SOFTWARE ENGINEER, AI BENCHMARKING

- Project/Division: SecureBio AI Team

- Start date: As soon as possible

- Status: Full-time

- Work location: Remote, with office locations in Cambridge, MA and Berkeley, CA

- Reports to: Head of Engineering

THE ROLE

SecureBio is looking for a Senior Software Engineer to build, scale, and run biosecurity evaluations on frontier AI systems. The tools you build will directly inform decisions made by frontier AI companies, policymakers, and researchers to improve the safety of AI systems.

Our biosecurity evaluations are regularly used to audit state-of-the-art models for biological capabilities and dual-use biology risks prior to their public release. Our work has been publicly cited in model cards from OpenAI, Anthropic, and Google, and has informed safeguards on frontier models. We are a member of the NIST US AI Safety Consortium, and a partner to the European AI Office.

RESPONSIBILITIES

- Perform pre-release and post-release assessments of frontier and open-source AI models using SecureBio's suite of biosecurity evaluations, which surface and measure dual-use biology capabilities in AI systems. - Build adapters for models, identify and troubleshoot unexpected agent behaviors, and ensure that our evaluations are methodologically rigorous.

- Build, scale, maintain, and continuously improve SecureBio's suite of biosecurity evaluations. - Improve our evaluations to use state-of-the-art agent frameworks, jailbreaking methods, and elicitation strategies.

- Develop and improve internal tools that enable our researchers to run capability assessments at scale.

- Contribute to and maintain cloud infrastructure to run our evaluations and analyses.

- Contribute analyses to our public dashboard of trends in AI biology capabilities (https://securebio.org/benchmarks/).

QUALIFICATIONS

A STRONG CANDIDATE WOULD HAVE THE FOLLOWING QUALIFICATIONS:

- A proven track record in software engineering, including 5+ years of experience in industry roles and/or open-source contributions.

- Fluency in Python.

- Proficiency with Docker and AWS (particularly ECS).

- Experience building software tools that use or evaluate LLMs or AI agents (for example, code that orchestrates agents for a specific application, or that evaluates model capabilities on a specific task).

- Self-directed and comfortable working on small, fast-moving teams.

- A commitment to good coding practices, including knowing when and how to use (or not use) coding agents.

- Care about code quality, readability, and maintaining human oversight of human- and AI-written code.

- Alignment with SecureBio's mission of preventing catastrophic pandemics.

PREFERRED:

None are necessary, but any individual one is a substantial plus.

- Experience with AI evaluations (agent systems, scorer and grader design, agent trajectory analysis, jailbreaking or red-teaming), whether for security, safety, or model performance applications.

- Experience with Terraform or other Infrastructure-As-Code tools (e.g. Tofu or AWS CDK).

- Experience with modern, component-based web frontends (e.g. React with TypeScript).

- Experience with our specific tech stack: - Evaluation frameworks: Inspect AI - LLM APIs: OpenAI-compatible, Anthropic, Together, etc. - Agent frameworks: Codex, Claude Code, etc.

ABOUT SECUREBIO

In joining SecureBio, you'll be joining a motivated, mission-driven, and international team of professionals dedicated to protecting the world from biological threats. Through in-house research & development, as well as collaborations with experts in academia, industry & government, we are working to build the tools, technologies, and institutions the world needs to be secure against future biotechnology.

Major projects by the SecureBio team include the SecureBio Detection https://securebio.org/detection/ project for reliable pandemic detection and our work to understand and reduce biological risks from advanced artificial intelligence https://securebio.org/ai/.

APPLICATION PROCESS

We will review applications on a rolling basis, and are looking for someone to start as soon as possible.

After considering your written application, we may ask you to complete a short coding test, and invite you to a short screening interview. High-performing candidates may be invited to further interviews and/or paid trial tasks. We will also require up to two reference checks.

We're committed to fostering a culture of inclusion, and we encourage individuals with diverse backgrounds and experiences to apply. We especially encourage applications from women, gender minorities, and people of color who are excited about contributing to our mission, and are happy to consider applicants from overseas. We're an equal opportunity employer.

If you need assistance or an accommodation due to a disability, or have any other questions about applying, please contact careers@securebio.org.

Similar jobs

• Applied AI Researcher, Benchmarking — Distyl · San Francisco
• Senior Software Engineer, AI Infrastructure — Thealleninstitute · Seattle, WA
• Senior Software Engineer, AI — Centari · NYC Office
• Senior Software Engineer, AI — August-health · Remote
• Senior Software Engineer, AI — Tripactions · New York, NY
• Senior Software Engineer, AI — Okta · Bellevue, Washington
• Senior Software Engineer, AI — Okta · Chicago, Illinois
• Senior Software Engineer, AI — Okta · San Francisco, California