AI Evaluations Research Associate

Full-time or part-time

Overview

RAND's Technology and Security Policy Center (TASP) is seeking mission-driven AI Evaluations Research Associates to develop and execute research projects and engineering efforts within our AI Capability Evaluations (ACE) team.

RAND's reputation for excellence is built on our commitment to high-quality, rigorous analysis and objectivity. TASP is at the forefront of research and implementation regarding the impact of high-consequence, dual-use technologies-such as artificial intelligence and biotechnology-on global competition and security. Our research has been used by the White House, U.S. government departments and agencies, the EU and UK governments, and industry leaders, among others. Our alumni have gone on to important roles at the NSC, Commerce, DOD, Congress, Google DeepMind, OpenAI, EU AI Office, UK AISI, other key think tanks, and founding mission-driven tech initiatives.

ACE develops and conducts evaluations of national security relevant capabilities of frontier AI systems, with a current focus on the intersection of large language models (LLMs) and AI agents with biological risk. We're hiring for people with research science and/or research engineering skills to play a key role in work that assists public policymakers at all levels in strengthening national security and mitigating catastrophic risks enabled by AI systems. They will work on complex problems at the intersection of AI and national security where technical details matter and will contribute to multidisciplinary project teams that include biosecurity experts, machine learning engineers, and policy researchers.

This position is initially structured as a focused 1-year appointment to create the urgency needed to drive ambitious change in this rapidly evolving field. Every day of your tenure will count toward that goal. The appointment may be renewed for up to a total of 3 years, with options for longer-term employment at RAND thereafter. Full-time and part-time (at least 20 hours per week) schedules will be considered, but with a strong preference for full-time.

Responsibilities

Given the breadth of valuable work our team could do, there is some ability to align responsibilities with an individual's skills, interests, and career goals, including in terms of the balance of research scientist- versus research engineer-style responsibilities. Responsibilities may include but are not limited to:

• Contribute to developing concrete threat models for high-consequence risks AI risks, working with internal and external partners

• Contribute to designing and executing rigorous, objective evaluations of AI capabilities relevant to key bottlenecks within those threat models

• Contribute to developing and maintaining the technical infrastructure required to support this research, working with relevant internal and external IT stakeholders

• Develop and maintain code for fundamental evaluation components that can be used across research efforts (e.g. prompting, automated grading, statistical analysis)

• Keep up to date with the latest advances in AI evaluation engineering and the science of evaluations to continually improve the rigor and efficiency of our evaluations

• Contribute to communicating research results to policymakers and other key stakeholders at all levels through written products and oral presentations

• Contribute to setting strategic and research priorities, with an emphasis on the policy impact of evaluations

Qualifications
All research positions at RAND require excellent analytic skills; the ability to communicate clearly and effectively in English, both orally and in writing; the ability to work effectively as a member of a multi-disciplinary team; and a strong commitment to RAND's core values of quality and objectivity.

Other required qualifications:

• Strong interest in understanding and addressing potential national security risks related to autonomy or high-consequence misuse of LLMs and AI agents, and in AI capability evaluations as a route to impact

• Proficiency in Python

• Familiarity with technical aspects of AI systems and related technologies, such as machine learning, computational infrastructure, or information security

Preferred but not required:

• Experience with evaluations and evaluation frameworks for LLMs and AI agents (e.g. Inspect)

• Experience with LLM elicitation techniques (e.g. fine-tuning, retrieval augmented generation, tool-use integration, agent scaffolding)

• Experience working on ML model development/deployment or working at/with leading AI companies

• Experience with cloud computing, in particular Azure and AWS, including government cloud environments

• Familiarity with common LLM frameworks (e.g. LangChain, LlamaIndex)

• Aptitude for project management

• Experience in government, intelligence community, other relevant decision-making offices, or policy analysis roles

Applicants who meet the required qualifications are strongly encouraged to apply; the preferred qualifications are not necessary.

Education Requirements

This role requires at least a Bachelor's degree in a relevant discipline. Higher levels of education (e.g. PhD, JD, Master's) or experience are preferred. Relevant disciplines may include Artificial Intelligence, Machine Learning, Computer Science, Cybersecurity, Electrical Engineering, Physics, Mathematics, Engineering and Public Policy, Security Studies, or similar.

Note: This team also has an opportunity for a Research Scientist. Applicants that possess the required qualifications are welcome to apply to either or both positions.

Security Clearance
Ability to obtain and maintain a U.S. security clearance, including having US citizenship, is preferred but not required.

Location

We are actively hiring for this position in Washington, DC; San Francisco, CA; Boston, MA; Santa Monica, CA; and Pittsburgh, PA. San Francisco or especially DC are preferred. We offer a hybrid work arrangement, combining work from home and on-site options. Fully remote work will also be considered.

Term

This position is a 1-year term appointment with a possibility of renewal for up to 3 years total, alongside options for longer term employment.

Application

Applications must include:

• A detailed resume highlighting relevant academic and professional experience.

• A writing sample demonstrating analytical and communication skills. This sample may be a recent, previously written paper or report (e.g., journal article, master's thesis or paper written for coursework, prior employment, or internship). Applicants whose study and work experience (e.g., model development) has not involved producing written products that are shareable may submit a short, written summary (i.e., less than one page) of one or more recent products they have developed.

• A code sample. You can link to this from the bottom of your cover letter.

• A cover letter which contains only responses to each of the following prompts:

• 1) Summarize in <200 words your career goals and why you are interested in this role.

• 2) Describe in <300 words one research direction or engineering infrastructure project you may want to pursue in this role. For a research direction: Describe what questions you would try to answer, what methods you would use, how many months of work would be required from you and/or colleagues, and what outcomes this research might help achieve (e.g., what important policy decisions it might inform). For an infrastructure project: You may make guesses about our goals and existing infrastructure, and propose a way you might help improve that, noting how you would implement that, how many months of work may be required from you and/or colleagues, and why this might be useful. This is just an assessment step and does not mean you would definitely work on this if hired.

Salary Range: $47,100- $156,500

In RAND's title system, this posting is for the following titles and corresponding salary ranges: