Level 2 Analyst in AI Safety and Evaluation

At Pieces, one of our core value statements is that our customer’s mission is a sacred mission—and it's a hard mission. Our goal is to apply advanced AI to make the work of our customers—patient care—easier in as many ways as possible. We use cutting-edge approaches in AI, such as natural language processing, natural language generation, neural networks, and human-in-the-loop AI, to help healthcare providers reduce documentation, improve communication, manage clinical risk, address social determinants of health, and improve patient outcomes.

Responsibilities

Pieces believes that AI for use in patient care will require rigorous supervision and oversight for the foreseeable future through multiple lenses, including health services research and clinical effectiveness evaluation. The Level 2 Analyst in AI Safety and Evaluation will support the evaluation of our AI technologies and performance, contribute to our AI safety strategy, and assist in ensuring our safety policies and practices align with the highest standards. The Analyst will also drive data analytics in support of the Company's customers and products.

Duties and responsibilities include but are not limited to:

• Assist in articulating and communicating principles for building and deploying responsible and safe AI across the company

• Support safety strategy for the design, development, and deployment of Pieces Predict product

• Contribute to client-facing and internal analytic reporting around predictive model and clinical effectiveness

• Conduct clinical effectiveness analyses on the company’s products

• Collaborate with Product, AI, Sales, Client Success, Implementation, Engineering, Legal, and clinical experts to meet safety and risk management objectives

• Stay updated on regulations and guidelines around responsible AI, especially in healthcare

• Develop and perform analyses on metrics—including business value KPIs and AI/ML technical metrics—to measure product effectiveness and safety

• Provide analytic support and related insights for the AI team, helping strategize model changes and validate updates in staging and production environments

Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.

Requirements

We’re looking for a Level 2 Analyst in AI Safety and Evaluation who is passionate about using data in a safe and effective way to solve some of the most critical issues standing in the way of healthcare providers pursuing their sacred missions in patient care. You should have expertise in the following areas:

Required:

• PhD in Health Services Research, Biostatistics, Public Health, Health Economics or equivalent

• Peer-reviewed 1st authored research or manuscripts that demonstrate expertise in rigorous quantitative methods and ability to conduct health and safety evaluation research

• Fluency in Python for data analysis and modeling

• Experience with incorporating all the major LLMs into programming approaches and into actual code

• 0-2 years of experience in healthcare, healthcare technology, or healthcare research environment (recent graduates encouraged to apply)

Preferred:

• Knowledge and experience in clinical risk adjustment or predictive models

• Experience working with EHR data

• Strong work ethic and adaptability to succeed in a fast-growth environment

• Ability to present technical information clearly to different audiences, including customers, peers, and team members

• Intellectual curiosity and persistence, with a strong drive to learn from experts

• Understanding of clinical environments and workflows is a plus, but not required

Benefits

We offer competitive salaries as well as the following benefits:

• Shared premiums: medical, dental, and vision, plus access to FSA and HSA

• 401K and life insurance

• Flexible vacation, holiday, and remote work policies

• Flexible work hours, with work-from-home options

• Paid maternity/paternity leave