Responsibilities:

• Perform site selection (if applicable), initiation, monitoring and close-out visits in accordance with contracted scope of work and good clinical practice. If applicable, Clinical Research Associates may be accountable for development of project subject recruitment plan on a per site basis. Support work with sites to adapt, drive and track subject recruitment plan in line with project needs to enhance predictability.
• Provide monitoring visits and site management for a variety of protocols, sites and therapeutic areas.
• Administer protocol and related study training to assigned sites and establish regular lines of communication with sites to manage ongoing project expectations and issues. Evaluate the quality and integrity of study site practices related to the proper conduct of the protocol and adherence to applicable regulations. Escalate quality issues as appropriate.
• Manage the progress of assigned studies by tracking regulatory submissions and approvals, recruitment and enrollment, case report form (CRF) completion and submission, and data query generation and resolution. May support start-up phase.
• Create and maintain appropriate documentation regarding site management, monitoring visit findings and action plans by submitting regular visit reports, generating follow-up letters and other required study documentation.
• Build awareness of features and opportunities of study to site.
• Collaborate and liaise with study team members for project execution support as appropriate.

Requirements:

• Bachelor's degree in a health care or other scientific discipline or educational equivalent.
• Minimum of 1+ years of on-site monitoring experience; or equivalent combination of education, training and experience.
• Computer skills including proficiency in use of Microsoft Word, Excel and PowerPoint and use of a laptop computer
• Good written and verbal communication skills
• Good organizational and problem-solving skills
• Effective time management skills

Job Responsibilities:

• Work with technical and non-technical stakeholders to design and execute research testing how humans evaluate large language model (LLM) features.
• Conduct research with humans to validate and create automated benchmarks for LLM evaluations.

Requirements:

• PhD or advanced Degree in Computer Science, Machine Learning, Cognitive Science, Psychology, Economics, or similar discipline.
• Minimum 2 years of relevant experience in large language model research.
• Fluent in at least one statistical programming language such as Python or R.
• Strong understanding of machine learning principles, especially in the context of LLMs.
• Knowledgeable about LLM evaluation techniques, such as human evaluation and automated benchmarks.
• Able to own and pursue a research agenda, including choosing impactful research problems and autonomously carrying out projects.
• Demonstrated background in collecting data from human participants (e.g. surveys, experiments) with knowledge in data quality, data validity, etc.
• Strong communications skills with the ability to work effectively across internal and external organizations and virtual teams.