Page 1 of 1
HMS - Empower Platform
This is a 5 minute survey to get more information on the local LLM project.
Your name and role
*
Rank these by how much they matter to you in the first six months
*
Use the up and down arrow keys to change the rank.
1
BA practice by text, with an AI persona and feedback
1
Spoken practice (live voice conversation with the persona)
1
Recorded video answers for review
1
AI second rating of real BA sessions
1
Peer support practice
What would scores be used for? Select all that apply
*
Formative feedback to the trainee or counsellor
Helping supervisors choose which sessions to review first
Input to certification or employment decisions
Not decided yet
Who controls the recordings of real patient sessions?
*
A
HMS
B
An affiliated hospital
C
A delivery partner outside Harvard
D
A mix of these
E
Not sure yet
Where could AI rating of those recordings run? Select all that could work
*
Harvard's regulated-data computing environment (Level 4 or 5 data)
HMS research computing (O2 or Longwood)
An affiliated hospital's or partner's own environment
A dedicated machine we supply, kept on your premises
A secure host run by us, receiving pseudonymised transcripts only
Not sure yet, we would need to ask IT
Practice sessions involve a trainee and an AI persona, with no patient present. Could practice run in a dedicated HMS tenant that we host, with each vendor under a BAA?
*
A
Yes
B
Probably, subject to IT and privacy review
C
No, practice must also run on HMS infrastructure
D
Not sure yet
Which languages are the real sessions in?
*
A
English only
B
English plus other languages
C
Mainly other languages
D
Not sure yet
How are items on the BA quality scale rated?
*
A
A checklist of behaviours, with a rule that gives the level
B
A single overall rating per item (for example 0 to 4)
C
A mix of both
D
Not sure
Can you share the BA quality scale with us, with permission to encode it in the platform?
*
A
Yes, now
B
Yes, after a permission step on our side
C
No
D
Not sure yet
Do real BA sessions already exist that two clinicians have each rated independently on this scale?
*
A
Yes, 30 sessions or more
B
Yes, fewer than 30
C
No
D
Not sure
To measure how closely an AI rater agrees with your clinicians, we need at least 30 real BA sessions, each rated independently by two clinicians. We estimate 75 to 120 clinician hours in total. Could HMS commit that?
*
A
Yes, within 3 months
B
Yes, within 6 months
C
Yes, but later than 6 months
D
Only for fewer than 30 sessions
E
No
F
Not sure yet
How quickly could a named clinician review and sign off new content, such as personas, hints and voices?
*
A
Within 1 week
B
Within 2 weeks
C
About a month
D
Longer, or not yet known
Who should we speak to in IT, privacy or research computing once we have a shareable SoW? Name and email please.
Anything else we should know before we scope?
Submit
