Virtual Patients Group - Improving communication skills education using virtual patients.

Technical Innovations
		Interpersonal Simulator (IPS) (Developers: Johnsen, Raij, Dickerson, Kotranza, Rossen, Chuah) The interpersonal simulator system enables students to practice communication skills with virtual humans. The system supports many interface modalities. IPS is a software system that enables users to practice an interpersonal scenario (typically a conversation) with a virtual human. Input: the user interacts with virtual humans using a combination of: typing natural language, natural language speech, multiple-choice, gestures, touch, and devices such as a Nintendo Wiimote. The user is tracked using cameras and retro-reflective markers. Conversational Modeling: Crowdsourcing approach to virtual human creation enables scenarios to be modeled in weeks. Cognitive: Currently we employ a keyword matching approach with a statistical model. More complex models can be employed. Output: The virtual human can be presented as any of the following: a life-sized 3D character using a HMDs, projectors, or large-screen TVs, as an avatar in SecondLife or a web-browser video chat-bot, or as an online text chat-pot. Demos: Interview a pharmacy patient online, Video chat with a pharmacy online Videos: [Overview] Publications: [Validity][IPS] Funded by the National Science Foundation.
		Mixed Reality Humans (MRHs)* (Developer: Kotranza) Integrated virtual humans (to practice communication skills) with active and passive haptics (to practice procedure). The MRH platform combines IPS with both passive and active haptic systems. The passive haptics are used to enhance the realism and guide the user’s touch. The active haptics are integrated with the virtual human such that they can respond to the user’s touch. For example, if the user presses too hard, the sensors in the physical simulator measurers the pressure and passes that to IPS. Then, IPS has the virtual human say that they are in pain and present the appropriate facial animation. MRHs have been shown to enhance user’s empathy to the VP, and it has been tested with hundreds of health profession students. Videos: [Clinical Breast Exam] [Bi-directional Touch] Publications: [Acceptability and Usability of MRHs] [Bi-directional Touch] IEEE Virtual Reality 2008 Best Paper Funded by the National Science Foundation.
		Virtual Multi-tools (NERVE) (Developer: Kotranza, Horton, Hwang) Integrated commodity interfaces to provide robust and interactive experiences with virtual humans with rarely-seen conditions. The IPS system has been integrated with commodity interfaces, specifically the Nintendo Wiimote. The Wiimote enables the user to have a robust mechanism to gesture and interact with the user. For example in the eye-exam scenario, the Wiimote represents the user’s hand, enabling them to robustly raise and lower fingers, move their hands about, use virtual tools such as an ophthalmoscope, and administer neurological tests. Studies showed that providing haptic interfaces enabled the system to mask errors with interfaces, such as speech recognition and understanding. Videos: [Cranial Nerve 3 Exam] Publications: [Virtual Multi-tools] Funded by the National Institutes of Health.
		Interpersonal Visualization (IPSViz)* (Developer: Raij) Developed technologies that enable students and educators to reflect and analyze their interactions with virtual humans. IPSViz – Interpersonal Visualizer is a software system that takes IPS interactions and analyzes and visualizes the log files. Two separate systems have been created: IPSViz^1 – This software takes an individual IPS log file and enables reviewing the interaction from different perspective, highlights conversational topics visited and avoided, and presents the interactions of experts as a comparison for self-reflection. IPSViz^n – This software enables exploration of multiple IPS log files, for example, educators identifying trends and outliers in a class wide use of IPS. Online demos: IPViz^1 (id/pass: 10157/00000) and IPSViz^n. Video: [IPSViz] Publications: [IPSViz] Funded by the National Science Foundation.
		Virtual Social Perspective-Taking (VSP) (Developers: Raij, Kotranza) Developed interactions where the user re-experiences an interaction with a virtual human from the perspective of the virtual human – in essence, interacting with oneself. The VSP system enables users to immediately revisit their IPS interaction with a VH, but from the perspective of the VH. Recorded video, head tracking data, and speech data are used to have the user play the role of the VH talking with a recorded version of themselves. This system enables users to interactively evaluate their affect, observe non-verbal behavior, and identify conversational opportunities for empathy. Videos: [VSP] Publications: [VSP]
		Distributed Conversational Modeling (Virtual People Factory)* (Developers: Rossen) VPF is software that enables content-area experts (e.g. educators and end-users) to rapidly create virtual humans. Leveraging crowdsourcing, VPF has end-users and educators model conversations between users and VHs. Using a web-browser interface, scripts can be created by leveraging a few educators and many students to model the conversation scenario. As opposed to the common model of having experts and VH developers model scripts used for training, the fundamental realization is that experts can not anticipate the varied approaches that novices will try to address a topic. Thus it is critical to involve both educators and students in the conversational modeling process. This results in new virtual human conversation scenarios being conversationally modeled in approximately two weeks as opposed to 6 months+ in pervious methods. Further the VPF created conversation models are more significantly accurate and contain more detailed responses by the VH. Papers: [XXX], Videos [XXX], Demos: Interview a pharmacy patient online
		Virtual Patients + Human Patient Simulators* (Developers: Hwang) Integrated METI’s Human Patient Simulator (HPS) with IPS to enable hands-on procedure, interaction, and communication skills training. IPS has been integrated with the Human Patient Simulator (HPS) via its HIDEP protocol. The VP is aware of the HP’s vital signs, such as blood pressure, heart rate, and respiration. This integration leverages the VP’s ability to 1) handle interpersonal communication, 2) present a variety of virtual human models (weight, ethnicity, and gender), and 3) present dynamically visuals difficult to show with physical devices (e.g. lips turning blue, vomit, burns, the state of internal components). This integration of IPS and HPS creates experiences that completely simulate a variety of complex verbal+haptic scenarios that focus on procedure and communicate affect. Videos: [VP communicating with HPS] Publications: [in submission]
		Bias with Virtual Humans (Developers: Rossen) Study to the degree that virtual humans elicit bias and can be used to educate users on cultural competency skills. We are exploring if real-world biases transfer to the virtual world. That is, do virtual humans that present different characteristics, such as ethnicities, gender, age, and weight, cause users to treat them similarly to the way they would treat real people with similar backgrounds? To study this, we have worked to correlate the Implicit Association Test (IAT) with video coder reviews of medical students interviewing and empathizing with virtual human patients. Our work has highlight the correlation of IAT skin-tone bias to empathy exhibited to VHs of varying skin-tones. Videos: [XXX] Publications: [Skin-tone bias with VHs] Funded by the National Science Foundation.