Understanding Human Internal States from Brain Imaging and Behavioral Data

Soo-Young Lee, Suh-Yeon Dong, and Bo-Gyeong Kim


To understand the human mind, i.e. the internal states of human brain, we usually rely on explicitly-represented features such as speech, gestures, and facial expression. On the other hand the true internal states, such as intention and goals, may sometimes be different from the explicitly-represented ones, for delicate issues especially in Asian cultures. With the popularity of intelligent machine, it also becomes important to perceive the internal states of the machine. To understand the internal states and come out with a computational model of their dynamical interaction with others, we had started with a hypothesis on human internal-state space, and designed several cognitive experiments.

In connection with understanding other’s mind, two main axes are considered for the internal-state space of human brain, i.e. (a) agreement/disagreement with others and (b) trustworthiness of others. For the former, agreement/disagreement study, we assume conversational dialogue between a subject and an (artificial) agent, and measured fMRI, EEG and audio-visual signals while the subject was responding to self-relevant sentences intentionally and also unintentionally. For the latter, trustworthiness study, we had notices that three components are required to have trust of others, i.e., reliability (consistency), good intention (willingness to cooperate), and high capability (good performance). Since the first two are relatively straightforward, especially for intelligent machine, we are focusing on the third, high capability, i.e., “whether or not the other is capable of conducting the job”. During an iterative computer game between a subject and an (artificial) agent, again we measured brain and behavioral signals. To present the capability of others, based on the ToM (Theory-of-Mind), we set the agents with 0-level or 1-level prediction capability. For the analysis we had utilized fMRI, EEG, eye-tracking, audio, and/or video signals. Also, other internal states currently under investigated include shopping intention, children’s crying, etc.