Cognitively and Empathically Intelligent Collaborating Robots (KEIKO)

On the one hand, a combination of these different and complementary strengths promises considerable potential for use in key application areas such as industrial manufacturing, healthcare and nursing, and assistance systems for plants and vehicles. On the other hand, this very combination requires an extremely high degree of sociocognitive and emotional intelligence in robots; without this, flexible cooperation in complex environments and with changing human partners is impossible. In this context, empathic intelligence is understood as the fundamental ability to understand the partner's mental states and intentions for action and to react adaptively to them in a context-related manner. The lack of empathic intelligence in current cobots is the main reason for the still low use of flexible collaborative human-machine teams (MMT) in practice. Even the simplest use case - the situational empathic handover of an object, e.g. a workpiece, requires a currently unrealizable performance of the robot. What humans intuitively master raises problems for the machine: for example, it must assess whether the human partner is even ready to take over the object ("take-over readiness", TOR) and react accordingly, adapting its own task planning to the situation. The reasons for these difficulties are manifold. There is a lack of suitable data as well as sensor technology to make relevant state measurements (distracted, interested, attentive, ...) on humans, there is a lack of practicality of abstract models that build on the data and consider humans and machines together, and, in case of insufficient implementation, there is a lack of acceptance if data security and sparseness are not considered for this very personal data.

The former represents the most fundamental research gap: In practice, a measurement on humans, however designed, must always be passive and contactless or at least - related to the activity of the human - minimally invasive, so as not to interfere with e.g. a production process or recycling process. Sensor technology that collects data on the emotional state of humans from physiological measurements (brain activity, breathing, pulse, etc.) has so far led to unacceptable restrictions on work activities due to cables, complex setups and susceptibility to interference. Future measurements in the field must not have this limitation. Artificial Empathic Intelligence (KEI) for robots is also the subject of current research. Results are urgently needed here, because there exist frightening examples of the lack of KEI in practice: for example, only after the introduction of driver assistance systems was the question raised to what extent the autopilot even knows whether the human is ready at that moment when the autopilot hands over control of the vehicle to the human.

The resulting danger is obvious; it can only be countered by taking situational-empathic information into account. These problems also occur in manufacturing and lead to conflicts or dangers. Accordingly, research is needed into methods that enable cobots to work hand in hand with humans without conflict and without danger.

Cobots therefore represent an important future version for the digitization of industry. However, humans can only work together with robots at the level of genuine human-human collaboration if cobots, like us, have empathic intelligence. This is a wide-ranging and innovative problem that requires an interdisciplinary effort of different research areas in the combination of software and hardware research with psychology and modeling, as in the Clausthal and Göttingen joint project "Cognitively and Empathically Intelligent Collaborating Robots" - KEIKO.

KEIKO: Cooperative project between Clausthal and Göttingen at the SWZ

The funded project KEIKO arose from research results of the preliminary project Heterogeneous Human-Machine Teams (HerMes), which was funded by the Clausthal University of Technology. The linking of man and machine to a joint, cooperatively working team, in which the robot adapts to the human instead of the human adapting to the robot - human-centeredness in the true sense - was and is being investigated.

Six different subprojects will explore trust estimation and building between humans and cobots, prediction of human motion trajectories, adaptation and optimization of cobot motions, human behavior patterns, acquisition of physiological and psychological parameters, and data fusion.

With data-driven modeling of human behavior as well as machine prediction of human action intentions and motion trajectories, predictions for the control of a cobot will be made. On the other hand, emotion-based parameters will be derived from neuropsychological studies, whose contactless measurement in interaction with wearables, wireless sensors worn on the body, will create the data basis that makes emotionally empathetic cobots possible.

The research of emotion, cognition and behavior, and how these can be modulated by environmental conditions and individual differences, also taking into account social aspects; to recognize action errors due to false mutual intention and movement predictions, and the conflict-free avoidance or mitigation of errors are thus the goals of the collaborative research. The findings will be demonstrated on a cobot, which at the end of the project should be able to perform a simple (dis)assembly task hand in hand with a human.

The project is funded by the Ministry of Science and Culture in Lower Saxony (MWK) within the framework of the SPRUNG program with more than 1.7 million euros. From January 1, 2023, KEIKO will be centrally coordinated at the Simulation Science Center Göttingen-Clausthal and implemented at the demonstrator.

The collaborative partners are Prof. Dr. Annekatrin Schacht and Prof. Dr. Florentin Wörgötter from the University of Göttingen and Prof. Dr. Jörg Müller, Prof. Dr. Andreas Reinhardt and Prof. Dr. Christian Rembe from the TU Clausthal. In addition, Prof. Dr. Michael Prilla, who moved from TU Clausthal to the University of Duisburg-Essen in 2022, is working on the project.

• 2023-04-20 medizin&technik: "Wenn der Pflegeroboter Gefühle erkennt"
• 2023-03-06 taz: "Wenn der Roboter den Menschen missversteht"
• 2023-01-27 TU Clausthal press release: "Empathische Zusammenarbeit von Mensch und Maschine"
• 2024-07-08 iQ-Journal: "Von Star Trek zur Wirklichkeit - Wie die TU Clausthal Roboter empathisch machen will"
• 10.9.2024 ti-Magazin: "Kollaborierende Roboter entwickeln Empathie"

• Prof. Dr.-Ing. Christian Rembe
• Niels Rohweder, M.Sc.