Research Field 1: Engineering and safeguarding of heterogeneous human-machine teams

PhD project 1.2: Sensor-based assessment of the controllability of persons interacting with machines using the example of intelligent dismantling robots for recycling critical products

Supervisors: Prof. C. Rembe and Prof. J. Gertheiss

State of the art: Various methods can be used to assess the controllability of an employee in a dismantling factory for recycling, e.g. identification of distraction. Especially with vehicle-based and behavior-based systems there is preliminary work on this topic. The former mainly use movements of the steering wheel and the position in the lane; the latter usually record eye, face, head or hand movements via one or more cameras. The only parameter that is estimated in existing systems, however, is the degree of fatigue or an attempt to identify certain ideal situations such as driving a telephone. For a team of employees and intelligent dismantling robots, however, a much greater degree of coordination between human and machine is necessary, since the robot must recognize whether the human is willing to take control of the dismantling process and demand active confirmation from the human prior to handover.

For this task, it is first necessary to find the appropriate medical parameters with which a willingness to take control of critical work steps can be identified. Vital parameters such as respiratory frequency, heart rate, heart rate variability, heart rhythm or blood pressure must also be determined. In fact, there are also approaches to determine and evaluate physiological parameters. The use of electrocardiography/ECG and electroencephalography/EEG, for example, is being researched. In these procedures, the human being must be wired with sensors, but for real use it is necessary to collect the data contactless. Current research approaches are also based on digital camera technology with image processing or Laser Doppler vibrometry. The project will now investigate whether these contactless methods are suitable for extracting relevant medical parameters that can be used to detect readiness to take over. In the existing literature on data evaluation, the respective measurements are generally considered over a fixed, defined time interval immediately before the current point in time. Such a data structure is also called functional data. In order to use this data to draw conclusions about the current state of a person, features are typically first extracted from the data (unsupervised), e.g. using wavelets, independent component analysis or simple summary statistics such as mean values, variances or correlations. Subsequently, these features are used as input for a machine learning process, whereby a wide variety of techniques are used, such as support vector machines, neural networks, K-Nearest-Neighbors, Linear Discriminant Analysis or Random Forests. In some cases, however, very simple rules are used, such as the question of whether the employee's eyes were closed for more than a certain period of time.

Research gap: What (almost) all methods that record physiological parameters have in common is that they use electrodes to be worn on the body/head. Routine use in everyday industrial work, e.g. in the dismantling process during recycling, is thus virtually impossible. Methods must therefore be researched with which the willingness to take over can be determined contactlessly.

Own preliminary work: Christian Rembe researches how medically relevant parameters can be determined using Laser Doppler vibrometry. For example, working group Rembe was able to show that Laser Doppler vibrometry can be used to contactlessly measure heart rhythm. In addition, the working group deals with the measurement of the motor function of persons by means of digital image processing and 3D object acquisition. Jan Gertheiss has been working on method development and (bio)statistical evaluation of functional data for many years. In particular, those methods should be mentioned: (a) methods based on principal functional components. Functional Principal Component Analysis (FPCA) can be used to determine the main directions of variation in the (functional) data. These can then be used as features for the classification. (b) Procedures for supervised feature/variable selection. (c) Modeling/identification of typical behavior (of animals) from sensor data and comparison of groups. (d) Prediction of behavior based on the immediate past.