Research Field 2: Simulation and optimization of heterogeneous human-machine-teams

PhD project 2.1: Data assimilation for sensor-based modelling of movement patterns of human and machine actors

Supervisors: Dr. A. Reinhardt and Prof. J.P. Müller

State of the art: Human-machine-teams can be found in many fields of application, such as mixed traffic, intelligent objects in transport and logistics or in production. The close cooperation of human and machine actors on shop floors of modern industrial plants is another important field of application for such teams, for example when employees have to decide in their daily work which way they want to take through the workshop to their workplaces. In order to avoid accidents, autonomous robot vehicles located on the shop floor at the same time are currently working reactively, i.e. they recognize people and stop when possible dangerous situations are detected. Any resulting impairments to the production processes are accepted in terms of higher occupational safety. However, a more precise knowledge of human movement patterns would make it possible to optimize the trajectories of the machines in such a way that the resulting impairments would be minimal. This accurate and realistic modelling of human and machine actors in practical areas of application of human-machine teams, with the objective of a realistic representation in simulations, is the subject of many currently undertaken research efforts. A more detailed analysis of the articles in this field shows, however, that the modelling of human behavior is primarily based on external observations (e.g. data from statistical surveys or stay in mobile radio cells). Active user participation - either through the conscious provision of data or the active confirmation of recognized movement patterns and connections - is rarely part of the scientific procedures employed. However, complementary analyses already point to the enormous information content of user-related data; the research field of activity recognition is even largely based on data collected in this way. In order to gain a comprehensive understanding of movement patterns and their triggers, the processing of such data promises a high gain in information, for example about the current mood, the time until the next appointment or the health condition of the user.

Research gap: Modern sensors, which are already contained in most smartphones or wearables today, offer a cost-effective way to collect information about a multitude of attributes of human and machine actors and to use them to create parameterizable motion models. In the literature, however, such physiological parameters have so far hardly been considered in order to make predictions about user activities and their behavior in industrial plants. In the context of this doctoral thesis, therefore, three essential open research questions are to be considered which, in their interaction, serve as accurate and versatile simulation models for user movements:

(1)    Development of a guideline for sensor configuration and application, with the help of which different movement patterns as well as physical and emotional states can be reliably recognized and related to each other,

(2)    Modelling and mapping in simulation models; in particular, it should be examined whether uniform parameterizable models are possible for different types of actors (e.g. employees with different tasks and/or qualifications in the context of industrial production or dismantling), or whether a more detailed breakdown is necessary,

(3)    Methodologies for evaluating the quality of the resulting models and validating them

A current approach to enriching existing models with data from observations, which will be used in this paper, is data assimilation, which will be used together with methods of Massive Online Analysis in this paper as a methodology. As an evaluation scenario, the modelling and simulation of human trajectories in the context of a recycling dismantling factory, for example in the context of occupational safety, is to serve as a basis for evaluation; further scenarios, e.g. from the field of mobility (pedestrian modelling), are to be used for generalization.

Own preliminary work: The preparatory work of the applicants serves as a basis for the investigations. On the part of AG Müller, these are far-reaching preliminary works for modelling and extracting the preferences of road users, e.g. on the basis of voting models, for agent-based simulation of socio-technical systems and for microscopic approaches for modelling mixed traffic with special consideration of pedestrians. In the field of mobile sensor technology, Dr. Andreas Reinhardt can already look back on preparatory work on several of the aforementioned topics. These include studies on the use of sensors on smartphones for localization within buildings and the positioning of smartphones using acoustic signals. On the other hand, in the context of a current research project he considers the possibilities of modelling and approximation of parameters from the energy grid; a transferability of the developed methods to user-centric data, which were collected with the help of context sensors, can be expected due to the characteristic patterns in such data series. Finally, it has also helped to develop procedures for the adequate protection of privacy in the provision of high-resolution and personal data, as it also plays an important role in this doctoral topic.