Internship on Intelligent Tutoring System in Virtual Reality based on machine learning or deep learning method with VR simulation and physiological data

In the context of Industry 4.0 and digital factory, digital twin and virtual reality represent technologies that can deal with several industrial issues like design, simulation and optimisation of industrial systems. They are helpful to perform operator’s training of industrial situation and evaluate them (Fratczak, Goh, Kinnell, Soltoggio, & Justham, 2019) (Havard, Jeanne, Lacomblez, & Baudry, 2019) (Matsas & Vosniakos, 2017).

               VR is thus useful to make people learn. However, the training scenario cannot be dynamically adapted to the learner profile or status. For example, an Intelligent Tutoring System (ITS) should be able to detect when a learner needs help, during the scenario, if he does not fully understand what he/she needs to do; or the ITS can detect that the learner feels easy enough to add a new unplanned event in the scenario in order to test the learner reaction to a more stressing situation. That is why, in this internship at LINEACT CESI Rouen, we need to analyse VR simulation data and physiological data in order to be able to adapt the VR scenario to the learner profile and status.

               LINEACT CESI has developed a flexible manufacturing system (FMS) involving several robots with different capabilities in a shop floor layout context. This use case is composed of an automatic production system and a set of manual workstations where the operator can be assisted by cobotic arms for assembly tasks. A digital twin associated to Human-machine interface (HMI) based on virtual or augmented reality is also present and allows to perform simulation in Virtual Reality Training Systems (VRTS). Particularly, Virtual Reality simulates a case study where an operator is working on a manual workstation to assemble a subassembly of a children's bike. Moreover, LINEACT owns physiological sensors (EEG, ECG, GSR, Head position, Gaze tracking) associated to the iMotions platform to record data, in a synchronous way. This use case and this hardware will be used during the internship in order to evaluate the learner profile and status thanks to physiological data.

Missions:

• Participation to the literature review on classification and prediction model for Intelligent Tutoring System in Virtual Reality Training System (VRTS) to adapt to the learner profile and status.
• Setup the data collection and storage components and participation to the design of the experimentation for acquiring data about the learner profile and status. With the platform available at LINEACT CESI Rouen, the following data are available: ECG, GSR, EEG, Head position, Gaze tracking, interactions and assembly simulation in VR, …
• Spatio-temporal data analysis from the experimentation.
• Proposition of a classification model  based on Machine Learning or Deep Learning algorithm for detecting the learner profile and status.
• Writing a paper about the results obtained.