The main goal of this work-package is to contribute to the development of an intelligent system, able to provide information both to teachers and to students. As such, this WP will develop a set of tools, which can be potentially used in different types of applications, but that will be applied and optimized to aid both teaches and students inside a classroom environment.
Our long term goal is to transform a classroom in a sensing environment, capable of giving advice to the teacher, which can use it to improve his/her teaching role, and to the students, aiming at increasing their academic performance. For teachers, the system will be able, for example, to produce graphs displaying the average level of students attention along the class (in lectures) or their level of activity (in labs). For the student, it will provide counselling regarding his/her behaviour during the class, for example, indicating in which parts he/she was less watchful and the corresponding topics that potentially need extra attention.
Sensing in the classroom will be based primarily on computer vision, through multi-camera setups, but it may be complemented with other data captured, for example, from wearable devices. In fact, wearable devices will be the main source of off-class data, contributing to improving the quality of the recommendations, for example by integrating and correlating information regarding sleeping patterns, physical activity, eating habits, etc. Besides automatic counselling information, students will also have the possibility of requesting the playback of sections of the video recordings where they are present, for checking behaviours pointed out by the system as potentially leading to lower academic performance. Note that the toolbox developed to assess our reference application can then be made available to other services through the infrastructure developed inside Task 3.2 and Task 3.3.
To attain the ultimate goal of a sensing campus, many problems will have to be tackled. Some of them will be addressed in this project, including the availability of a high-performance infrastructure for sensing and transporting data (coming from WP 2), as well as the development of beyond state-of-the-art algorithms for extracting useful information from the huge volumes of data that will be continuously produced from different sources (the toolbox developed inside Task 5.2), and finally (inside Task 5.3) generating potentially useful information to teachers and students (both in real-time and in an aggregated form). However, we are well aware that the ultimate goal of a fully functional sensing campus is very ambitious. Hence, this work package will act as a brick, although fundamental, in this endeavour.
Privacy will play a key and decisive role in the success of such a system. Student acceptance and involvement in the project is fundamental, and it can only be achieved if the privacy issues are settled right from the beginning. Hence, mechanisms will be put in place to ensure that individual data will be protected (an essential part of Task 5.1). For example, procedures will be implemented to permit automatic blocking of parts of images corresponding to persons not willing to be surveilled and counselled by the system (for example, some of the students in a classroom).
This work package will also handle scientific and technical dissemination, as relevant for the project.
| Data | Milestones | Description |
|---|---|---|
| 2017-10-16 | Design and deployment of a sensing system (M4-M14) | This system will be mainly composed of a fixed set of cameras, covering the classroom from several viewpoints, allowing the recording of the activity of all students, as well as the activity of the teacher. The system will allow automatic masking of zones corresponding to places containing students not willing to be recorded. This will be possible by assigning fixed places to the students, allowing building and loading into the system a precomputed mask. Posteriorly, this occlusion operation might evolve to a procedure capable of tracking the position of those students for applying the mask dynamically (such a need arises for example in some lab classes where the students have to move inside the classroom). In this task, we will also investigate which additional individual sources might be useful to produce relevant information regarding the academic performance of the students. Although many wearable devices do exist nowadays, capable of delivering several types of signals from different sources, it is not currently clear which of those signals may convey the most relevant information to the problem at hand. Hence, it is an objective of this project to find out which signals/sources will best suit our needs, relying on the infrastructure work developed inside Task 3.3. The ubiquitous smartphone will be also a valuable source of data to be explored. |
| 2017-10-16 | Development of computer vision and data compression algorithms (M7-M27) | This is a fundamental task---the success of the whole system will depend on how good are the computer vision algorithms at discriminating different human behaviours. Ideally (and this is the ultimate goal), we would have real-time feedback, i.e., if requested by the student, upon an event detected by the system as relevant, the student would be immediately alerted (for example, through its smartphone or through a wearable device). However, due to the computational power required for real-time operation, this might not be possible in an initial phase of the project, forcing off-line processing and after-class reporting/counseling. This alternative mode of operation will imply transmitting and storing large amounts of video data, most of them highly redundant, due to the multi-view camera setup. Hence, it will be fundamental to adapt and develop compression algorithms tailored to this problem. |
| 2017-10-16 | Extracting knowledge from the data (M9-M32) | Signal processing, data mining and machine learning will be technologies playing a leading role in this task, with an overall complexity which will surpass the transversal toolbox developed inside Task 3.2. It is here that the huge amounts of data will be conveniently transformed and mined, in order to create models capable of delivering appropriate counselling. These models will provide recommendations using predictions based on knowledge acquired previously. This knowledge will result from the combination of all the sensory data together with the academic record of the students. To create these predictive analytics models, we will explore novel techniques, based on the notion of algorithm entropy and on its approximation by data compression algorithms. In fact, about half a century ago, Ray Solomonoff, Andrei Kolmogorov and Gregory Chaitin came up with ground-breaking ideas that led to what is now known as the Kolmogorov complexity or algorithmic entropy. Some of these ideas have recently started to change the way some fundamental problems are addressed, in particular on how to measure the similarity between two digital objects and on how to predict future values from past ones. |