Task description

For the purpose of this competition our team acquired data generated by “smart jacket” – a wearable set of body sensors for monitoring kinematics and psychophysical condition of firefighters. The data were obtained during training exercises conducted by a group of eight firefighters from The Main School of Fire Service. The sensors were registering firefighter’s vital functions (i.e. ECG, heart rate, respiration rate, skin temperature) and movement (i.e. seven sets of accelerometers and gyroscopes placed on torso, hands, arms and legs). Each exercise session was also captured on video and the recordings were synchronized with time series representing the sensor readings. An exemplary video recording from one of such exercises is available in the data files folder. All this data were presented to experts who manually labeled it with activities.

The objective in this competition is to devise efficient methods for automatic labeling of short series of the sensory data with basic activities of a firefighter. On the one hand, this task is very challenging due to a fact that different people tend to perform the same activities in different ways. On the other hand, however, automatically generated and accurate activity labels would facilitate monitoring of firefighter’s safety and contribute to development of efficient command support systems.

training exercises

Data format: We provide the data for this competition in a tabular format. The training data set, namely trainingData.csv, is a comma-separated values file. It contains 20,000 rows and 17,242 columns. Each row corresponds to a short time series (approximately 1.8 s long) of sensory readings. The first 42 columns represent aggregations of data from sensors monitoring firefighter’s vital functions. These measurements were obtained using Equivital Single Subject Kit (EQ-02-KIT-SU-4). The remaining columns are divided into 400 chunks that represent consecutive readings from sets of kinetic sensors attached to firefighter’s torso, hands, arms and legs (a total of seven sets). Each set is composed of an accelerometer (dynamic bandwith: +/- 16G) and a gyroscope (scale up to 2,000 deg/s). Therefore, a single chunk of columns consists of 43 numeric values, from which the first one is time from the beginning of the series and the following 42 values represent the readings from the accelerometers (measured in m/s^2) and gyroscopes (measured in deg/s, drift and temperature compensation was applied). More details regarding the description of data columns can be found in the column_info.txt file. An average time difference between consecutive sensory readings in the data is 4.5 ms. Labels for the training data are provided in a separate file, trainingLabels.csv. Each row in this file contains two labels for a corresponding row in the training data. The first label describes a posture of a firefighter and the second label describes his current main activity. Test data file, namely testData.csv, is in the same format as the training data set, however, the labels for the test series are hidden from participants. It is important to note that the training and test data sets consist of recordings which were obtained from different groups of firefighters.

Format of submissions: The participants of the competition are asked to predict labels of the time series from the available test set (testData.csv) and send us their solutions using the submission system. Each solution should be sent in a single text file containing exactly 20,000 lines. In the consecutive lines, this file should contain exactly two strings indicating label names for the posture and main activity of a firefighter, separated by a comma – the format of this file should be the same as the format of the trainingLabels.csv file.

Evaluation of results: The submitted solutions will be evaluated on-line and the preliminary results will be published on the competition leaderboard. The preliminary score will be computed on a random subset of the test set, fixed for all participants. It will correspond to approximately 10% of the test data. The final evaluation will be performed after completion of the competition using the remaining part of the test data. Those results will also be published on-line. It is important to note that only teams which submit a short report describing their approach before the end of the contest will qualify for the final evaluation. The winning teams will be officially announced during CEIM'15 workshop devoted to this competition (https://fedcsis.org/ceim) at the FedCSIS'15 conference.

The assessment of solutions will be done using the balanced accuracy measure which is defined as an average accuracy within all decision classes. It will be computed separately for the labels describing the posture and main activities of firefighters. The final score in the competition will be a weighted average of balanced accuracies computed for those two sets of labels. Namely, if for a vector of predictions preds and a vector of true labels labels we define the balance accuracy as: \[ACC_{i}(preds,labels) = \frac{|j : preds_{j} = labels_{j} = i|}{|j : labels_{j} = i|}\] \[BAC(preds,labels) = \left(\sum\limits_{i = 1}^l ACC_i(preds,labels)\right)/l\] and we denote by: $$ \begin{array}{ccl} BAC_{p} & - & \textrm{balanced accuracy for labels describing the posture}, \\ BAC_{a} & - & \textrm{ balanced accuracy for labels describing the main activity}, \end{array} $$ then the final score in the competition for a solution s will be computed as: \[score(s) = \left(BAC_{p}(s) + 2*BAC_{a}(s)\right)/3\]

Last modified: Friday, 10 April 2015, 3:25 PM