مدل شناسایی خستگی راننده بر اساس ترکیب اطلاعات و شبکه های بیزی دینامیکی

We propose a driver fatigue recognition model based on the dynamic Bayesian network, information fusion and multiple contextual and physiological features. We include features such as the contact physiological features (e.g., ECG and EEG), and apply the first-order Hidden Markov Model to compute the dynamics of the Bayesian network at different time slices. The experimental validation shows the effectiveness of the proposed system; also it indicates that the contact physiological features (especially ECG and EEG) are significant factors for inferring the fatigue state of a driver.

The recent advances in cognitive science, psychology, and related fields have indicated that the human emotion (such as anger, fear, stress, distraction, and fatigue) plays a critical role in a person’s behavior [28] and [21]. The behavior of drivers has been an active field of study for decades [23], and it has attracted considerable attention recently [34]. The driver fatigue remains to be one of the important factors that contribute to traffic accidents. The National Highway Traffic Safety Administration (NHTSA) of USA estimates that there are annually about 100,000 crashes in USA that are caused by fatigue and result in more than 1500 fatalities and 71,000 injuries [15]. Some studies have demonstrated that the driver drowsiness accounts for 16% of all crashes and over 20% of the crashes in the highways [8]. Thus, the driver fatigue assessment remains to be a big challenge to meet the demands of future intelligent transportation systems [3]. Developing a system that actively monitors the driver’s fatigue level in real time (and produces alarm signals when necessary), is important for the prevention of accidents, and this is the main motivation of our paper. One of the key steps towards developing a fatigue monitoring system is to consider the features that could be effectively used for fatigue recognition. We can classify these features into four general categories: (1) causal/contextual features, (2) physiological features, (3) performance features, and (4) multi-features. In the following paragraphs, we discuss these categories. 1.1. Contextual features based method The contextual features mainly include (i) the personality, sleeping quality, circadian rhythm, physical condition, (ii) the work conditions such as noise, driving hours [11] and [30], and the cab temperature; (iii) the environment such as monotony of road, density of cars, and number of lane. Such contextual features are collected mainly by questionnaires, and then the driver’s fatigue is inferred from the collected data using some statistical methods or other means such as neural network or fuzzy set theory [10], [11], [33] and [35]. 1.2. Physiological features based method The drivers may exhibit some easily observable physiological features from which their fatigue can be inferred [15], [28], [32], [39] and [41]. Physiological features may be classified into: contact features, including the brain activity, heart rate variability, and skin conductance – these can be easily detected by EEG (electroencephalogram), ECG (Electrocardiograph), and EMG (electromyogram); and contactless features, including the eye movements (EM), head movement, and facial expressions – these can be easily observed from the dynamic images provided by a CCD camera. Consequently, two approaches for research are feasible: the contact feature based method and the contactless feature based method. The contact feature based method focuses on inferring the driver’s fatigue from the contact features. Using the fact that the EEG can represent abundant information on the human cognitive states, an algorithm based on the changes in all the major EEG bands (delta, theta, alpha, and beta bands) during the fatigue was developed by Lal et al. [19] to detect different levels of fatigue. Combining the EEG power spectrum estimation, principal component analysis, and fuzzy neural network model, Jung et al. [17] designed a system to estimate and predict the drowsiness level of a driver. Taking the associated wavelet representations for the EEG at different scales as system inputs, Wilson and Bracewell [41] constructed a neural network to detect the onset of the driver’s fatigue. Zhou et al. [46] proposed a new feature extraction method based on the bi-spectrum and applied it for the classification of the right and left motor imagery for developing EEG-based brain–computer interface systems. Budi et al. [2] assessed the four electroencephalography (EEG) activities, (delta (δ), theta (θ), alpha (α) and beta (β)) during a monotonous driving session for 52 subjects (36 males and 16 females), and got the results for conditions stable delta and theta activities over time, a slight decrease of alpha activity, and a significant decrease of beta activity. The ECG is another contact feature, including the LF (low frequency), VFH (very low frequency), HF (high frequency), and the LF/HF ratio, that contains relevant information about fatigue [33]. By taking the Hermite polynomial coefficients of the ECG as inputs, [24] presented a neuro-fuzzy network approach that was used to recognize and classify the heart rate variation. It is noted that Picard et al. [28] also applied this to affective computing, and proposed a hybrid recognition algorithm combining the Sequential Floating Forward Search and the Fisher Projection for the emotion recognition, by selecting the means, the standard deviations, the first differences, and the second differences of the EMG, BVP (blood volume pulse), GSR (galvanic skin response), and respiration from the chest expansion as physiological features. In addition, the fast Fourier transforms (FFTs) and three other modeling techniques, namely, the autoregressive (AR) model, the moving average (MA) model and the autoregressive moving average (ARMA) model, are used to estimate the power spectral densities of the RR interval variability in Zachary et al. (2008). The spectral parameters obtained from the spectral analysis of the HRV signals are used as the input parameters to the artificial neural network (ANN) for the classification of the different cardiac classes. The contactless feature-based method focuses on inferring the driver’s fatigue from the contactless features [13], [18], [26], [44] and [1]. Experiments have demonstrated that the driver in fatigue should exhibit some visual cues Ji et al. [15]. Horng et al. [13] proposed a driver fatigue detection algorithm based on the eye tracking and dynamic template matching. Norimatsu et al. [26] investigated the detection of the gaze direction using the time-varying image processing in which the facial and the gaze directions, without considering the facial direction, were detected separately, and then they were integrated into the final gaze direction. Kim et al. [18] constructed a fuzzy neural network-based method for fatigue recognition by taking the openness degrees of the mouth and eyes respectively, and the vertical distance between the eyebrows and eyes as inputs. 1.3. Performance measurement based method Driver’s fatigue can contribute to the deterioration in the operational performance (such as the reaction time, lane position deviation, and hand movement of controlling the steering wheel). A fuzzy set-based method involving the small movement of controlling the steering wheel was put forward by Vysoký [38] and [37] to calibrate and predict the driver’s fatigue. 1.4. Multi-feature fusion-based method The three methods described above focus only on a certain specific aspect, and that may lead to inaccurate results because the driver’s fatigue is not directly observable but can only be inferred from the information available. There are a number of reasons for the inaccuracies using the method mentioned above: (i) the driver’s fatigue derived from the contextual features contains much subjectivity that can not always reflect the real objectivity; (ii) inferring the driver’s fatigue from the facial expression is not always reliable because of the following two limitations: (a) the current techniques for image processing can not always ensure the recognition accuracy; (b) an introverted person might have a tendency to control his or her display of emotions, especially in the presence of people he/she is not well acquainted with [6], which leads to an inaccurate interpretation of the facial expression. Thus, to fuse as many as possible features from uncertain events is a better way to make an accurate inference [5]. Further, Picard et al. [28] pointed out that it was necessary to fuse the contextual and physiological features, and the driver’s performance in order to make the fatigue recognition more reliable. By considering the evidence and beliefs of the contextual information and the visual cues from a single time instant, Ji et al. [15] constructed a static Bayesian network (SBN) to infer and predict the fatigue of human beings, enhancing the reliability of fatigue recognition. However, such a network does not consider the physiological contact features and is not suited to systems that evolve over time [16], [21], [43] and [45]. The Dynamic Bayesian network (DBN) has been developed to overcome this limitation. Li and Ji [21] introduce a new probabilistic framework based on DBN to dynamically model and recognize the user’s affective states and to provide the appropriate assistance in order to keep the users in a productive state. Considering the evidence and beliefs of contextual information and visual cues from multiple time stamps, a new probabilistic framework based on DBN has been introduced in Ji et al. [16] but it has excluded the contact physiological features. Besides the contextual information and visual cues (the contactless features), the contact physiological features (such as the EEG, and the ECG) may also contribute significantly to the fatigue because a person usually has little control over these contact features and so they could provide reliable source of information on a person’s emotion [6]. However, it is very difficult to apply these two physiological signals non-intrusively because usually the electrodes and wires are used to contact a driver intrusively in order to obtain the EEG and ECG signals. There have been some efforts in developing non-obtrusive EEG and ECG technologies, and there has been a certain degree of success related to the non-obtrusive instrument use in laboratory settings. It is thus expected that the non-obtrusive EEG and ECG technologies (e.g. wireless technology) will become feasible in the near future for applications to fatigue estimation for drivers. Therefore, it is useful to investigate fusing the ECG and EEG features with other features for getting reliable fatigue estimations, which lays a strong foundation for the real application of getting reliable fatigue estimation from multiple features. Inspired by the research of Ji et al. [16], in particular by the inclusion of the contextual features with the contactless features (mainly the facial expression features), we have developed a DBN-based fatigue recognition model along with its inferring algorithm to make the fatigue recognition task more feasible. However, our approach differs from that of Ji et al. [16] in the following ways: (i) we have included the contact physiological features – in particular the ECG and EEG, and (ii) in the treatment of the dynamics of a Bayesian network from one time slice to another, we have used the first order HMM (Hidden Markov Model). The rest of the paper is organized as follows. Section 2 presents the problem description and a general architecture of the proposed DBN-based fatigue recognition model. In Section 3, the proposed DBN-based fatigue recognition model using multiple contextual and physiological features is presented along with the inference method corresponding to the model. In Section 4, the validation of the model is presented. Section 5 concludes the paper with further discussions.

In this paper, a new method for inferring driver’s fatigue estimation based on the dynamic Bayesian network was proposed. Multiple features, including contextual, contact physiological, and contactless physiological features were used, which have the widest coverage of the categories of features. The first-order Hidden Markov Model (HMM) has been employed to compute the dynamics of a Bayesian network at two different time slices. Simulation-based experiments were performed to demonstrate the validation of the proposed model. Two important conclusions can be drawn from this study: (i) more features, especially the contact physiological feature category, which covers more features implying driver fatigue recognition, are favorable for inferring the driver fatigue more reliably and accurately; (ii) the ECG and EEG are two important features for fatigue recognition, and they should not be absent from consideration in any driver fatigue detection system. It would be of significant interest to extend the current model of a discrete random process to a continuous random process to handle more practical situations; and also to investigate how to decrease the subjectivity in determining the transition probability (i.e., learning the DBN structure and parameters from the samplings), which is very important for the problem discussed in this study. We hope that this work would generate further interest in this challenging research problem.