تجزیه و تحلیل حساسیت برای سیستم های بیومتریک: متدولوژی بر اساس طراحی متعامد تجربی

The purpose of this paper is to introduce an effective and structured methodology for carrying out a biometric system sensitivity analysis. The goal of sensitivity analysis is to provide the researcher/developer with insight and understanding of the key factors—algorithmic, subject-based, procedural, image quality, environmental, among others—that affect the matching performance of the biometric system under study. This proposed methodology consists of two steps: (1) the design and execution of orthogonal fractional factorial experiment designs which allow the scientist to efficiently investigate the effect of a large number of factors—and interactions—simultaneously, and (2) the use of a select set of statistical data analysis graphical procedures which are fine-tuned to unambiguously highlight important factors, important interactions, and locally-optimal settings. We illustrate this methodology by application to a study of VASIR (Video-based Automated System for Iris Recognition)—NIST iris-based biometric system. In particular, we investigated k = 8 algorithmic factors from the VASIR system by constructing a (26−1 × 31 × 41) orthogonal fractional factorial design, generating the corresponding performance data, and applying an appropriate set of analysis graphics to determine the relative importance of the eight factors, the relative importance of the 28 two-term interactions, and the local best settings of the eight algorithms. The results showed that VASIR’s performance was primarily driven by six factors out of the eight, along with four two-term interactions. A virtue of our two-step methodology is that it is systematic and general, and hence may be applied with equal rigor and effectiveness to other biometric systems, such as fingerprints, face, voice, and DNA.

Biometrics is the automated recognition of individuals based on their biological and behavioral characteristics [1]. The characteristics can include fingerprints, face, iris, ocular area, retina, ear, voice, DNA, signature, gait, and hand geometry among others. The use of biometrics has many advantages—especially as an alternative to keys, passwords, smartcards, and other artifacts for physical entry. In this regard, biometric-based technologies are increasingly being incorporated into specific security fields and applications, such as industrial access control, law-enforcement, military, border control, and forensics [2]. A significant problem in biometric studies is that researchers/developers often present results that lack an assessment of intrinsic system uncertainty. A high degree of input and output numerical precision often gives the impression of great accuracy, but neglects to give attention to the critical questions of the sensitivity of the final results to different algorithms, environments, subject characteristics, and biometric sample conditions [3]. Hornberger and Spear [4] made the following paraphrased statement about simulation models: Most such models are complex, with many parameters, state-variables and underlying non-linear relations; under optimal circumstances, such systems have many degrees of freedom and—with judicious adjustments—are susceptible to over-fitting with both plausible structure and “reasonable” parameter values. We believe that the above statement applies equally well for biometric systems, especially for iris recognition system. Sensitivity analysis has been successfully conducted in areas such as computer vision and computer network [5], [6] and [7]. Sensitivity analysis is the study of how the output of a system is affected by different inputs to the system [8]. In essence, a biometric system is a data monitoring and decision-making “machine.” A good biometric system has a high proportion of correct decisions. All biometric systems are susceptible to incorrect decisions—especially in the presence of less-than-optimal conditions. In practice, the performance of many biometric systems is frequently examined and optimized via a series of one-factor-at-a-time experiment designs in which most factors in the system are held constant while one factor is focused on and varied to examine its effect. This design, though popular [9] and [10] has the disadvantage that it can yield grossly biased (incorrect) estimates of factor effects. Further, this design has no capacity to estimate factor interactions—which are intrinsic to many biometric systems. The motivation for this paper is thus to introduce to the biometrics community an alternative method for conducting a sensitivity analysis with the advantage that: (1) The system will be better understood. (2) The system will be better characterized. (3) The system will be better optimized—with the net effect that system performance is significantly improved in a computationally efficient fashion. Thus, in short, the objective of this paper is to introduce and apply a structured “Sensitivity Analysis” approach for gaining insight and understanding about the system’s key components—those which most affect the quality and performance of a biometric system—and to optimize the settings of these key components. Sensitivity analysis as we describe it has two separate and distinct steps: (1) Experiment Design (the structured plan for collecting the data), and (2) Statistical Analysis (the structured methodology for analyzing the data). Both parts are critical, and when optimally used in concert yield enhanced insight into the relative importance and effect of the various computational components (and interactions) affecting biometric system performance. The experiment design and data analysis are demonstrated by a particular iris-based biometric system, VASIR (Video-based Automated System for Iris Recognition), which has verification capability for both traditional still iris images and video sequences captured at a distance while a person walks through a portal [11]. The general structure of this paper is threefold: (1) Orthogonal fractional factorial design: Introduce to the biometrics community a structured orthogonal fractional factorial experiment design methodology to efficiently gain insight and understanding (“sensitivity analysis”) of critical system parameters, interactions, and their optimal settings—this introduces and applies an established method within the statistical community [12] and [13]. (2) Statistical analysis: Present effective and insightful statistical analysis methodologies for carrying out sensitivity studies. (3) Demonstration with VASIR: Demonstrate our experiment design and analysis methodologies for VASIR, with potential application to the broader biometrics field. This sensitivity analysis approach provides a tool for understanding the computational components affecting the overall performance of a biometric system. Based on such understanding, the logical follow-up is to carry out an optimization analysis (identifying the optimal global settings of the components), and a robustness analysis (assessing the range of validity for our sensitivity and optimization conclusions). Our current paper focuses on the details of the sensitivity analysis only. To demonstrate the elements of the sensitivity analysis approach, we restrict our focus to a fixed setting for two robustness factors: (1) eye position (left eye only), and (2) image type (video matching: non-ideal to non-ideal image only).

We introduced to the biometrics community a structured methodology for sensitivity analysis to foster an understanding of the key factors (parameters) in biometric systems. This Sensitivity Analysis methodology consists of two components: (1) Experiment design in which we utilize efficient orthogonal fractional factorial designs to estimate not only the k main effects but also the View the MathML sourcek2 two-term interactions of a biometric system. (2) Graphical data analysis in which we utilize three procedures: Main Effects plot, Ordered Data plot, and Interaction Effects matrix to determine important factors, two-term interactions, and optimal (local) settings. To demonstrate the utility of this methodology, using the MBGC distant-video dataset, we chose a (k = 8 factor, n = 384 run) (26−1) × (31) × (41) orthogonal fractional factorial experiment design for our VASIR system—investigating eight algorithmic factors (X1 to X8) to determine the most important, their optimal settings, and the relative importance of the 28 two-term interactions. For this Video to Video (VV) Left eye case (the focus of this paper), our experiments showed that the three most important (see bold below) out of these eight algorithmic factors that we studied in VASIR were X2 (IQMetr: Image quality metrics) with factors X3 (SegEye: Eyelids segmentation) and X7 (SMAlg: Similarity metrics) being next in importance. The least important factors were X5 (FXWL: FX wavelength) and X6 (FXMask: FX masking with magnitude). We found that the optimal settings were (+1, −1, +1, +1, −1, +1, −2, +1) with details as follows: – X1 (EyeAlg): Eye position alignment (+1: ON), – X2 (IQMetr): Image quality metrics (−1: Sobel operator [SOB]), – X3 (SegEye): Eyelids segmentation (+1: Curves), – X4 (NorRes): Radial resolution for normalization (+1: 32), – X5 (FXWL): FX wavelength (−1: 18), – X6 (FXMask): FX masking with magnitude (+1: 0.9), – X7 (SMAlg): Similarity metrics (−2: Hamming distance [HD]), – X8 (SMsh): Horizontal shifting number (+1: 5). In order of decreasing importance, the statistically significant factors that had an influence on VASIR performance were the six factors (X2, X7, X3, X1, X8, and X4). On the other hand, factors X5 and X6 were found to have barely any effect on VASIR’s overall performance. We found that some two-term interactions did in fact exist—they involved factors X1 (EyeAlg) (primarily), X2 (IQMetr), and X7 (SMAlg)—in particular, the X1 * X2 and X1 * X7 interactions were found to be important. On the other hand, for our VV Left eye case, we found that most of the interactions had minor effect on VASIR performance—hence the effect on performance of VASIR’s algorithmic component was mostly additive and independent. It is noteworthy that when the VASIR settings were changed from worst to best, VASIR’s verification rate at FAR = .01 increased significantly (12.8%) and EER decreased significantly (9.2%). In summary, the choice of the image quality metric for selecting the best quality image in video had the strongest effect on VASIR performance, followed by the choice of similarity metric. Our data analysis also reaffirmed that eyelid segmentation was important and that VASIR’s approach (Curves) had a significant improvement over IrisBEE’s approach (Lines). Further, VASIR’s two factors for correcting rotational difference due to head tilt or subject movement (eye position alignment, Horizontal shifting) were both important. It is of interest to note that comparing across studies, we found that optimal value of bit shifting for correcting inconsistency depended on the dataset (or the different imaging conditions of the dataset). Finally, we found that the larger radius size of the iris template had a better matching performance than the smaller size. Given our eight algorithmic factors, we found that VASIR is a near-linear system; thus, optimization of a particular factor is unlikely to influence the effects of the other algorithmic factors. We believe that the sensitivity analysis methodology demonstrated herein can be applied to other biometric systems. Based on our study, opportunities for future research would include as follows: (1) carrying out a follow-up experiment to ascertain the robustness of our conclusions over other scenarios (eye position (left/right), matching scenarios (VV: Video to Video, VS: Video to Still, and SS: Still to Still); (2) replacement of the two unimportant factors with other VASIR key algorithm factors; (3) reaffirming our conclusions by using larger datasets (e.g., more videos and subjects); and (4) applying the same sensitivity analysis methodology to simultaneously examine a considerably larger (e.g., k = 20 factor) set of VASIR algorithmic factors.