از فرآیندهای پیش توجه تا نمایندگی پایدار: شاخص ERP از حواس پرتی های بصری

Abstract Visual search and oddball paradigms were combined to investigate memory for to-be-ignored color changes in a group of 12 healthy participants. The onset of unexpected color change of an irrelevant stimulus evoked two reliable ERP effects: a component of the event-related potential (ERP), similar to the visual mismatch negativity response (vMMN), with a latency of 120–160 ms and a posterior distribution over the left hemisphere and Late Fronto-Central Negativity (LFCN) with a latency of 320–400 ms, apparent at fronto-central electrodes and some posterior sites. Color change of that irrelevant stimulus also slowed identification of a visual target, indicating distraction. The amplitude of this color-change vMMN, but not LFCN, indexed this distraction effect. That is, electrophysiological and behavioral measures were correlated. The interval between visual scenes approximated 1 s (611–1629 ms), indicating that the brain's sensory memory for the color of the preceding visual scenes must persist for at least 600 ms. Therefore, in the case of the neural code for color, durable memory representations are formed in an obligatory manner.

Introduction A lurid and influential theoretical claim has been made that vision has no memory in excess of 100 milliseconds: “Vision has no memory, it exists in the present tense” (Wolfe, 2000). Evidence for this claim about sensory memory has stemmed from demonstrations that a large change in a visual scene, or the sudden onset of a visual object, can go unnoticed – phenomena, such as change blindness (O'Regan et al., 1999), inattentional blindness (Mack and Rock, 1998), and the attentional blink (Shapiro et al., 1997). Evidence from a flicker version of the change blindness paradigm (Rensink et al., 1997) suggests that sensory memory for the visual stimulation is thought to not even persist for 80 ms, unless stimulation receives some form of extensive attentional processing (Rensink, 2002). Accordingly, as in the inattention blindness and attentional blink paradigms, vision is shown to have no sensory memory. On the other hand, the change detection process in vision is manifested as the visual mismatch negativity (vMMN) component of the event-related potential (ERP). This vMMN is typically elicited in a visual oddball paradigm when a repeated standard visual stimulus, such as a red square standard, is unpredictably and occasionally replaced by a deviant stimulus that differs from the standard stimulus by one feature, such as color, e.g., a green square deviant. Importantly, the inter-stimulus intervals between presentation of standard and deviant exceed 100 ms by far, suggesting that some representation of the standard persists. Such representation seems not to depend on attentional processing. The vMMN is also elicited when participants are ignoring the vMMN-eliciting features while attending to other aspects of the visual stimulation (Berti and Schröger, 2001, Berti and Schröger, 2004, Berti and Schröger, 2006, Kimura et al., 2008a and Kimura et al., 2008b). The existence of vMMN, a scalp-elicited posterior bilateral negativity in response to visual deviance, has remained, until recently, a controversial topic (Näätänen, 1990, Näätänen, 1991, Cammann, 1990, Czigler, 1990, Pazo-Alvarez et al., 2003 and Heslenfeld, 2003). However, throughout the past decade and into the current, a multitude of independent replications of the vMMN have placed vMMN upon a firm empirical footing (Astikainen and Hietanen, 2009, Astikainen et al., 2004, Astikainen et al., 2008, Berti, 2011, Clifford et al., 2010, Czigler, 2007, Czigler et al., 2004, Czigler et al., 2007, Czigler and Pato, 2009, Czigler and Sulykos, 2010, Fisher et al., 2010, Flynn et al., 2009, Kimura et al., 2010a, Kimura et al., 2010c, Kimura et al., 2010d, Liu and Shi, 2008, Lyyra et al., 2012, Maekawa et al., 2009, Mao et al., 2004, Stefanics et al., 2011, Stefanics et al., 2012, Shtyrov et al., 2013, Sulykos and Czigler, 2011, Sušac et al., 2004, Sušac et al., 2010a, Sušac et al., 2010b and Sušac et al., 2011; for reviews, please see Kimura et al., 2011, Kimura, 2012 and Winkler and Czigler, 2012). The vMMN is believed to be an analog of the more well-studied auditory MMN (Näätänen et al., 1978 and Tiitinen et al., 1994), elicited at similar latencies and largely pre-attentively as well (Näätänen et al.; for a complementary perspective, see Erlbeck et al., 2015 and Campbell, 2015). Yet functional differences could also exist between modalities — a key difference relating to the durability of the form of internal sensory memory representation indexed by vMMN. The orientation vMMN is elicited by an unexpected occasional change in orientation, occurring in response to visual stimuli separated by intervals of 200 ms, attenuating at intervals of 400 ms (Fu et al., 2003), and disappearing completely at intervals of 1100 milliseconds (Astikainen et al., 2008). Accordingly, the sensory memory responsible for the vMMN to orientation change is thought to only endure very brief intervals for vMMN. Thus the fleeting sensory memory for orientation in question is thought to have a duration of less than 1 s, as contrasts with estimates of 4–10 s for the pitch mismatch negativity in the auditory domain (Bottcher-Gandor and Ullsperger, 1992). However, the sensory memory for color may be considered more enduring: vMMN to color deviance can be elicited after intervals as long as 800 ms (Stefanics et al., 2011) albeit attenuated relative to a shorter interstimulus interval (ISI). The question of the duration of the to-be-ignored visual stimuli has remained open. The value of an internal sensory memory representation for visual information has been subject to debate (Kimura et al., 2010b and O'Regan and Noë, 2001), the relatively static visual world typically being available as an external memory representation (Ballard et al., 1997). Indeed, the symbolic use of such external representations has been hailed as a major transition in human evolution (Donald, 1993), which compensates for the inherent limitations of working memory (see Miyake and Shah, 1999 for an overview). Visual distraction paradigms have been shown to be promising in that they have revealed behavioral disruptions of performance produced by visual deviance, alongside a significant vMMN (Berti and Schröger, 2001, Berti and Schröger, 2004, Berti and Schröger, 2006, Kimura et al., 2008a and Kimura et al., 2008b). Further, it has been shown that when the to-be-ignored background exhibits deviance in the same dimension as the to-be-attended figure (color, orientation), the disruption of performance produced by the background is increased; alongside a concomitant vMMN augment (Czigler and Sulykos, 2010). However, vMMN to color change has not been shown to index distraction upon an individual level of performance (e.g., Czigler et al., 2002), whereas position deviance was effective in eliciting a vMMN as well as a behavioral distraction effect (Berti, 2009). To account for this difference, Berti (2009) suggested that peripheral presentation of color deviance might be necessary. In the present investigation, the amplitudes of significant differences between individual scalp-measured ERPs are thus evaluated as indices of behavioral distraction effects produced by to-be-ignored visual deviance, with the objective of assessing the functional relevance of vMMN to color change. For this reason, a visual search task was adopted after Czigler et al. (2002); a paradigm which presented task-unrelated color deviance in the visual periphery, during which participants were required to look at a central fixation cross while searching for a unique target shape and ignoring a uniquely colored distractor within surrounding stimuli (Hickey et al., 2006). While the task of Czigler et al. (2002) did not prove sensitive to the distracting effects of color deviance, visual search distraction paradigms have been shown to be sensitive to color (Hickey et al., 2006). The present investigation aimed to evaluate the amplitudes of vMMN to color change of a distractor as indices of behavioral distraction, measured by the slowing of target processing. Crucially, the visual search task was extended by including a serial component: the distracting object differed in color from all other simultaneously presented objects, but that distractor occasionally changed in color within a sequence of trials. Hence, the effect of distraction was investigated in a series of visual search displays, where the distractor color was either a standard or a deviant color. Repetition of the uniquely colored “standard” distractor should facilitate the accumulation of a sensory memory trace for this color. To examine if the sensory memory mechanisms of the brain supported the detection of color change, an unexpected improbable “deviant” distractor (e.g., green) was employed. Note that this deviant differed in color from the preceding standard. Both the standard distractor and the deviant distractor differed in color from the remaining objects, which were blue. Visual Event-Related Potentials (ERPs) were derived from high-density EEG recordings to characterize the time-course of scalp-measured indications of the brain's responses to visual scenes containing deviant-colored distractors. A previously unexplored objective of the present investigation was thus to use these methods to build a new bridge between the discourses upon visual search and vMMN. If as evidence from attentional blink, inattention blindness, and change blindness paradigms have suggested, vision has no memory (Wolfe, 2000) of functional consequence, upon an individual level, then color deviance would not affect behavior in our serial distraction task. To examine whether vision has a sensory memory for color, the interval between presentations of visual arrays of objects was at least 600 ms. That is, if a sensory memory for the color of the preceding distractor existed, that memory must endure that interval for a color change to influence: (a) performance, and (b) the generation of brain processes.

3. Results 3.1. Color deviance causes a behavioral distraction effect Performance upon the task demonstrated that accuracies approached ceiling (97.92 ± 0.28%; mean ± SEM) and drove the effects of distraction into the speed domain. Mean RTs for visual search were slowed by the presence of a deviant distractor (1178 ± 85 ms) relative to the presence of a standard distractor (1155 ± 85 ms). This color deviance-related slowing was revealed by inferential statistical analysis to be significant via a paired t-test, t(11) = 2.5, p < 0.05, η2 = 0.354. To anticipate part of the results shown in Fig. 3, 10 out of 12 participants demonstrated a distraction effect, with the difference between mean RTs for standard minus deviant distractors being negative (pointing upwards in Fig. 3). A scatterplot of the behavioral distraction effect as a function of color vMMN ... Fig. 3. A scatterplot of the behavioral distraction effect as a function of color vMMN amplitude collapsed across significant electrodes. The stronger the slowing of responses due to the distraction, the stronger was the ERP difference due to color change. Figure options 3.2. Color deviance elicits two broadly distributed negativities There were clusters of electrodes that exhibited significant differences due to color deviance. Moreover, different clusters of electrodes exhibited significant differences at different points in time, each of which is considered separately. Depicted in Fig. 2a is a data reduction of the time course of significant differences between responses to visual scenes containing standard and deviant distractors. As illustrated, there were well-defined spatiotemporal clusters of activation responsive to the color change of the deviant distractor. While the negativities were of key theoretical interest, all significant clusters are discussed in the order in which they occurred. The first cluster, upon which no theoretical weight is placed, was a positivity. This early Change-Related Positivity or CRP (Kimura et al., 2006a, Kimura et al., 2006b, Busch et al., 2010 and Stefanics et al., 2011; for a kindred phenomenon see Berti, 2011) was apparent at left centro-parietal sites at the latency of 70–110 ms within the time-range of the P1 wave. However, the small size of the effect would suggest that the paradigm utilized in the present investigation was not optimal for CRP elicitation. The second and third clusters demonstrated two reliably measured negativities: (1) the early color vMMN at the latency of 120–160 ms, apparent at left posterior sites, and (2) a Late Fronto-Central Negativity (LFCN), at a latency of 320–400 ms, apparent at fronto-central electrodes and some posterior sites. ERPs to standards and deviants and the corresponding difference waves for both negativities are illustrated in Fig. 2b, at selected electrodes; shaded areas between ERPs denoting time periods when the cluster was active. As depicted in Fig. 2b, the grand-averaged peak amplitude of color vMMN at O1 was at 141 ms, when the sample-based t-value of the difference wave was maximal, t(11) = 8.33, p < 0.001, η2 = 0.86, while that of LFCN at FT8 was at 339 ms, t(11) = 4.01, p < 0.01, η2 = 0.59. The distinct topography of the difference waves during the time period of these negativities is illustrated by isopotential maps for each cluster in Fig. 2c. 3.3. Color vMMN predicts the extent of behavioral distraction While significant clusters of activity were elicited by color change of the distractor, it remained to be determined whether neurophysiological activity could be related to the behavioral effect of the distraction. In fact, a correlational analysis of this distraction effect with the deviant-standard difference for each cluster revealed that the color vMMN (120–160 ms) showed a marked relation: behavioral slowing increased with the amplitude of the color vMMN, as was confirmed by significant correlations, r(10) = 0.81, p = 0.002. Fig. 3 depicts this significant increase in distraction effect as a function of color vMMN amplitude averaged across the whole cluster. The linear regression predicting distraction from the color vMMN was also significant, F(1, 10) = 16.52, p < 0.002, such that the slope differed significantly from zero, t(10) = 4.303, p = 0.002. By contrast, the amplitudes of CRP and LFCN clusters did not correlate with the distraction effect, r(10) = − 0.09, p = 0.79 and r(10) = 0.12, p = 0.67 respectively for CRP and LFCN. Accordingly, the amplitude of color vMMN indexed the extent of the distraction effect. As already pointed out, Fig. 3 depicts that 10 out of 12 participants demonstrated a distraction effect, in addition the Figure shows that 11 out of 12 volunteers demonstrated a color vMMN. Note that this ERP component, at a latency of around 120–160 ms, predicted a manual response that emerged several hundred milliseconds later. 3.4. Color vMMN: distribution, duration of sensory memory trace, and visual hemifield The effect of distraction on ERP and behavioral response times indicated that the sensory memory representation for the preceding stimuli was still vivid when the distractor occurred. The ISI between search arrays varied from 610 to 1629 ms. An auxiliary question was thus whether this ERP index of the sensory memory trace for distractor color attenuated as the ISI increases. To address this question, epochs time-locked to the visual search array were binned upon the basis of the duration of the preceding fixation interval (short: 610 to 1110 ms vs. long: 1110 to 1629 ms). A further auxiliary question concerned the hemifield of presentation. Previous investigations of vMMN have addressed the influence of the visual hemifield in which the visual deviance is presented: some investigations have reported a vMMN that is confined to deviance presented in the lower hemifield and not the upper hemifield (Czigler et al., 2004 and Müller et al., 2012), while other investigations revealed the converse (Berti, 2009). Accordingly, epochs binned upon the basis of ISI were further partitioned into subgroups according to whether deviants were presented in the upper or lower hemifield. Of further interest was the distribution of the color vMMN that was examined by comparison of its amplitude in a window of integration 120–160 ms after the search array at the left and right occipital electrodes. These prior observations and questions of theoretical interest motivated the ensuing analyses of this section. As depicted in Fig. 4, the difference of ERP amplitudes integrated across the color vMMN temporal window (120–160 ms), demonstrated a strong color vMMN with a short ISI, in a manner that was only apparent over the left hemisphere (O1) and upper hemifield. Such a strong color vMMN was neither present over the right hemisphere (O2) nor even at O1 with a long ISI. This pattern of mean differences suggested that the color vMMN depicted in Fig. 2 may have been caused by a left-distributed posterior response to color deviance in the upper hemifield with short ISIs. To foreshadow the results of the analysis, this observation was statistically supported. Mean amplitude of color vMMN as a function of ISI (x-axis), visual hemifield ... Fig. 4. Mean amplitude of color vMMN as a function of ISI (x-axis), visual hemifield (upper — black, lower — white), and hemisphere (left vs right panels). Figure options A 4-way repeated-measures Analysis of Variance (ANOVA) with a 2 (Color deviance: standard, deviant) × 2 (ISI: short vs. long) × 2 (Visual Hemifield: upper, lower) × 2 (Hemisphere: left at O1, right at O2) performed on the ERP amplitude data from this window of integration confirmed these observations. The main effect of color deviance was marginal, F(1, 11) = 4.33, p = 0.061, η2 = 0.283, although a significant Hemisphere × Color Deviance interaction, F(1, 11) = 14.66, p = 0.003, η2 = 0.571, corroborated cluster analysis results ( Fig. 2a) of a left-distributed color vMMN. The significant ISI × Hemifield × Hemisphere × Color Deviance interaction, F (1, 11) = 6.10, p = 0.031, η2 = 0.357, revealed that the left-lateralized color vMMN varied significantly as a function of hemifield and interstimulus interval. The other significant ISI × Hemifield × Hemisphere and ISI × Hemifield interactions, p < 0.05, suggested that brain response in the 120–160 ms time range is influenced by ISI, Hemifield and Hemisphere, irrespective of Color Deviance, although these results should be regarded with caution due to significance of higher-order interactions. No other effects were significant, Fs < 1, ps > 0.107. The predicted effect of Color Deviance was investigated separately for each ISI, separately for distractors in each Visual Hemifield, separately for each Hemisphere via 8 pairwise critical planned comparisons using Holm's adaptation of the Bonferroni test. These analysis revealed that the effect of color deviance was only significant for the “short” ISI at the “upper” level of visual hemifield for the “left at O1” level of hemisphere, t(11) = 3.87, tα/8 = t0.00625 = 3.30, all other ts < 1. That is, as depicted in Fig. 4, the color vMMN was only significant over the left hemisphere when the color deviant was presented in the upper hemifield and with a short ISI (< 1100 ms), as exhibited an effect size of 1.58 standard deviations. 3.5. Color vMMN: ruling out the lateralized component hypothesis A hypothesis considered was that a potential lateralization of the component of interest was due to 10 out of 12 of the color distractors being lateralized, even though distractors were equally often presented in the left visual field and right visual field. A corollary of this possibility was that there could have been an incidental rejection of more artifact epochs when color deviant distractor stimuli were presented in the left side of the search array than in the right side of the search array. To rule out this possibility, a subset of standard and deviant arrays – those scenes containing only vertical distractors – were included in an additional analysis. Vertical distractors were presented in a non-lateralized manner on the vertical meridian at a position above or below the fixation cross, i.e., a 6 or 12 o'clock position. Isopotential maps for the component of interest were compared visually and confirmed using paired Student's t-tests: the significant color vMMN was present only at sites over the left hemisphere (O1, PO7, P9, P5, CP5, p > 0.05, Fig. 5), as out-ruled this lateralized component hypothesis that the left distribution of color vMMN was a byproduct of the lateralization of distractors. Isopotential maps of color vMMN only for vertical distractors. The difference ... Fig. 5. Isopotential maps of color vMMN only for vertical distractors. The difference wave was derived from subtraction of measured ERPs to standard distractor from the ERP to those containing a deviant distractor and integrated across a 120–160 ms temporal window. Electrodes highlighted in white exhibited significant differences between standard and deviant (ps < 0.05). Note the left-distribution. Figure options 3.6. Color vMMN: ruling out N1 modulation and the latency shift hypotheses The infrequent deviant stimuli might have elicited a higher amplitude of N1, compared to frequent standard stimuli, since the state of refractoriness of afferent neuronal populations that specifically respond to the feature values of deviant stimuli is lower than that of afferent neuronal populations that respond to the feature value of standard stimuli (see e.g., Kimura, 2012). To rule out this N1 modulation hypothesis a comparison of the color vMMN topography with those of standard ERP from the color vMMN time range (120–160 ms) was necessary. As depicted in the first row of Fig. 6 these topographies exhibited marked differences. Isopotential maps of the difference wave derived from subtraction of measured ... Fig. 6. Isopotential maps of the difference wave derived from subtraction of measured ERPs to visual scenes containing standard distractors from the ERP to those containing a deviant distractor, with a window of integration 120–160 ms, isopotential maps of ERP for standard distractors, again integrated over 120–160 ms, and isopotential maps of the artificial difference wave derived from the subtraction of the measured ERP to visual scenes containing a standard distractor from an artificially time-lagged copy of that ERP also with a window of integration 120–160 ms. Please note that the topography of color vMMN over the left hemisphere is not exhibited in the distribution of the difference waves produced by these artificial latency shifts. Figure options Additional analyses tested if the color vMMN component was, rather, a latency shift of one of the neighboring ERP waves, namely the P1 and N1 waves. This “latency-shift hypothesis” assumed that color vMMN might be a byproduct of a P1 or an N1 latency shift. Here, O1 was the electrode of interest, as this site was where the color vMMN exhibited the optimal signal-to-noise ratio. In all but two of the included volunteers, the P1 peak was evident as a positive ERP deflection 90–140 ms post-stimulus onset, followed by a distinct N1 wave in the time range 127–194 ms, as was exhibited in all but one of the volunteers. Subtle numerical differences in the latencies of P1 peaks (standard: 115.5 ± 3.1 ms, mean ± SEM; deviant: 115.8 ± 3.4 ms) and N1 peaks (standard: 161.4 ± 6.4 ms; deviant: 157.1 ± 5.5 ms) at the O1-electrode were found not to be reliable via paired two-tailed Student's t-tests, p > 0.05. It was also worth considering that the grand-averaged P1 (115 ms) and N1 (159 ms) peak latencies were either outside of, or bordering upon the temporal interval of significant amplitude differences, as was revealed by paired t-tests in the color vMMN time range (120–160 ms, see Fig. 2). As peak latencies did not necessarily reflect all aspects of P1 and N1 generation, a further test of the latency shift hypothesis was necessary. We assume that the observed peaks in the ERP correspond to the real peaks of the underlying ERP component, which is not always the case (Luck, 2005a and Luck, 2005b). Here, the idea was that if color vMMN reflected a latency shift of the N1 or P1 wave, then the artificial subtraction wave representing the difference between the original standard ERP and the standard ERP time-lagged by the hypothetical latency shift would have the scalp distribution of the color vMMN within the component's time range. Thus, an artificially time-lagged version of the ERP waveform to the visual scenes containing a standard distractor (lags: − 30, − 20, − 10, − 4, 4, 10, 20, 30 ms) was subtracted from the originally measured ERP waveform in response to that standard distractor. The isopotential maps of this difference wave were plotted. The results of this approach are illustrated in Fig. 6, where the scalp distribution maps for the color vMMN time window, 120–160 ms, are presented. If the latency-shift hypothesis was valid, and a P1 or N1 latency shift caused the color vMMN effect, the scalp topography would have a pronounced distribution over the left hemisphere. As is depicted in Fig. 6, such a hypothetical dominance of the distribution over the left hemisphere was unapparent. The artificial latency shift of the standard ERPs produced a rather symmetric distribution with even a subtle tendency towards a higher voltage over the right, rather than the left hemisphere, as can be seen in Fig. 6. As has already been mentioned, under the assumption that the time-course of the components mirrors the timing of the underlying ERP exactly, these auxiliary analyses ruled out the latency shift hypothesis that color vMMN simply represented a latency shift between responses to standard and deviant stimuli. Therefore these analyses tentatively support the hypothesis that the elicited color vMMN was a distinct component and not a byproduct of P1 or N1 latency shifts.