به سوی یک مدل شناختی از حواس پرتی با اخبار شنوایی: نقش جلب توجه غیر ارادی و پردازش معنایی

Abstract Unexpected auditory stimuli are potent distractors, able to break through selective attention and disrupt performance in an unrelated visual task. This study examined the processing fate of novel sounds by examining the extent to which their semantic content is analyzed and whether the outcome of this processing can impact on subsequent behavior. This issue was investigated across five laboratory experiments in which participants categorized visual left and right arrows while instructed to ignore irrelevant sounds. The results showed that auditory novels that were incongruent with the visual target (e.g., word “left” presented before a right arrow) disrupted performance over and above congruent novels (semantic effect) while both types of novels delayed responses in the visual task compared to a standard sound (novelty effect). No semantic effect was observed for congruent and incongruent standards, suggesting that novelty detection is necessary for involuntary semantic processing to unravel. While the novelty effect augmented as the difference between novels and the standard increased, the semantic effect was immune to this variation. Furthermore, the novelty effect decreased across the task while the semantic effect did not. A general cognitive framework is proposed encompassing these new findings and previous work in an attempt to account for the behavioral impact of irrelevant auditory novels on primary task performance.

Introduction Efficient cognitive functioning often requires selective attention, that is, the ability to filter out task-irrelevant stimuli in order to concentrate on the task at hand. While resisting distraction can be desirable in many circumstances, possessing brain mechanisms able to break through selective attention and detect unexpected but potentially important stimuli is crucial from an adaptive perspective. To the early hominidaes, the brisk sound of a bush’s moving leaves may have indicated the location of potential food or that of a predator. To us, the sound of screeching tyres might signify an imminent danger that one would want to be able to detect even if deeply concentrated reading our preferred journal. A trade-off between selective attention and auditory novelty detection is therefore advantageous. The interplay of these two phenomena can be studied in the laboratory and has primarily been investigated from an electrophysiological perspective. The cognitive analysis of the mechanisms involved in cross-modal attention capture remains, in comparison, largely unexplored. A recent study showed that rare auditory stimuli (referred to as novels) among otherwise repeated sounds (referred to as standard) distract performance in an unrelated visual task not through the disturbance of target processing but as a result of dynamic shifts of attention to and from the novel sound ( Parmentier, Elford, Escera, Andrés, & San Miguel, 2008). This study seeks to extend this cognitive examination and address an important, yet little researched, issue: what is the fate of the novel sounds once oriented to? More specifically, the results of five experiments examine whether novel sounds are semantically processed outside one’s voluntary control and whether such processing can impact on one’s subsequent behavior. Finally, a theoretical framework will be proposed, aiming to fathom the cognitive mechanisms underpinning the cross-modal distraction induced by auditory oddballs. First, I begin with a brief review of research on the involuntary detection of novel and deviant sounds and the resulting attention capture and orientation response (Sokolov, 1966). 1.1. The involuntary detection of novel sounds Numerous studies have established that one’s attention can be involuntarily captured by sudden changes (oddball, novel, or deviant) in a train of otherwise repeated sounds (standard). This type of attention capture has mostly been studied from an electrophysiological perspective and is characterized by a pattern of three specific brain responses ( Schröger, 1997, Schröger and Wolff, 1998a and Schröger and Wolff, 1998b): mismatch negativity (MMN) and an enhanced N1 when the distractor deviates a great deal from the repetitive background ( Alho et al., 1998), P3a (sometimes referred to as novelty P3; see Friedman, Cycowicz, and Gaeta (2001) for a review) and re-orientation negativity (RON; e.g., Schröger & Wolff, 1998b). The MMN response reflects the pre-attentive detection of an unexpected change (or even the illusion of a change, see Stekelenburg, Vroomen, and de Gelder (2004)) in the auditory context, and follows from the comparison between a memory trace for past acoustic stimuli and the current auditory signal (Cowan et al., 1993, Näätänen and Winkler, 1999, Picton et al., 2000 and Winkler and Cowan, 2005). The P3a response represents the involuntary orientation of attention towards the novel sound (Friedman et al., 2001, Grillon et al., 1990 and Woods, 1992) and is thought to result from an attentional interruption involving frontal areas (Opitz, Rinne, Mecklinger, von Cramon, & Schröger, 2002). A re-orientation negativity is also observed when participants are performing a primary task and must redirect their attention towards that task (Berti et al., 2004, Berti and Schröger, 2001, Escera et al., 2001 and Schröger and Wolff, 1998a). Behaviorally, responses in a primary task are delayed following the presentation of a task-irrelevant novel sound. For example, MMN-eliciting deviants presented among standards to an unattended ear delay participants’ response to auditory targets in the other ear (Schröger, 1996). Schröger and Wolff, 1998a and Schröger and Wolff, 1998b reported similar findings in a task in which participants had to discriminate between long and short sounds irrespective of their frequency. Even though frequency was irrelevant to the participants’ discrimination task, response latencies in the primary task were significantly longer for rare frequency deviants relative to standards (see also Berti and Schröger (2003) and Roeber, Berti, and Schröger (2003)). Such studies used the so-called one-channel paradigm in which targets and deviants are presented auditorily, usually as distinct features of the same auditory object (e.g., frequency and duration). Remarkably, however, target and irrelevant stimuli need not be presented at the same time or in the same sensory modality for distraction to occur, highlighting the a-modal nature of the attention capture phenomenon. Indeed, performance in a visual discrimination task is also affected by auditory deviants or novels (two-channel paradigm), as demonstrated in studies using the cross-modal oddball task (e.g., Andrés et al., 2006, Barceló et al., 2006, Escera et al., 1998, Escera et al., 2002 and Jääskeläinen et al., 1996). In this task, participants categorize visual digits, presented in sequence, as odd or even. Each digit is preceded by a task-irrelevant sound that participants are instructed to ignore. In most trials (e.g., 80% of trials), this sound is repeated (standard) while in the remaining trials it is replaced by a deviant or a novel sound. Behavioral distraction consists in the lengthening of response latencies in the visual task, following deviants or novels compared to standards. In studies using the cross-modal oddball task, the primary interest is typically in the electrophysiological responses to the novel sound while behavioral distraction is secondary and usually not discussed in great depth. From a cognitive point of view, however, one may wonder exactly why hearing a task-irrelevant novel sound should disrupt the subsequent categorization of a visual digit. To answer this question, Parmentier et al. (2008) reported experiments designed to pinpoint the cognitive locus of distraction in the cross-modal oddball task. Their hypothesis was that if distraction was due to a reduction in the amount of attentional resources available to process the visual digits, then making this processing more demanding should amplify distraction. However, neither the visual degradation of the digits nor the use of more difficult categorization instructions affected distraction (as measured by the difference in response latency between novel and standard trials), despite sizable main effects of these manipulations. The authors suggested that distraction was therefore not the result of a slower processing of the visual targets. Instead they argued that it reflected a delay in initiating their processing that resulted from time penalties associated with involuntary shifts of attention to and from the novel sound. In line with this interpretation, the results of their Experiment 3 showed that distraction in the cross-modal oddball task could be eliminated by forcedly re-capturing attention after a novel sound by presenting a task-irrelevant visual distractor (characterized by an abrupt visual and motion onset) prior to the appearance of the visual target. The distractive effect of deviants on behavioral performance in an unrelated task is not specific to the cross-modal oddball task. For example, Hughes, Vachon, and Jones (2005) reported that serial recall performance for visual verbal stimuli was reduced when an auditory temporal deviant was introduced among a sequence of irrelevant sounds presented concurrently with the sequence of to-be-remembered visual stimuli (see also Lange (2005)). Interestingly, data suggest that the distractive effect of a single deviant is functionally distinct from that caused by continuous changes in an irrelevant sound stream (the irrelevant sound effect, or ISE, which reflects a clash between seriation processes applied to relevant and irrelevant sequences; Jones et al., 1999, Jones and Macken, 1993, Jones and Macken, 1995, Jones et al., 1993, Jones et al., 1992 and Jones et al., 1999). Indeed, Hughes, Vachon, and Jones (2007) found that (1) deviants only affect serial recall performance when presented concurrently with the to-be-remembered stimuli and not when presented during a retention interval (while the ISE is observed in both situations) and (2) that deviants disrupt performance irrespective of whether participants are tested with an order reconstruction task or a missing item task (which does not involve serial memory; Murdock, 1993) while the ISE is observed with the first but not the latter task (Beaman & Jones, 1997). 1.2. Involuntary orientation of attention to novel sounds: to what aim? The cognitive analysis of distraction as described above has a number of implications. One is that distraction may occur, at least partly, irrespectively of the nature of the primary task, since it appears to relate to the shifts of attention to and from an auditory novel prior to the processing of an upcoming target stimulus. For example, distraction in the cross-modal oddball task is observed not only in a digit categorization task but also in a living/non-living judgment of pictures (e.g., Van Mourik, Oosterlaan, Heslenfeld, Konig, & Sergeant, 2007). To claim that the processing of visual targets per se could never be affected by auditory novelty would be premature, however. Indeed, we do not currently know whether novels are processed to any depth and whether, in situations where the nature of the novels may have particular relevance to participants or relate to the task set involved in a primary task, the locus of distraction may be the processing of a target stimulus. From an evolutionary perspective, if attention capture and novelty detection systems have evolved to alert us of possible dangers or stimuli of potential interest, one would reasonably expect that some degree of analysis of the novels should occur in order to guide an appropriate action by the organism. While there is, to my knowledge, no systematic study examining whether task-irrelevant novel sounds are analyzed in a way that may affect subsequent behavior, there are, however, studies indicating that a MMN response can be triggered by changes more complex than acoustic ones. For example, phonological (Dehaene-Lambertz, 1997 and Winkler et al., 1999), lexical (Jacobsen et al., 2004), semantic (Shtyrov, Hauk, & Pulvermüller, 2004), grammatical (Pulvermüller & Shtyrov, 2003), and even syntactic (Hahne, Schröger, & Friederici, 2002) changes have been found to yield MMN. Studies using sounds of varying semantic salience have reported that the latter can modulate the MMN response. For example, sexually suggestive whistle sounds yield a larger MMN response than meaningless (but acoustically similar) whistle sounds, regardless of whether participants attend to the sounds or not (Frangos, Ritter, & Friedman, 2005). Such studies suggest that a variety of discontinuities between past auditory events and novels of particular semantic salience are picked up very early by the brain. Pulvermüller (2001) has argued that the presentation of an oddball word can activate vast cortical networks involving assemblies of neurons coding for perceptual features as well as actions associated with that word through correlational learning, and that such patterns of spreading activation correspond to what we commonly call the “meaning” of a word. It is worth noting, however, that the above MMN studies used deviants defined by semantic but also physical characteristics. As a result, it is not possible to establish that the semantic deviancy per se trigger change detection. Indeed it is possible that change detection is initially triggered by physical deviancy, with semantic processing occurring as a consequence (as suggested, for example, by Wetzel and Schröger (2007)). Furthermore, the aforementioned MMN studies typically lack a measure of behavior through which the distraction caused by novel sounds can be measured, however. As such, they do not address the question at stake in this study, namely whether the analysis of novel sounds can impact on later behavior. Only one study found evidence of the unintentional lexico-semantic processing of novels independently of a co-occurring acoustic change (Muller-Gass, Macdonald, Schröger, Sculthorpe, & Cambell, 2007). Muller-Gass and colleagues asked participants to judge the prosody of words and pseudowords (each of which served as standards and deviants across different blocks of trials). Acoustic change across auditory stimuli was constant while the violation of the lexical status was rare. This violation yielded a positive enhancement about 165 ms following the sound’s onset (and maximal at Cz) as well as a later negative enhancement at about 375 ms (maximal at Pz). While this finding suggests that deviants are analyzed from a lexical point of view, two remarks are warranted. First, such a result does not speak to the issue of whether lexical activations can progress to higher levels and interfere with the processing of subsequent stimuli. Secondly, it may be that the lexical changes were detected because this level of stimulus complexity was the simplest one shared by the standards. This does not mean that any deviant would incur lexico-semantic processing on the basis of violations other than lexical. Few studies have examined systematically the effect of the novels’ semantic content on the orientation (P3a) response, and their results are somewhat discrepant. Mecklinger, Optiz, and Friederici (1997) found that identifiable and unidentifiable sounds (as rated by an independent group of participants) produced similar novel P3 responses when participants did not attend to the sounds and performed a visuo-motor tracking task. The authors did not report the performance in the visuo-motor task, however, so it is not clear whether novel sounds had a distractive impact on behavior. Escera, Yago, Corral, Corbera, and Nuñez (2003) carried out an experiment using the cross-modal oddball paradigm. The novel sounds were environmental sounds that participants were later asked to rate with respect to their identifiability. The authors then retrospectively analyzed brain activity (MMN, P3a, and RON) separately for identifiable and unidentifiable novels. The results showed a larger P3a response to the identifiable compared to the unidentifiable novels when participants covertly attended to the novel sounds but not otherwise, leading the authors to suggest that the above difference must reflect the semantic analysis of the novel sounds and not merely a difference between identifiable and unidentifiable sounds in the effectiveness with which they elicit an attention trigger (since N1 and MMN responses were similar for the two sound types). With respect to behavioral performance, participants were significantly delayed in the digit categorization task following an identifiable novel compared to an unidentifiable. The authors concluded that the “semantic analysis of significant stimuli takes place only after a presumably transitory attention switch to the eliciting stimuli” (p. 2411). Such a suggestion was echoed by Wetzel and Schröger (2007), who reported that long environmental sounds produced, relative to shorter environmental sounds, enhanced early and late P3a responses, as well as a post-P3a (400–600 ms) response. The authors suggested that one possible explanation of this effect may be that novels capture attention but that the amount of information they carry (assumed to be higher if longer) mediates the amount of semantic evaluation of these sounds. They concluded that “this (possible) semantic effect ‘hooks’ on the P3a initiated by the initial part of the novel. This may then, in turn, trigger additional higher-level analyses which prolong/enhance the attention switch indicated by P3a” (Wetzel & Schröger, 2007, p. 30). Since the participants’ task was passive, no behavioral data were offered allowing one to determine whether this additional processing does translate in longer response latencies. The study by Escera et al. (2003), while suggesting that identifiable novels delay responses in the visual task more than unidentifiable novels (in line with the finding that distraction in this paradigm results from a delayed onset of target processing, Parmentier et al., 2008) does not allow one to examine whether the semantic analysis of novel sounds can yield activations in memory able to interfere with the participants’ behavior in a subsequent unrelated task. Finally, in a study comparing ERPs for aversive (e.g., cry of babies) and non-aversive everyday sound novels, Czigler, Cox, Gyimesi, and Horváth (2007) found that the aversive and everyday sounds differed insofar as the former yielded a larger negative wave (parietal and central) in the late part of N1 and early part of P3a, an anterior positivity in the descending limb of the P3a and a later positivity (450–650 ms) which was interpreted as the cognitive evaluation of the novel. Because participants attended to the sounds in order to detect a deviant target, this study does not provide evidence of the involuntary semantic analysis of the novels’ content. Furthermore, it does not offer a measure of the potential behavioral impact of such analysis on performance in a subsequent task. 1.3. The automaticity processing of task-irrelevant stimuli While studies of attention capture by novelty have not systematically examined whether auditory oddballs are semantically evaluated, numerous studies suggest that vast semantic networks (including action responses) are activated by verbal stimuli even when their semantic processing is irrelevant to the task at hand. For example, Glenberg and Kaschak (2002) reported that participants asked to judge whether sentences made sense responded faster when the response required was compatible with an action associated to the sentences’ meaning. A sentence such as “open the drawer”, implying an action toward the body, was judged faster when the “yes” response button was located in the participants’ near space than further away; while the reverse was observed for sentences affording actions away from the body (e.g., “close the drawer”). Zwaan and Taylor (2006) extended this type of finding to manual rotations, showing that sentences affording a rotation in a specific direction were responded faster to when the response require a compatible manual rotation. Furthermore, sentence comprehension was facilitated when sentences were presented with a visual stimulus rotating in a compatible direction. Recent imaging data indicate that word meaning is closely related to action. Hauk, Johnsrude, and Pulvermüller (2004) showed that, in a passive reading task, the brain activity elicited by face, hand, and leg-related action elicited activations of motor and pre-motor networks activated by actual movements of the tongue, fingers, and feet, respectively. Such findings fit well with some theoretical integrative frameworks relating perception and action (Hommel, Müsseler, Aschersleben, & Prinz, 2001; Prinz, 1990). Crosstalk interference has also been examined across modalities and for asynchronic stimuli. For example, auditory word and letter primes can facilitate or interfere with the naming of a subsequent visual picture (Brooks and MacWhinney, 2000 and Jerger et al., 2002), or the identification of visual letters (Larsen, McIlhagga, Baert, & Bundesen, 2003). Perhaps the clearest example of semantic interference between auditory and visual stimuli is that observed in the cross-modal Stroop task. In that task, participants are asked to name colors while trying to ignore incongruent color words presented over headphones (Elliott and Cowan, 2001, Elliott et al., 1998, Hanauer and Brooks, 2003 and Roelofs, 2005). In this paradigm too, interference is observed when auditory distracters precede the visual targets, although it appears to peak when both stimuli are presented simultaneously (Roelofs, 2005). Finally, some studies of the so-called irrelevant speech effect (the detrimental effect of to-be-ignored auditory stimuli on the recall of visual information presented sequentially) provide further evidence that the auditory distractor can undergo involuntary semantic evaluation. Neely and LeCompte (1999) found that participants recalling (free recall) visually presented exemplars of a specific category were more disrupted by irrelevant auditory stimuli related to that category than by unrelated irrelevant stimuli. Furthermore, related irrelevant stimuli increased the probability of to-be-ignored words intruding on the participants’ responses. Beaman (2004) replicated these findings and showed that participants with low working memory capacity are more prone to produce such semantic intrusions. More recently, Marsh, Hughes, and Jones (2008) reported further empirical evidence of the pre-attentive semantic processing of irrelevant auditory stimuli and suggested that irrelevant stimuli, when presented concurrently with a task involving semantic processes, yields traces which then compete for selection with to-be-recalled stimuli. The studies cited above, using various tasks, all suggest that semantic aspects of stimuli can undergo some level of analysis and, in some circumstances, yield memory activations able to compete for selection with task-relevant stimuli. Based on these studies, one may hypothesize that auditory novels too may be subject to semantic processing. The nature of this processing and its potential impact on subsequent behavior are at the center of the present investigation. 1.4. Rationale for this study In this study, I sought to extend the cognitive analysis of distraction in the cross-modal oddball task (Parmentier et al., 2008) by examining whether the content of a novel sound impacts on distraction. Existing studies of the involuntary capture of attention have demonstrated that an effective auditory distractor must differ from the standard sound and must be rare and unpredictable (Näätänen, 1990). The cognitive fate of novel or deviant sounds remains relatively unexplored, however, and despite evidence in other paradigms that semantic processing can occur pre-attentively. The questions at stake here are these: does the capture of attention by a novel sound trigger its semantic analysis? If so, can the outcome of this analysis impact on our behavior? These questions were addressed using a cross-modal oddball task designed to test whether performance in the primary task may be affected by the meaning of novel sounds.

Conclusions This study examined the processing fate of auditory novels and the extent to which the involuntary orientation response to these stimuli is accompanied by a lexico-semantic analysis of their content. The results of four experiments clearly indicated that in a situation in which participants are engaged in a visual task requiring the directional categorization of visual arrows, performance was affected by the congruency between the auditory novels and the visual target, reflecting the semantic processing of the former. It was argued that the impact of novels on behavioral performance in the visual task reflected two distinct effects, namely a novelty effect and a semantic effect. The first appears to be sensitive to practice and the degree of featural mismatch between the participant’s mental representation of the standard sound and the novel, and follows from the detection of a change in the auditory environment. The semantic effect, on the other hand, reflects crosstalk interference between conflicting activations and marks the involuntary analysis of the novel’s lexical and semantic characteristics. This effect was not, contrary to the novelty effect, modulated by the dissimilarity between standard and novel, or practice. The effects were captured within an overall cognitive framework describing the trajectory of attention across modalities, task sets and spatial locations.