“Bilingual subtitles, a unique subtitle mode that presents subtitles in two different languages simultaneously, are gaining popularity around the world, especially in Mainland China (Li 2016). This is partly attributed to the belief that bilingual subtitles could deliver the benefits of both intralingual and interlingual subtitles, with intralingual subtitles providing the written forms of spoken words that can facilitate vocabulary learning and interlingual subtitles providing the meaning (translation) of words that can enhance viewers’ comprehension and absorption of the content (García 2017).
There exists however an inherent risk that subtitles, as a written form of spoken dialogue, generate redundant information that may overburden the visual processing channel and deplete people’s limited cognitive resources that could have been used to process other essential information (Zheng et al. 2016). When watching subtitled videos, viewers have to cope with a rich combination of multimodal and multiple-source information: visual images (visual-nonverbal), spoken dialogue (audio-verbal), subtitles (visual-verbal) and background sounds (audio-nonverbal) (Gottlieb 1998). This could place high demands on viewers’ attentional and cognitive resources because processing too much information simultaneously has been shown to exceed the capacity of working memory and result in cognitive overload (Kalyuga et al. 1999).”
“Compared with monolingual subtitles, watching videos with bilingual subtitles could be more cognitively demanding due to the concurrent presence of subtitles in two different languages, which, if the audience understands both languages, is likely to generate more redundancy and impose additional cognitive load on working memory. However, due to the scant research in this field, little is currently known about the actual visual and cognitive processing of bilingual subtitles.”
“The conceptualisation of cognitive load has been well-established in CLT (Sweller et al. 2011), one of the most influential theoretical frameworks accounting for cognitive processing during learning (Martin 2014). As this study approaches bilingual subtitles for the purpose of enhancing learning, we argue that CLT is a most appropriate theoretical framework for the current study since it has a well-established empirical basis in educational psychology, instructional design, and educational technology.
Three components of cognitive load have been identified in CLT literature, namely intrinsic cognitive load, extraneous cognitive load and germane cognitive load. Intrinsic cognitive load is created by dealing with the inherent complexity of the task (Van Merriënboer and Sweller 2005; Sweller 2010), while extraneous cognitive load is generated by dealing with instructional features that do not contribute to learning. Germane cognitive load, on the other hand, is created when learners are engaged in processing essential information that contributes to learning (Sweller et al. 1998; Sweller 2010).”
“Learners have also been shown to be more likely to experience high extraneous cognitive load when they process redundant information that is unnecessary for learning (Kalyuga and Sweller 2005). More specifically, numerous empirical studies on cognitive load effects have found that presenting the same information in different forms (e.g. presenting verbal information in both written and audio forms) would hinder learning and cause the redundancy effect (Mayer et al. 2001; Diao and Sweller 2007). The redundancy effect is very relevant to subtitling in that subtitles transfer auditory information into a written form and thus could produce verbal redundancy, which is likely to induce extraneous cognitive load. However, subtitles in different linguistic formats generate different degrees of redundancy and could exert a differential impact on cognitive load and, as a consequence, on task performance and learning outcomes within educational settings.”
“Based on formal linguistic parameters, subtitles can be categorised into three types, namely intralingual subtitles, interlingual subtitles, and bilingual subtitles (Díaz Cintas and Remael 2007). Intralingual subtitles (or same-language subtitles or bimodal subtitles), which are presented in the same language as the spoken dialogue, are primarily used by deaf and hard-of-hearing viewers, but also in language learning and other educational contexts (Doherty 2016; Kruger and Doherty 2016; Doherty and Kruger 2018). Interlingual subtitles (or standard subtitles or L1 subtitles) refer to subtitles that are displayed in a language different from that of the dialogue, normally in the viewers’ native language (Raine 2012). Different from intralingual and interlingual subtitles, which consist of only one language, bilingual subtitles (also known as dual/double subtitles) present subtitles simultaneously in 2 different languages. This category is mostly used in multilingual countries or regions where two or more languages are spoken, such as Finland, Belgium, Israel, Singapore, Malaysia and Hong Kong (Gottlieb 2004; Kuo 2014; Corrizzato 2015). In Mainland China, for example, bilingual subtitles are gaining currency as China’s dominant TV broadcaster is stepping up its effort to present television programs with subtitles in both English and Chinese in order to attract a wider audience. The increasing usage of bilingual subtitles in online videos is attributed to the efforts of amateur subtitlers who translate foreign language videos online on a voluntary basis (Zhang 2013; Hsiao 2014).”
“Thus, our research question aims to identify the effects, if any, of bilingual subtitles on viewers’ distribution of visual attention, cognitive load, and comprehension of audiovisual stimuli.”
“This also provides support to previous arguments that ‘the number of lines does not play as big a role in the processing of subtitles as previously thought’ (Kruger and Steyn 2014: 105). However, it is worth noting that adding subtitles in a non-native language may cause a different interaction between the language of the subtitles and the language of the soundtrack, which could consequently impact on the attention allocated to subtitles, as the viewer may automatically try to read along with the narration, in what we could call the karaoke effect. This assumption is being investigated in our other studies.”
“It is also possible that in the bilingual condition, with two lines of subtitles presented in different languages, subtitles presented on the first line (i.e. L1 subtitles) can grasp viewers’ attention more easily and viewers may feel less motivated to read L2 subtitles once they have gained sufficient information from L1 subtitles. To sanction this assumption, more empirical research is needed to investigate the impact that subtitle positioning in bilingual subtitles has on the distribution of visual attention. Another possibility is that L2 subtitles render more redundancy than L1 subtitles when L2 audio information is available and therefore are less attended to by participants.”
“Moreover, the fact that viewers spent time reading subtitles in both languages in spite of their redundancy provides evidence for the automatic subtitle reading behaviour hypothesis as originally proposed by d’Ydewalle et al. (1991).”
“we propose that adding bilingual subtitles that contain both L1 and L2 subtitles makes the video easier to understand and allows for more available cognitive resources than not providing viewers with any written text as linguistic support. This finding also supports the growing body of evidence that processing subtitles is cognitively effective and does not cause cognitive overload if optimised spatio-temporally (Kruger, Hefer and Matthew 2013; Lång 2016; Perego et al. 2010).”
Seclusão Anagógica 