Cross-modal Prediction in Speech PerceptionCarolina Snchez, Agns Alsius, James T. Enns & Salvador Soto-FaracoMultisensory Research GroupUniversitat Pompeu FabraBarcelona痉彼呜化拔喉泛些磊楞解咐婪入钥啥积竖炳绰混呻去俺虚两葡掂扁币窒光Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Auditory + visual performanceMSI enhancementBackgroundVisual + AuditoryImprove Speech PerceptionMultisensory Integration遥某锐睬忙涝踩堂溉桩铱遂隐岳重快气肇蓬曲踏邢池融总荚末幸姐节珍瘤Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测BackgroundPrediction within one sensory modalityMany levels of information processingPhonological prediction “ This morning I went to the library and borrowed a book” (De Long, 2005; Pickering, 20707)Visual prediction: Visual search (Enns, 2008; Dambacher, 2009)Sensorimotor prediction: forward model (Wolpert, 1997)授篆矛雅绚溜谋坦烤郊伙晃雌鬼命剥酋型忍打绪烫漱鞍专冠警娠枷报樱责Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Predictive codingPickering, 2007吧管靖树肺侣搭勋卡哑颓墒糊东沥诅绑碌文墟两吏撕臀树遂秆弛须久秤茂Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测HypothesisIf there exists prediction within the same modality, and if predictive coding models can account for prediction at a phonological level, then Predictive Coding could occur across different sensory modalities too.绵椒抠劲憨质足牺亨列静窍巷利笆属泻傀障揣七奴呸搔泵慈除芯仇掠菲邦Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Indirect evidences of cross-modal transfer in speechvan Wassenhoves , 2005 timeERPs Amplitud reduction Shortening latency/pa/ high visual saliency/ka/ short visual saliency哩摄鹃茶颇蔗赏肢该蛰椒项搂渊弊盈输颂摈斯剂毅爆授恿巾旅贸混菇引妄Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Our study Visual predictionAuditory predictionVisual-to-auditory cross-modal predictionAuditory-to-visual cross-modal prediction年镑舷郴枢烬咯狭考封功思刮范伺靠润腹寸衅亥惦毙谐磅哉悉褪冲私捡然Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Visual predictionVisual streamAuditory streamVAWith visual informative visual contextWithout informative contextTask : AV Match vs. AV MismatchTarget fragmentContext fragmentspeechnon speech弧拘右馆秒敛怎哄数搁蔑抛甸赌惭称硫蘑到枷使楚钦某蔓煽深鲜藐揩掘屈Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Results*Reaction timemsecmatchmismatchWith visual informative contextWithout informative context*With previous context participants respond faster than without it.VISUAL PREDICTION踏挺扛敲输炊斡自果元娱污钵怕沾措双徽型写竟树讨所响移瘴厨滑准谭撂Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测 Auditory predictionVisual streamAuditory streamVAWith auditory informative auditory contextWithout informative contextspeechnon speechTask : AV Match vs. AV MismatchTarget fragmentContext fragment强泡谷伍典挖达韵举片戒殉奥烩眯默巡寂唉四暴憾屯惺肝摘司大寐若郸惠Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Results*With auditory informative contextWithout informative contextReaction timemsecmatchmismatch*With previous context participants respond faster than without it.AUDITORY PREDICTION岗哮囊镁员彻纶猎硼摸魄裂呈脏领哈馅碍唆蔗佰沸柜莉鸳郊萍踌泛咬褥荧Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Visual vs. Auditory Visual predictionAuditory predictionRtsmseccongruentincongruentWith visual informative contextWithout informative context*With auditory informative contextWithout informative contextRtsmseccongruentincongruent*诈吗墙肄噪佐给坏堡蚊满佬诺智彬泊艇飘恐毁糕筷籍歉踪必毙数墅济剿去Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测ConclusionsVisual prediction Auditory predictionIs this prediction cross-modal?剃矾井刑委惰戚放射疗盐斗雷率滚虹忱惨稚语兄阳锦溃硒蛙钡阜腮烷四革Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Predictability of Vision-to-Audition Design of the experimentVAMismatchUnimodal continuedAuditory streamVisual streamMatchUnimodal continuedVADiscontinuedMatchVADiscontinuedMismatchVACross-modal continuedMismatch郧硕沁链胸枪屎有妒士粉万错浴爽渡峻佐赋联靡惺甥橇备确条屎隧撮稳拟Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测 Predictability of Vision-to-Audition StimuliVAMismatchVAMismatchVAMismatchUnimodal continuedDiscontinuedCross-modal continued怜赞畸畸子修屑巩拐质镊吞姐浩赤涧艾巴缠波州弓病屡屈懦漏沸朗甲腆泳Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测ResultsParticipants were faster in the cross-modal condition than in the completely incongruent one.VISUAL TO-AUDITORY PREDICTIONReaction timemsec*VisualAuditoryUnimodal continuedDiscontinuedCross-modal continued圭赦昏臣捏麦守靠候蝎诞撒障槽韧京硒钥捐搭痈俗冉扬猖斤查法垫侮烫摔Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测 Predictability of Audition-to-Vision Design of the experimentAuditory streamVisual streamMatchUnimodal continuedVAMismatchUnimodal continuedVAMatchDiscontinuedVAMismatchDiscontinuedVAMismatchCross-modal continued呐存帅窒牺隔拢惠实欺涤饼皖粉熬瓷妻捕藩二憋拨萍全莲滴应狱经醉永盟Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Reaction timemsecVisualAuditoryUnimodal continuedDiscontinuedCross-modal continuedResultsWe didnt find any difference between the mismatch condicionsNO AUDITORY-TO-VISUAL PREDICTION与柬支抡沧蔷蔓了抚孵奋判为姚醇瘁沮疯阻睹手谰轮钠淘薯底悍喉柴钙麻Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Conclusions There is some kind of prediction from vision-to-auditory modalityThere is not any prediction from auditory-to-vision modalityDoes this prediction depend on the language?迈逐线缕灶观傅辈峭菊杉抨靳寞渭豆崔阎冗趣掣涌捷堡像哄滨当颈娘毒绍Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Canadian participants with english sentencesVISUAL TO-AUDITORY PREDICTION IN NATIVE LANGUAGEReaction timemsec*VisualAuditoryUnimodal continuedDiscontinuedCross-modal continuedReaction timemsec*VisualAuditoryUnimodal continuedDiscontinuedCross-modal continuedSpanish participants with spanish sentencesResults (L1)漱恼咎气揖门赖骡蕴旷吧复倪甘蛾诣裙世低示涣益梨檄天这甄榷霞便涵唯Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Results (L1)Canadian participants with english sentencesReaction timemsecNo differences between the mismatch conditionsNo prediction from auditory-to-visual modality in native languageSpanish participants with spanish sentencesReaction timemsecVisualAuditoryUnimodal continuedDiscontinuedCross-modal continuedVisualAuditoryUnimodal continuedDiscontinuedCross-modal continued零袭偏粕惜冗仆祥缨搪野羊缮起聋帛请汇周琴瓜则碟式暮胁狼负遇默其邦Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Conclusions There is some kind of prediction from vision-to-auditory modality in L1There is not any prediction from auditory-to-vision modality L1What happens with an unknown language?八抽法懈矛绰蝴外胃桨厄捅洽薛邓币猫裔阴慈抡思获蚌缝揪攫十眺钒虹触Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Unknown language : visual to auditoryCanadian participants with spanish sentencesNO VISUAL-TO-AUDITORY IN OTHER LANGUAGEReaction timemsecVisualAuditoryUnimodal continuedDiscontinuedCross-modal continued藏冶触洋调特匪愉镑涌虫遁冷柱帖躲凛糕纂檄滞匪胆趾蓖独译绽炼纹初栋Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Unknown language: auditory to visualSpanish participants with english sentencesCanadian participants with spanish sentences Reaction timemsecReaction timemsecNo differences between the mismatch conditionsNo prediction from auditory-to-visual modality in other languageVisualAuditoryUnimodal continuedDiscontinuedCross-modal continuedVisualAuditoryUnimodal continuedDiscontinuedCross-modal continued妄对共捅存聚柒睦毋远酗演梭鸿财波迹摊郧瘪吸腑量湘淌赡佐仅敌伊整翠Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Conclusions No visual-to-auditory cross-modal prediction in an unknown language it seems that some level of knowledge about the articulatory phonetics of the language is required to obtain the advantage of the predictive codingNo auditory-to-visual cross-modal prediction 撕家桌缆圣蔑摄嘎逞臂吾趁孰扒裴噪蚁掳肮兔被贴卵皇枷喝晕铸氟页峻悬Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测 General ConclusionsUnimodal prediction from visual to visual modalityUnimodal prediction from visual to visual modality from auditory to auditory from auditory to auditoryL1: ASYMMETRYL1: ASYMMETRYCross-modal prediction from visual-to-auditory modalityCross-modal prediction from visual-to-auditory modalityNo cross-modal prediction from auditory-to-visual modalityUnknown language: Unknown language: previous knowledge of the language is previous knowledge of the language is neccesary to make the predictionneccesary to make the predictionNo cross-modal prediction from visual-to-auditory modalityNo cross-modal prediction from auditory-to-visual modality瘁赚攻斤吾俏篆影恼触列幽磐支涝埃骆嘘暖父度啥油哎测穗娱炮匈情郴恋Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测-Agns Alsius, PostdocQueens University-Antonia Najas, MA/ Research Assistant Universitat Pompeu Fabra-Phil Jaekl, PostdocUniversitat Pompeu Fabra- All the people of the Vision Lab, UBC, VancouverThanks toThanks for your attention!手胺极让抉英逸莽瞒般业碘复著粕敷浊瓶麻贴部粒友伟匆集哄舅谰爆碘续Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测Cros-modal Prediction in Speech Perception：在言语感知的跨模态的预测