WIREDGOV

Fed up of people not listening to you or taking your opinions on board? Maybe it's your hand gestures that are the problem. A new study funded by the Economic and Social Research Council (ESRC) shows that it's not just what you say but how you say it. What you do with your hands when you talk can make the difference between being understood, and leaving others baffled and scratching their head.

"When we talk face to face there's a lot of 'extra' information that comes along with language," says Dr David Vinson, a cognitive psychology lecturer at UCL (University College London) who led the research in collaboration with Professor Gabriella Vigliocco (UCL) and Dr Pamela Perniss (University of Brighton). "This includes hand gestures as well as other visual information like facial expressions, posture, and other sounds like the tone of your voice, even vocal sound effects. The same is true in signed languages.

"What we wanted to know was - does this 'extra' information actually contribute to communication?"

Dr Vinson and his team recruited over 100 volunteers and played them video clips of actors saying phrases whilst making hand gestures.

He instructed them to pay attention to the speech, ignore the gestures and decide whether the phrase matched an image on a computer screen.

Unbeknownst to the volunteers, half the videos had been edited so the hand gestures didn't correspond to the words spoken; for example saying "open the door" while gesturing "turning a dial". Even though the volunteers had been told to ignore the gestures, they still made more than twice as many errors when the spoken words and hand gestures didn’t match.

This phenomenon wasn't limited to spoken languages. Vinson and colleagues also tested users of British Sign Language. When people communicate using sign language their mouth movements often look like words from the spoken language they are most exposed to. So what happens when the word they're mouthing isn't the same as the one they're signing? The results were the same - volunteers made twice as many more errors when the signer's mouth didn't match their hands, even when told not to pay attention to this information.

So why is this? When you're chatting to someone your brain is combining what you are seeing with what you are hearing. If the two inputs match up then it takes less time to understand the meaning of the words. However if the inputs don't correspond then it takes the brain longer to process the information, delaying understanding. So using hand gestures can help you get your message across, but only if you use them properly. Confusing hand signals can actually prevent others from understanding you.

To understand what goes on in the brain during this process, Vinson hooked up volunteers to a functional MRI machine which measures blood flow in the brain. This allows the researchers to see what areas of the brain are more active at any moment in time. He played them clips of speech-only videos, gesture-only videos, and videos where the two were combined. Like before some of the videos were edited so the gestures didn't correspond to the words being spoken.

He found that a widespread network on the left side of the brain, containing regions often thought to be the 'language network', were more active when speech and gestures mismatched.

"These results suggest that physical gestures affect language processing and that this area of the brain has a pivotal role in understanding meaning," says Vinson

"It also shows the so-called 'language network' doesn't just process linguistic information, but can also combine it with different sources of meaningful information, such as hand gestures.

"When those sources are in conflict these brain networks are more active because they have to work harder to extract the meaning."

Further information

• Harry Dayantis, UCL Media Relations Manager, Medical Sciences & Brain Sciences
  Email: h.dayantis@ucl.ac.uk
  Telephone: +44(0)20 3108 3844 (Int: 53844)
  Mobile: +44(0)7747 565056

Notes for editors

• The Economic and Social Research Council (ESRC) is the UK’s largest funder of research on the social and economic questions facing us today. It supports the development and training of the UK’s future social scientists and also funds major studies that provide the infrastructure for research. ESRC-funded research informs policymakers and practitioners and helps make businesses, voluntary bodies and other organisations more effective. The ESRC also works collaboratively with six other UK research councils and Innovate UK to fund cross-disciplinary research and innovation addressing major societal challenges. The ESRC is an independent organisation, established by Royal Charter in 1965, and funded mainly by the Government.