Demos - The Representivity Puzzle: Transforming Big Data Analysis

10 Sep 2015 10:13 AM

A new paper from the Centre for the Analysis of Social Media at Demos, and Ipsos MORI, tackles the biggest problem in social media research: representivity.

The Road to Representivity: Sociological Research Using Twitter argues that whilst social media platforms offer powerful new ways of understanding society, they are often researched poorly, with little examination of the extent to which insights are representative of social media users or wider society. The future of harnessing these new bodies of data is to wrap new technology around the principles of social science. Only then can it influence important decisions across Government and commerce.  

The first in a landmark series seeking to set gold-mark standards for social media research, the paper looks at how to ensure that the wealth of new big data available can best be accurately measured and honestly interpreted.

The paper illustrates the challenges of representivity by contrasting online and offline data-sets on three key themes:

• Issues: Which social and political issues are most prominent on Twitter, set against the issues people mentioned as important when asked face-to-face. 
• Brands: The brands most spoken about on Twitter, set against the brands most mentioned by people when asked face-to-face.
• Politics: How Ed Miliband and David Cameron were spoken about on Twitter, set against their offline favourability scores.

The research found a number of hard questions that social media research has to confront and overcome; from demographic skews to the inordinate influence of power-users and the presence of bots.

Based on these studies, the paper presents a new research method, ‘the road to representivity’ – to try to confront these problems. It involves:  

• New methods to collect possibly relevant data and filter out data that is not
• Balancing the under- and over-representation of certain socio-economic and demographic groups, and ensuring regional representivity
• Correcting for prolific social media users distorting data-sets through over-representation of their viewpoints
• Differentiating between posts and tweets made by bots and humans
• Countering against the strong influence of institutions, versus individuals, when analysing viewpoints

The paper also considers whether social media research can ultimately be compared against offline research, given social media is a different kind of window into social life. It concludes that both types of research serve distinct, but often aligned, purposes and that they must now be used together to understand a society that lives in both worlds.

Commenting on the report, Carl Miller, report author and the Research Director of the Centre for the Analysis of Social Media at Demos, said:

“This paper is part of a broader project that has brought together leading technologists and social scientists to work out how to make the most of the enormous potential that social media research could hold for governments, businesses and society more broadly. The insights that new digital data-sets can provide could transform the way that companies and public sector bodies operate and deliver services, but this can only be realised if we can research social media in ways we can trust. Like other parts of big data, social media research has to move beyond an obsession with bigger and bigger numbers. We have to remember what is behind all this data: human beings. And as the data available to us continues to grow, we need to remain sensitive to our basic task - the messy difficult task of understanding human beings through their everyday, lived experiences.” 

Steven Ginnis, report author and Head of Digital Research at the Social Research Institute, Ipsos MORI, said:

“The pace of advances in technology has resulted in a body of social media metrics that do not truly represent the views of those we claim to listen to, or appreciate the fluctuations seen in online conversations.  We need a new approach, putting more emphasis on the human analyst before we can generate the quality of insight required by Government, academics and the private sector.”

Notes to Editors

The Road to Representivity: Sociological Research Using Twitter is available at: http://www.demos.co.uk/files/Road_to_representivity_final.pdf?1441811336

It is the first in a series of papers, which will culminate in a major new report, to be published in late-2015 and launched at a formal event. This project is part of a wider effort between social researchers at Demos and Ipsos MORI, and technologists at the University of Sussex and CASM Consulting LLP, to build new and better ways of conducting social media research.

For more information regarding the event, please contact Megan Poole at Demos (megan.poole@demos.co.uk), or Aalia Khan at Ipsos MORI (aalia.khan@ipsos.com).