Illuminating neglected uncertainties

How metascience can help rather than hinder democracy

By Andy Stirling (RoRI)

"A murmuration of starlings" by Adam (Flickr)

At a moment when science has never been more politically important, democracy has rarely seemed more fragile. Around the world, governments are investing heavily in research to address challenges ranging from climate change and public health to artificial intelligence and public security. At the same time, public debate has become increasingly polarised. Understandings are cynically rejected and fabricated, institutions are challenged and scientific advice is seemingly drawn ever more deeply into political conflict.

The instinctive response in prevailing cultures of Modernity is to ask for better evidence. Better data. Better models. Better indicators. Better ways of knowing what works. Few fields embody this ambition more than metascience. By studying science itself, metascience promises to improve how research is organised, funded, evaluated and used. The recent RoRI report for the UK Metascience Unit is an excellent illustration of the field’s growing sophistication and importance.

Yet after more than three decades researching the relationships between science, technology and public policy, I have come to think that the greatest challenges rarely arise because we have too little evidence. More often, they emerge because we ask too little about what evidence cannot tell us – and about ever-present openness of data to divergent legitimate views.

Much though I appreciate the vital – often inspiring – research and policy work of this community, too many of the discussions at last year’s Metascience conference reinforced my concerns. These discussions suggested three issues that deserve to sit much closer to the centre of the field: uncertainty, directionality and democracy.

The first issue is uncertainty. Much of the public language surrounding science assumes that evidence points towards a single correct course of action. We hear calls for “science-based policy”, “evidence-based decisions” or simply to “follow the data”. Such phrases have understandable rhetorical appeal, especially in an era of misinformation. Albeit often denied when challenged in detail, the general body language implies that science on its own can be a sufficient basis for policy prescriptions. This implication is false.

The truth is that science has never worked in a purely analytic way. Uncertainties, ambiguities, variabilities and sensitivities all afford as much of a role for interpretations, values and interests, as they do for “hard data”. Resulting arrays of equally-valid judgements are always plural. Evidence and analysis are necessary for good policy making for and with science, but they are not sufficient.

In my own experience of studying and contributing to policy appraisal – from research evaluation and technology assessment to risk analysis, cost-benefit analysis and ecosystem services – I’ve never encountered a model whose conclusions did not depend upon these factors. Change the framing, the boundaries, the treatment of uncertainty, or the questions being asked, and different – frequently radically so – policy options emerge as preferable.

This should not surprise us. In fact – as pub conversations typically testify better than many formal committee discussions – none know this better than experts themselves. Complex societies – and disciplines – inevitably involve multiple legitimate perspectives. The real danger lies not in uncertainty itself, but in hiding uncertainty behind apparently objective numbers.

For metascience, this carries important repercussions. Sensitivity analysis should not be buried in technical appendices. It should become one of the principal ways findings are communicated. Alternative valid interpretations and their implications for action, should be openly highlighted in a balanced way, not hidden away. Just like objects viewed from different angles in everyday life, ‘truth’ is not singular, but ‘plural and conditional’. Being open and accurate about the conditions shaping the viewing, are as crucial as the pictures themselves.

This is not about post-modern ‘anything goes’ abstraction, but grounded pragmatism and rigour. Showing how conclusions vary under different assumptions does not weaken evidence. It demonstrates intellectual honesty – as well as democratic accountability and analytic rigour – about the conditions under which particular conclusions do (and don’t) hold.

This leads to the second issue: directionality. This is not just jargon but a technical term with crucial meaning for metascience. It refers to the multiplicity of particular directions for understanding and action in which funding bodies, scientific interpretations, research orientations or technological pathways can become locked-in. Albeit inconvenient for interests wishing to justify misleadingly singular, seemingly definitive interpretations, the ever-present truth is that any body of evidence and analysis can always support a diversity of equally justified but contrasting possible actions. Without recognition for this meaning of directionality, simply steering a particular direction fails to attend in an open symmetrical way to the fact of there always existing a plurality of reasonable alternatives.

Research policy often asks how science in a given setting can become faster, more productive or more internationally competitive. These are significant queries. But they are not necessarily the most important ones. Whether the issue is artificial intelligence, energy, agriculture, public health or security, the defining questions are often remarkably simple: Which ways should we go?

Across different areas of research and innovation policy, the fact that much-vaunted ‘road maps’ typically only have one road is a transparent illustration of the problem. For anyone other than those wishing to force their own interests, what use in the real world is a map with only one road? Around the world, too much science and technology governance is blinkered by the constraints imposed by this kind of powerfully-imposed expedient policy tool.

Appreciation of the importance of all this is not just a matter for ‘critics’. It is on these crucial factors that concrete outcomes of all kinds depend across many key areas of metascience for policy. The implications are highly practical and relevant under any view. For instance: what balance to strike in allocating scarce public resources across renewable energy or nuclear power? Agroecology or gene editing? Preventive public health or pharmaceutical innovation? Highly centralised AI or distributed digital infrastructures? Peaceful human security or mass projection of military violence? These are not questions that evidence alone can settle. They concern competing visions of desirable futures.

We often treat key foci of metascience as if they are simply about magnitudes, whose main relevance is whether they grow or shrink in scale – more or less. This is true, for instance, of ‘excellence’, ‘novelty’, ‘impact’, ‘productivity’, ‘pace’ – or ‘winning’ or ‘losing’ in some particular research or innovation ‘race’. All are treated as if they are simple self-evident magnitudes – independent of any judgement about how to measure, by what scales, aggregated how and with what relative weightings?

In other words, these major subjects of metascience are treated as ‘scalars’ (simple single numbers) rather than as what they really are: vectors (with multiple constituting dimensions that give them directions as well as scale). All the above values, for example, rest not just on ‘how much?’, ‘how fast?’ or ‘who wins?’ but on fundamentally qualitative alternative directions for possible change. Inconvenient to interests that wish to conceal awkward resulting questions, a crucial truth in metascience is that the most defining properties of any given field of research or innovation are given not simply by associated magnitudes, but by their directions.

This suggests a subtle but profound shift for metascience. Evaluation should become less concerned with demonstrating what one particular pathway can achieve, and more concerned with comparing alternative pathways. Objectivity lies not in presenting a single notionally authoritative picture, but in exploring systematically how different assumptions, values and priorities generate different pictures.

And so this leads to the third issue: democracy. Much discussion today rightly focuses on authoritarian political movements as threats to science. Those threats are real. But they are often discussed as though they originate entirely outside science itself. Here, there is another possibility that deserves reflection.

Some of the habits that characterise authoritarian politics – confidence in singular answers, denial of uncertainty, impatience with ambiguity, appeals to unquestionable authority – can also appear (albeit unintentionally) within scientific institutions … and even within metascience. This happens whenever we suggest (or imply, or let it be understood) that policy should simply “do what the data say”. Whenever this occurs, a misleading claim is made (if only tacitly) that data offer an authority that they cannot possess on their own. Robust evidence is a crucial, but never enough.

Asking “what do the data say?”, a question sadly all too familiar in science policy, makes no more sense than arguing to “do what the words say”. With language, we immediately recognise the need to ask: Which words? In which language? Read in what way? Telling whose story? Equally conditioned by context, data deserve to be interrogated with exactly the same questions. Yet – when dragooned into asserting false singularities – the authority of mathematics, algorithms and elegant graphics can make these interpretive choices disappear. This is what happens, for instance, when vectors are turned into scalars according to some invisible pre-set frame.

Perhaps this helps explain what is otherwise a paradox: that emergence of nationalist authoritarianism should be so internationally concerted? Why are such parochial and prejudiced perspectives, so adversarial to difference, so apparently coordinated across precisely the borders they seek to entrench? The picture is complex. Many factors are in play. But it is difficult to dismiss an interpretation that rising populism is to a large extent itself a reaction. Has not the globalising institutionalisation of progressive causes around the world – on climate, biodiversity, trade, regulation, social justice – itself taken forms that are technocratic and authoritarian in precisely the manner criticised above in too much metascience?

Is it not also ‘post truth’ to assert expert prescriptions for policy as if these are definitive and non-negotiable ‘sound science’, without mentioning the significant extent to which these prescriptions are also shaped by assumptions, values and interests? Yet with this ‘scientism’ becoming a growing new public management style in social and environmental fields, regressive forms of authoritarianism can be understood (at least in part) as a backlash. When progressives lament the false certainties of their opponents, do they unwittingly see themselves in the mirror? Is doubling down on assertive scientism arguably more a catalyst than a remedy?

A similar uncomfortable flip can appear in discussions of trust. We devote considerable attention to the strengthening of ‘public trust in science’. Far less attention is given to the reciprocal question: how can scientific institutions and communities themselves become more trusting of citizens – and society at large? Messy and unruly though they are – and with all their own predispositions to eccentricity and error – it is by social movements, after all, that many of the most cherished progressive technological transformations of the past century have been shaped. Social equality, women’s emancipation, anti-racism, gay liberation, the welfare state – and many once-denied but now burgeoning possibilities for renewable energy and green technology, have been driven more by these kinds of murmuration than by evidence-based policy or science-based decision-making.

This social reciprocity of trust matters because democracy is not simply about communicating expertise more effectively. It is about creating institutions capable of engaging constructively with legitimate disagreement. Public engagement is therefore not most valuable when it manufactures consensus, as is typically assumed across too many instruments of ‘public engagement’. It is instead when ‘uninvited participation‘ illuminates competing perspectives and divergent options – and clarifies why disagreements exist – that alternative futures can be properly explored. Productive dissensus is often a healthier democratic outcome than premature agreement.

Nor is this somehow in tension with science. The Royal Society’s motto, Nullius in verba – “take nobody’s word for it” – is often celebrated as a key aspiration in science. And this can also equally be a principle of democracy. Idealised norms in democracy and science both depend on the creation of cultural and institutional space to question authority, expose assumptions, explore diversity, remain open to alternatives and change interpretations if and when necessary. Democracy and science need not be in tension, but mutually reinforcing against authoritarianism.

Perhaps this is where metascience’s greatest opportunities now lie? Certainly, it can continue to improve the efficiency, integrity and accountability of research systems. But it can also help cultivate something broader, more qualitative and more valuable: a science that is more explicit about uncertainty, more reflective about direction and more open and confident that democratic disagreement is not a problem to be overcome, but a resource through which better knowledge – and better futures – can emerge.