In a recent pair of essays, Jonathan Fuller and Marc Lipsitch discuss how we should understand the disagreements between epidemiologists over how to respond to COVID-19. Epidemiologists are divided into two groups: “clinical” epidemiologists and “infectious disease” epidemiologists. Broadly, clinical epidemiologists favour the results of large-scale, randomised controlled trials that aim to estimate the direct effects of interventions on disease transmission by accounting for other factors that could affect the outcome. Such trials are expensive, time-consuming, and are ideally repeated several times to try and tease out any unknown confounders that might have had an impact on the outcome. On the other hand, infectious disease epidemiologists use theory-based models, ideally built and tested on previous outbreak scenarios, to extrapolate trends from noisier, lower quality, and often smaller amounts of data. Such models necessarily involve assumptions and simplifications about the biological properties of the disease and the human population through which it spreads. The difference between these groups has been characterised as driven by different philosophies of evidence (Fuller) and different philosophies of action (Lipsitch). I propose that it might be better to explore the difference between the two mindsets in terms of their use of causal reasoning. Furthermore, the debate that has spawned the examination of these differences in philosophy has arisen because evidence taken from epidemiological models is being held up against other types of evidence, primarily economic, when governments are choosing their COVID-19 response. I argue that, in trying to understand the differences in approaches to evidence between these two types of epidemiologist, we can also understand how evidence from infectious disease models compares to evidence from economic models. Infectious disease models are criticised for making assumptions during the process of causal reasoning, yet any model (including economic) attempting to answer counterfactual questions must make assumptions.
Fuller characterises the difference in attitude between the two groups as one of philosophies of evidence: clinical epidemiologists are reluctant to act on the quality of evidence that infectious disease epidemiologists use to parameterise their models. The quality of evidence that goes into models might be considered weak by clinical epidemiologists, making strong actions in response to their predictions unwarranted. Indeed, we have implemented costly and intrusive interventions in order to avoid the predictions of infectious disease models. However, as Lipsitch points out, in situations such as COVID-19 it is highly unlikely that the sort of evidence demanded by clinical epidemiologists will ever arrive. There has never been – and now never will be – a controlled trial undertaken on whether lockdown measures are a reasonable and proportionate response to initial COVID-19 outbreaks. Each country has embarked on its own particular combination of control measures. It will be very hard to disentangle the effectiveness of their policies from other factors which may have influenced the extent of their outbreak. To avert potential disaster, we must act on available evidence that may not be of the quality we desire, since the outcome of no action is potentially unbearable. An alternative way to characterise the difference between the two types of epidemiologist (which, as Lipsitch points out, really lay on a spectrum, since there is considerable methodological and departmental overlap) is through their use of causal reasoning, and how mathematical modelling is used as part of this reasoning.
Judea Pearl, in his quest to program a machine that thinks causally like humans do, defines three levels of causal inference on what he refers to as the “ladder of causation”. The ladder has three levels of cognitive ability regarding causal reasoning: association, intervention, and counterfactuals. We can understand these three levels through the lens of the debate about widespread mask-wearing during the COVID-19 pandemic:
| Rung on ladder of causation | Description | Example |
|---|---|---|
| Assocation | We think of events as being associated with each other when “observing one changes the likelihood of observing the other”. There might be an obvious causal interpretation or there might not. | Many people have produced graphs showing how there have been smaller COVID-19 outbreaks in Asian countries, where there is widespread wearing of face masks, compared to European and North American countries, where there is not. From observing this association they have interpreted a causal relationship between mask wearing and outbreak size. |
| Intervention | Reasoning about what the results of performing an action will be. We now want to know how us doing one event changes the likelihood of the other event. | Many people have called for mandatory mask use in public in Europe and North America. This involves a belief that intervening (wearing masks) reduces the likelihood of transmission. However, it is possible that Asian countries had smaller outbreaks due to other reasons, such as fast, effective contact tracing systems. |
| Counterfactuals | Reasoning about what the world would be like if something had not happened. To quote Pearl: “No experiment in the world can deny treatment to an already treated person and compare the two outcomes, so we must import a whole new kind of knowledge.” | Researchers are using causal reasoning when they develop mathematical models to answer the question such as “what would have happened if we had implemented mandatory mask use in March?”. This involves assumptions because we can’t collect data on something that didn’t happen. |
In the distinction I am making, clinical epidemiologists primarily answer questions from the association and intervention rungs of the ladder. They spot associations, develop hypotheses about causal relationships, and then perform trials to measure the effects of interventions. Randomised controlled trials (RCTs) use randomness to try and ensure that the only meaningful difference between two populations is that one has an intervention and the other does not. That way we can directly measure how performing one event (the intervention) changes the likelihood of another event (the outcome). Doing RCTs tries to rule out confounders, these are other factors, often unseen, that might produce the associations that we are observing and lead us to wrongly assume that A causes B whereas in fact C causes both A and B. Infectious disease epidemiologists, on the other hand, answer questions from the intervention and counterfactual rungs of the ladder. They ask questions like “What will the effect of mandating mask use in public be on the epidemic?” as well as “What would have happened if the UK had gone into lockdown one week earlier?”. Infectious disease epidemiologists tend to be less involved in running trials and spend more time developing mathematical models that aim to approximate the physical process of disease transmission. To me, the clinical epidemiology to infectious disease epidemiology spectrum seems to run upwards along the ladder of causation. It’s important to point out that rungs nearer the bottom are not necessarily “easier”, it is often exceedingly difficult to narrow down the causal effect of an intervention within a complex web of causal interactions.
Clinical epidemiologists see infectious disease epidemiologists answering intervention-level questions that are familiar to them, such as “what is the effect of wearing a mask on transmission?”, for which there is a gold-standard of evidence, RCTs, yet theory-based models have been used. This leads clinical epidemiologists, such as John Ioannidis, to warn that, despite a deluge of modelling papers, there is not adequate evidence that lockdowns should be implemented. This warning is interpreted by infectious disease epidemiologists as advocating a potentially dangerous, strict non-interventionist approach, yet in subsequent writings Ioannidis clarifies that he doesn’t think that no action should be taken and goes on to argue for renewed attention on the potential negative side effects of lockdown. These arguments are on the counterfactual rung, since the implication is that these negative side-effects, such as mass unemployment, would not have happened if lockdown had not happened. The split between theory-based models and empirical trials is muddled because models seem like they are asking questions on the intervention rung of the ladder, when they are in fact implicitly asking questions on the counterfactual rung of the ladder.
Developing a mathematical model to answer a prima facie intervention-level question usually involves the development of a counterfactual causal model at the same time. You would begin by making sure that your model accurately captures the observable disease dynamics (confirmed cases and deaths) up until the present. Then you can introduce into the model that 60% of the population begins to wear a facemask and project the course of the outbreak forward into the future, assuming that people who wear masks have some reduced probability of infecting others if they are themselves infectious. At the same time, to derive any meaningful value for the impact of masks you also need to project the course of the outbreak forward in a possible world where no-one adopts mask-wearing to compare your intervention world to. To model the impact of a possible world with an intervention, you always implicitly imagine a counterfactual possible world where the intervention did not occur. Since modelling involves answering counterfactual questions, at some point the model has inevitably involved making assumptions rather than utilising data (since, as Pearl points out, such data can often never be collected). The point of infectious disease epidemiology as a discipline is to make these assumptions as reasonable as possible through the study of past outbreaks, the interventions that have previously worked, and other relevant demographic or biological factors that can change how diseases move through a population.
To give an example, to help guide your assumptions on the effect that country-wide lockdowns will have on COVID-19 transmission you could turn to existing data on how infectious disease outbreaks change the amount of social contact we have. There might be data available from Sierra Leone on how people’s mobility changed during the Ebola outbreak measured by mobile phone activity. Let’s say it suggests that people’s movement decreased by 45% during the outbreak without a lockdown occurring. How relevant is this data to our model? What difference does it make that the data is for Ebola instead of COVID-19? Is data from Sierra Leone applicable to the UK? Is using mobile phone data a good way of capturing how population movement changed? An infectious disease epidemiologist should consider all of these questions, drawing on knowledge and experience from the study of past outbreaks. The conclusions that they draw will hopefully be scrutinised, both by colleagues and during the formal peer-review process. To make this scrutiny possible the assumptions need to be laid out and often precisely described using mathematical formulae. The merit, as far as I can see, of infectious disease epidemiologists employing theory-based mathematical modelling is that it makes explicit all of the causal reasoning involved (as long as it is made public, as it should always be). Answering a counterfactual question involves employing a causal model based upon your prior knowledge and experience. This is still true even if you did not, or are not able to, commit that causal model to paper using mathematical symbols.
Mathematical models of infectious disease that answer counterfactual questions are often criticised using the phrase “it’s just a model”, but answering counterfactual questions always employs the use of “a model” even if you don’t express it using mathematics. The act of committing a causal model to paper using maths, as infectious disease epidemiologists do, does not make it more reliant on assumptions about the causal model structure than the answer given by anyone else, it just makes those assumptions more visible and therefore more open to criticism. If we are interested in the transparency and scrutiny of decisions made during this outbreak, then we should want the assumptions made to be stated explicitly. What is happening during this crisis is that some people want to have their cake and eat it; they decry mathematical models as “just models” while employing a causal model with its own assumptions and causal reasoning that remains inside their head. I don’t think it’s possible to escape doing what could reasonably be thought of as “modelling” when answering questions relating to how we should respond to an emerging pandemic, no matter what flavour of epidemiologist you are.
But why go to such pains to elaborate how we are thinking when we are “just modelling”? The answer lies in the fact that, in this particular pandemic, policies suggested by mathematical models of disease transmission are competing directly with the interests of global financial capital, which has a much longer tradition of determining policy. At the turn of the millennium, the inclusion of disease burden reduction in the World Health Organisation Millennium Development Goals and an increased interest in biosecurity in the wake of the September 11 attacks precipitated a boom in funding for infectious disease research. The mid-to-late 2000s saw the emergence of large and well-funded research groups at a broader range of universities, such as the MRC Centre for Outbreak Analysis and Modelling at Imperial College London in 2008.Since the year 2000, the University of Oxford has established new research units in Indonesia, Vietnam, Cambodia, Laos, Myanmar, the Democratic Republic of Congo, and Uganda. At the same time, the fiscal budget of the United States Centre for Disease control has increased from $2.9 billion in 1999 to over $7 billion in 2019. Private philanthropy projects such as The Bill and Melinda Gates Foundation have also steadily increased their monetary contributions to disease eradication efforts; often their grants made to African nations require infectious disease modelling to estimate the expected impact of the grant as a pre-condition.
As the research output of infectious disease epidemiology has grown, so has the role of modelling and modellers in guiding government outbreak response policy. There have been several infectious disease outbreaks of major concern since the turn of the millennium: Ebola, SARS, H1N1 influenza, foot-and-mouth disease. However, the COVID-19 pandemic is the first infectious disease outbreak since then to cause significant morbidity and mortality in the global north. The outbreak of SARS-CoV-1 back in 2003 resulted in 774 deaths across the entire world, a number that is smaller than the daily death tolls for SARS-CoV-2 in several countries. The 2014 West African Ebola outbreak resulted in tens of thousands of deaths in Liberia, Sierra Leone and Guinea, but the subsequent scrutiny of infectious disease epidemiologists and the public health response to Ebola from the British press was non-existent compared to the current pandemic. While the British press will of course focus more heavily on outbreaks in the UK, the truly unique aspect of the COVID-19 response that is responsible for bringing modelling under such intense inquiry is that the stringent interventions that it recommends have interfered with capital circulation in some of the world’s biggest financial centres.
In the UK we have incurred great economic and social cost to avoid the alternative scenario predicted by mathematical models. The model with the first bout of publicity, produced by Imperial College London and released as a report on 16th March, predicted that without significant intervention measures a large epidemic would completely overwhelm the NHS’s critical care capacity, and cause hundreds of thousands of deaths. Seven days later, on 23rd March, the UK went into lockdown, the aim of which was to drastically reduce the number of social contacts that people made – if there is no contact, there can be no transmission. At the time of writing, as the lockdown is beginning to be lifted piece by piece, an ONS survey estimates that 6.2% of the country has been infected, and there have been 41,369 recorded deaths. From the data available so far, it seems the thrust of the ‘Imperial model’ was correct in that, had a significant proportion of the country become infected, the death toll would have been in the hundreds of thousands. Simultaneously, the lockdown caused mass disruption to workers and businesses, and the economy has now entered into recession. In light of this, one line of criticism of the lockdown strategy that emerged over the course of the outbreak is that the economic damage suffered as a result of the lockdown is more damaging than letting the outbreak continue unimpeded. This view, however lamentable many epidemiologists may find it to be, is merely at the extreme end of how you might weigh up economic costs relative to public health costs. Such a consideration always occurs in the mind of the policymaker in some form, the only difference is how much importance they place on each consideration.
On the 13th May, roughly two months after the lockdown began, the UK government urged people that ‘cannot work from home’ to return to work, avoiding public transport if possible, in order to re-open the economy. Given that around 0.25% of people in the UK (2.5 people in every 1000) were estimated to be infected around this time, and that infections seem more likely to occur due to prolonged, indoor contact, it is certain that new infections occurred as a result of people returning to work and having social contact with their colleagues. When the government made this decision they either implicitly or explicitly weighed up two counterfactual scenarios – one was the output of a causal model that estimated the human cost in terms of new infections (some of which will be fatal), the other a causal model that estimated the economic cost of businesses remaining closed. This is where the claim of being entirely “led by science” breaks down: choosing between these two scenarios is an ethical rather than a scientific decision, as Fuller points out. I would argue that, seeing as these opposing scenarios have been posed in the formal language of mathematics, the brutal calculations that occurred behind the policy decision should be laid out, just as they are for the models created by infectious disease epidemiologists. The people working in jobs that cannot be performed from home deserve to know the monetary value they are modelled to produce – very little of which they will receive directly as salary – for the sake of which they are being exposed to a potentially deadly virus. Perhaps the UK government feels the need to obscure the reasons why these decisions are made because people would be insulted by the relatively small amount of money the government is happy to expose people to the virus for compared to money spent bailing out large companies. For reference the National Institute of Health and Care Excellence (NICE) is happy to pay about 20-30 thousand pounds per quality adjusted life year. If so, then what does this say about the transparency, fairness and accountability of our society?
Such frank choices over the amount of money available to try and stop someone from dying from a communicable disease do not only occur in desperate scenarios such as the COVID-19 pandemic. It is routine in the field of infectious disease epidemiology to use complex optimisation algorithms to best allocate, in a way that will save the most lives, the finite resources available to tackle diseases like HIV and malaria across the world. The upper bound on lives saved is set by the amount of funding available, which is made up of contributions from the governments of endemic countries, the Global Fund, the World Bank, contributions from private philanthropists, and varying contributions from governments in the global north. The output of a disease model with a funding constraint is something akin to lives saved per dollar spent. However, if you invert this value to fit a scenario where you reason that economic incentives are more important than the human cost, then it is equivalent to lives lost per dollar saved. There is no reason why a pounds-saved-per-life-lost estimate is not available in the UK right now, other than perhaps that people in the UK are not used to thinking about their safety only being prioritised up to a certain cost threshold.
The benefit of being “led by science” in this pandemic is that policymakers will hopefully have the best possible prediction of the consequences of their disease control policies. As I have argued, that these epidemiological predictions contain assumptions does not make them less valuable as evidence, since anyone looking to answer questions about interventions and counterfactual scenarios will be employing a causal model containing assumptions. What we should champion from these models is transparency with regards to their assumptions. While epidemiological models try to predict what will happen if you act in a certain way, they do not tell you how you should act. Policymaking during a pandemic still requires normative judgements, whereby other considerations are weighed against the answers to epidemiological questions. In capitalist nations, these other considerations have largely been economic. Since interventions to stop COVID-19 transmission will also interrupt production, the economic models, with their own assumptions, should be laid out as clearly as epidemiological models are.