St Anne’s Academic Review 8 – 2018
All Models are Wrong. Some Models are Useful
Shuchi Vora, School of Geography and the Environment
STAAR 8 – October 2018, pp. 6-11
——————————–
Published: 03 Oct 2018
Review process: Open Peer Review.
Draft First Uploaded: 11 Jul 2018. See draft and reviewers’ comments.
Abstract
Models are representations of reality with suitable omissions. On the one hand, there is increasing reliance on models to inform policy design which are based on mathematics and scientific rigour as they are considered free of any biases. However, on the other hand, there is a call within and outside the scientific community to subject models to increased scrutiny due to value-based assumptions and uncertainties especially with respect to the prediction of future scenarios. In case of climate change and water resources management, while climate predictions are increasingly called upon to make policies for mitigation and adaptation, questions have been raised on the effectiveness of models due to various reasons. Cape Town in South Africa is currently at the centre of this debate as it is in the midst of the worst water crisis it has ever seen. This paper highlights the debate on the reliance of models and the issues with the reliance on modelling using the case of the current water crisis in Cape Town, South Africa. It highlights the philosophical underpinning of what makes models wrong, while arguing that models continue to be useful. This piece makes an argument for risk-based drought management using models, while highlighting the need for transparency regarding the limits to current knowledge in water management that models can simulate. The paper concludes that the onus lies on the scientific as well as decision-making communities to bridge this gap through responsible communication.
Cape Town in South Africa is being watched keenly by climate scientists, hydrologists, water managers and policy makers alike as it goes through the worst water crisis the city has ever seen. Its main water source, Theewaterskloof dam, is depleting everyday and a disaster relief plan is in place which includes heavy restrictions in water consumption, water supply to public water collection points and deployment of armed forces to manage crowds in the city (Chambers, 2018). This severe drought in the Western Cape region is being attributed to the effects of climate change (Welch, 2018). The media has begun questioning the failure to forecast the severity of the drought and a political blame game has ensued (Welch, 2018).
At this point, George Box’s profound aphorism to the scientific as well as decision-making communities in 1976, “All models are wrong; some are useful” is more pertinent than ever. The policy makers in Cape Town did not prepare for the falling water levels in the reservoirs due to the prediction of a wet summer this year (December-January-February) (Wolski, et al., 2017). However, they did not account for the fact that seasonal forecasts are probabilities and not certainties of a weather event occurring (Davis, 2011).
This paper uses the Cape Town water crisis as an example to discuss Box’s aphorism. The paper first conducts a literature review on policy responses to drought and the role of modelling in drought risk management. It highlights that while there is an increasing reliance on models for policy-making on climate change and water resources management, there is a call both within and outside the scientific community to subject models to increased scrutiny. The paper argues that while models are useful tools which are simplified representations of reality, there is a need for transparency in communicating the assumptions and uncertainties underlying the models in order to avoid a crisis scenario like the one in Cape Town. The role of models is not to avert risks but to inform policy-makers of the risks, and then plan for them. The paper concludes that the onus lies on the scientific as well as decision-making communities to bridge this gap through responsible communication.
Policy Responses to Drought – A Review of Literature
Droughts are related to images of knee-jerk policy reactions, often to pacify disappointed constituencies and manage demand when situations have reached crisis points. A drought has impacts on ecology, economy and the society in various ways because of the shortage of water due to variations in the hydrological cycle. It is difficult to estimate the spatial and temporal extent of drought, which means that the start and end of a drought as well as exact locations that it affects are difficult to determine. Hence, drought is little understood at present, and a lot of effort has gone into understanding and measuring droughts.
Africa has been especially vulnerable to the impacts of droughts. Drought has been the cause of 95% of the disaster-related death toll in the continent (Sivakumar, et al., 2014). It has direct as well as indirect effects on crops, livestock and the larger economy due to environmental degradation, water scarcity and the increased vulnerability of households exposed to drought shocks. The indirect effects can often be larger than the direct effects (Shiferaw, et al., 2014). Long-term drought resulted in widespread starvation and famine in many parts of Sub-Saharan Africa, which faces a higher risk of failed crops due to droughts (Shiferaw, et al., 2014). Countries of eastern and southern Africa have been dependent on crisis management as a policy response to drought, which has been rendered ineffective due to lack of data, monitoring capacity and coordination in governance (Shiferaw, et al., 2014; Sivakumar, et al., 2014).
Management of droughts requires planning for water shortages that would affect sowing season for farmers, domestic consumption and wildlife in biodiverse regions. It is important for regions to recognise the risk of droughts and plan for the risk through appropriate policy interventions. A risk based approach towards management of droughts may involve a portfolio of interventions including increasing resilience of agriculture, augmentation of water provisioning capacity for domestic and industrial supply by diversifying the supply options to include waste water reuse, managing demand and restoring the natural resilience of ecosystems as well. Sayers et al. (2016) have argued that the challenge of managing droughts requires a change in approach from crisis management to strategic risk management. They have outlined a Strategic Drought Risk Management framework that relies on a strong scientific understanding of drought indicators for monitoring, planning and decision-making.
Risk management approaches are generally strategies created with the awareness of the inevitable risks while “pursuing positive goals” (Hansen et al., 2014 in Shiferaw et al., 2014). Sayers et al. (2016) define a risk management approach as “a continuous process of data gathering, analysis, adjustment and adaptation of policies and actions to manage drought risks (over the short term and long term). Sivakumar et al. (2014) argue for National Drought Policies in countries that “place emphasis on risk management rather than crisis management” by using drought indicators in monitoring and forecasting droughts. While they give clear details of early warning and prediction systems for African countries based on global circulation models, they do not outline any roadmap for a regime for measurement of local data in these data sparse conditions. Thus, management of droughts using a risk based approach involves the identification and monitoring of variability in hydro-meteorological cycles.
Moreover, droughts should be viewed as a long-term development challenge which requires investment in preparedness and transformative policy responses. Decision-making for droughts should include planning, preparedness and monitoring using reliable drought indices which are suitable for the geography and context of East Africa. There is a need for better data gathering and monitoring capabilities to change the approach of drought management from a crisis based approach to that of risk management.
The Role of Modelling in Water Policy – A Case of Cape Town
The regions of eastern and southern Africa are characterized by mainly sub-humid and semi-arid climates. They have a pronounced dry season in the year and the variability of precipitation is concentrated in shorter time scales. The rainfall variability is directly dependent on the global circulation phenomena such as El Niño-Southern Oscillations (ENSO) and the La Niña cycles as well. The Inter-Tropical Convergence Zone (ITCZ) passes through the sub-Saharan African region, and ENSO also impacts the ITCZ and global wind currents. Thus, ENSO has a strong influence on the anomalies in rainfall over many parts of the sub-Saharan African countries (Masih, et al., 2014). These impacts may vary seasonally and geographically within the region.
The Western Cape region has a climate with winter rainfall and dry summers. The region has been historically drought prone with long-term forecasts predicting more prolonged dry periods (Jaubert & Hewitson, 1997). Cape Town is completely dependent on surface water, with all its rivers dammed, and the impacts of droughts are a common phenomenon. However, the city continues to manage droughts in a crisis mode with the municipality enforcing restrictions on domestic consumption every time there is a drought (Sorensen, 2017).
The current drought in Cape Town is supposed to have a return period of 400 years, although this is based on limited, coarse resolution or bad quality data (Wolski et al., 2017).The prolonged dry period that led to Cape Town’s current drought was not predicted by most weather forecasts. As far as weather models are concerned, there are issues of limited data, coarse resolution and scale of models used for seasonal forecasting in Southern Africa (Davis, 2011). Further, at present the decadal forecasting of climate change is experimental. Further, water infrastructure planning is medium to long-range in nature. Moreover, while the models are either regional or global in scale, the policy response towards adaptation of water resources management is expected at a municipal scale (Mukheibir & Ziervogel, 2007).
A Philosophical Perspective to Modelling – Why are Models Wrong?
Chorley and Hagett (1967) define models as “a simplified version of reality built in order to demonstrate certain properties of reality.” Models can be descriptive, visual, iconic or numerical. Based on this concept, models can range from definitions, maps, case studies, flow charts to complex numerical models that simulate the material world (Brunet, 2001). In case of climate change and water resources management, models are increasingly used to measure historic patterns and predict future events within environmental systems.
Why are models wrong? The answers to this question such as the neglecting of processes of society, over-parameterization and their mechanistic nature (Brunet, 2001) seem superfluous when one scratches the surface to reveal a more fundamental philosophical basis underpinning this statement. In order to examine, these philosophical issues in present-day modelling, it is necessary to understand what makes a useful model.
Models are essentially tools to test hypotheses regarding the material world. They should be deductive in nature instead of purely inductive or data-based, since deductive methods use both scientific and empirical techniques and involve “logical comparison of conclusions, comparing with other theories and the empirical application of the final conclusion” (Popper, 1959). Further, unlike normal research, models should continue to challenge the “paradigm choice” of science and not tend towards cumulative research based on methods and concepts already in existence (Kuhn, 1962). Problems should be tested using models keeping in mind that that they are most useful when they challenge existing theories instead of demonstrating the truth in them (Oreskes, et al., 1994).
However, Oreskes et al. (1994) highlight models for policy-making cannot demonstrate the truth (verify) or lend legitimacy (validate) to the predictions because the natural world is an open system. They also argue that there is a bias of “affirming the consequent” in the scientific community and that there is no absolute way to know if models truly represent all the phenomena of the natural world or only exhibit the relative performance of dependent parameters with respect to empirical observations. This argument of theirs is in line with Hume (1999) who examines the nature and foundation of human reasoning and states that demonstrative reasoning entails all ideas, including models, that judge the future based on past experience. He further states that “whatever is intelligible and can be distinctly conceived implies no contradiction and can never be proven false by demonstrative reasoning.” Beven (2018) tries to bridge this gap in modelling by suggesting “model rejection” and “limits to acceptance” as the basis for acceptance of models for decision-making in order to introduce the rigour that Oreskes et al. (1994) have pointed out is lacking.
Towards Drought Risk Management – But, Can Models Be Useful?
The failure of seasonal forecasts in Cape Town is a symptom of the very issues that have been highlighted in this paper. However, despite limitations of models and data in the Western Cape Region, there is a need for modelling to underpin the policy responses to droughts. Drought management needs to be risk-based with the acknowledgement of inevitability of drought risk, rather than a crisis management response. Existing literature in drought science and drought policy highlights the role of modelling droughts for more effective policy responses. Measuring and understanding drought risk while being cognisant of embedded uncertainties is the foundation of robust drought risk management policies.
Thus, models are extremely useful tools that support decision-making in these times of increasing uncertainty. The issues of uncertainty, validation, verification and confirmation of models should be communicated beyond the scientific community to the end-consumers of the forecasts from these models – the policy-makers as well as the public. There is a need for increased transparency and responsible communication by the scientists and decision-makers to retain the usefulness of models and prudently identify trade-offs.
Bibliography
Beven, K., 2018. On hypothesis testing in hydrology: why falsification of models is still a really good idea. WIREs WATER.
Brunet, R., 2001. Models in geography? A sense to research. Cybergeo: European Journal of Geography.
Chambers, D., 2018. Day Zero: The city of Cape Town is about to run out of water – its main reservoir is only 12% full. [Online]
Available at: http://www.independent.co.uk/news/long_reads/day-zero-cape-town-drought-no-water-run-out-reservoir-supply-12-per-cent-16-april-south-africa-a8195011.html [Accessed 07 02 2018].
Chorley, R. J. & Hagett, P., 1967. Models in Geography. s.l.:s.n.
Davis, C., 2011. Climate Risk and Vulnerability: a handbook for Southern Africa. s.l.:s.n.
Hume, D., 1999. An enquiry concerning human understanding. s.l.:Oxford University Press.
Jaubert, A. M. & Hewitson, B. C., 1997. Simulating present and future climates of southern Africa using general circulation models. Progress in Physical Geography, I(21), pp. 51-78.
Kuhn, T. S., 1962. The Structure of Scientific Revolutions. Second ed. Chicago: University of Chicago Press.
Masih, I., Maskey, S., Mussa, F. E. F. & Trambauer, P., 2014. A review of droughts in the African continent: a geospatial and long-term perspective. Hydrology and Earth System Sciences, Volume 18, pp. 3635-3649.
Mukheibir, P. & Ziervogel, G., 2007. Developing a Municipal Adaptation Plan (MAP) for climate change: the city of Cape Town. Environment and Urbanization, 19(1), pp. 143-158.
Oreskes, N., Shrader-Frechette, K. & Belitz, K., 1994. Verification, Validation and Confirmation of Numerical Models in the Earth Sciences. Science, 263(5147), pp. 641-646.
Popper, K., 1959. The Logic of Scientific Discovery. London: Routledge.
Sayers, P. B. et al., 2016. Drought Risk Management: A Strategic Approach. Paris: UNESCO on behalf of WWF.
Shiferaw, B. et al., 2014. Managing vulnerability to drought and enhancing livelihood resilience in sub-Saharan Africa: Technological, institutional and policy options. Weather and Climate Extremes, Volume 3, pp. 67-79.
Sivakumar, M. V. K. et al., 2014. High Level Meeting on National Drought Policy: Summary and Major Outcomes. Weather and Climate Extremes, Volume 3, pp. 126-132.
Sorensen, P., 2017. The chronic water shortage in Cape Town and survival strategies. International Journal for Environmental Studies, 74(4), pp. 515-527.
Welch, C., 2018. Why Cape Town is Running out of water, and who’s next. [Online]
Available at: https://news.nationalgeographic.com/2018/02/cape-town-running-out-of-water-drought-taps-shutoff-other-cities/ [Accessed 07 02 2018].
Wolski, P., Hewitson, B. & Jack, C., 2017. Why Cape Town’s drought was so hard to forecast. [Online] Available at: http://theconversation.com/why-cape-towns-drought-was-so-hard-to-forecast-84735 [Accessed 07 02 2018].
‘All models are wrong. Some models are useful’ by Shuchi Vora is licensed under a Creative Commons Attribution 4.0 International License.
<< Back to Contents
<< Back to Publications
St Anne's Academic Review (STAAR) A Publication by St Anne's College Middle Common Room ISSN 2048-2566 (Online) ISSN 2515-6527 (Print)