Guide for the evaluation of a warning system for people vulnerable to heat and smog
Weather warning systems are among the main policy tools used by public authorities to protect the population from vagaries in the weather. More particularly, weather warning systems are implemented so that the government can monitor weather conditions and issue warnings when extreme weather events, such as heat waves, cold waves, heavy rainfall, etc., threaten the population. As is the case in all public interventions, it is important to analyze the performance of these systems, in order to evaluate their contribution to the population's safety and well-being.
In this report, a guide for evaluating warning systems for people vulnerable to heat and smog is presented. It is intended for managers and evaluators of these systems. It provides them with an analytical framework and methods for evaluating their relevance, implementation and impacts.
The guide proposes that relevance analysis must address the need for warning systems for people vulnerable to heat and smog. More particularly, the emphasis should be on knowing whether the population in general, or some social groups in particular, require a warning system to be well informed about heat waves and smog episodes and about the behaviours to adopt to protect themselves from these vagaries.
It is also proposed in this guide that the implementation analysis focus on the capacity of the warning system to detect heat waves and/or smog episodes and to warn people who are vulnerable to them at the appropriate time. The quality of the warning messages, mainly the accuracy, utility and clarity of the embodied information, are also part of the implementation analysis.
For its part, the impact analysis investigates the effects of heat and smog wave warning messages on the following:
- Knowledge about the occurrence of heat waves and/or smog episodes, their consequences on health, as well as the most effective protective behaviours,
- Attitudes towards the recommended behaviours during heat waves and/or smog episodes,
- Perceptions of social norms relating to the recommended behaviours during heat waves and/or smog episodes,
- Perceptions of the capacity to adopt the recommended behaviours during heat waves and/or smog episodes,
- Intentions to adopt the recommended behaviours during heat waves and/or smog episodes,
- Behaviours during phases of heat and/or smog, and finally,
- The health status of vulnerable people exposed to heat waves and smog episodes.
The guide also proposes 28 indicators for evaluating heat waves and smog episodes warning systems. This list covers the main aspects of the evaluation, namely the analysis of the relevance, the analysis of the implementation, and the analysis of the impacts of heat waves and smog episodes warning systems.
The methods for evaluating heat waves and smog episodes warning systems are also included in this guide. Particular attention has been paid to the main techniques of impact evaluation, namely the experimental methods, the difference-in-difference estimator, propensity score matching, panel data analysis, time series analysis, and the instrumental variables. The strengths and weaknesses of each of these techniques are presented.
Finally, the evaluation of warning systems is considered, in this guide, to be a complex operation that requires advanced knowledge of evaluation methods as well as the capacity to adapt them to the particular context of each system. Considerable funds are also necessary to finance the different evaluation activities, such as the development of the evaluation protocol, the collection and analysis of data, and the dissemination of the evaluation results. The support of public health authorities is also needed to properly carry out a comprehensive evaluation of warning systems as suggested in this guide.