CR 157: Monitoring the National Road Safety Strategy: Cost-Effectiveness of Road Safety Measures (1995)
This study reviewed what economic appraisals have been performed on road safety measures within the priority areas identified in the National Road Safety Strategy and Action Plan, and appraised their methodological soundness. The focus has been to establish the cost-effectiveness of measures, and to discuss the role of cost-effectiveness analysis in relation to the current reliance on cost-benefit analysis for resource allocation decisions.
A literature search was conducted of Australian and international transport research and related databases, using university and transport sector library networks. The search keywords were framed in terms of road safety, road trauma prevention, effectiveness, evaluation, economic appraisal, cost-effectiveness and cost-benefit analysis. The aim was to identify effectiveness and cost-effectiveness studies that could be utilised in road safety priority settings. Inclusion criteria also extended to studies which presented sufficient effectiveness and cost data to construct a meaningful cost-effectiveness analysis.
Despite the comprehensiveness of the literature search, only a few genuine cost-effectiveness studies of road safety programs were found. As in the transport sector generally, the methodology used by the majority of economic appraisals, and in effectiveness studies that included economic analysis, was cost-benefit analysis. These varied in the strength of effectiveness evidence, and in those most concerned with program planning or funding proposals, the effectiveness was often only a crude estimate.
There was an extensive literature on the effectiveness of road safety programs, but mostly without an economic appraisal component. Again these studies varied widely in the quality of their design, their data, and analysis. Due mainly to the time constraints of this consultancy, it was decided to confine our analysis to only those studies which had an economic component and that were, or could be converted to, cost-effectiveness analysis. A full review of road safety effectiveness studies is probably warranted, but would be a major undertaking, and beyond the scope of this study.
A cost-effectiveness approach was applied to those studies where there was sufficient data for the analysis (ie. studies which included both effectiveness information, even if only an estimate, and cost data). Convertible studies were compiled on a database (MS Accessâ 2) using an economic analysis framework developed for this study. The total number of studies so treated was 48. Specific configurations of the database are presented in table form in the report.
Field consultations were conducted with representative interstate road safety stakeholders, with face to face meetings taking place in Melbourne, Canberra, Sydney, and Adelaide. Advice was sought from the perspectives of NRTAC, state road safety managers, road trauma researchers, insurance offices and road user representatives. Project time-line constraints prevented a wider direct consultation, but there was some indirect feedback on preliminary findings from an Austroads meeting held towards the end of the consultancy.
Overview of Findings
Cost-effectiveness analysis of road injury interventions is a useful supplementary form of analysis to cost-benefit analysis . For some research questions, it may be the preferred form of analysis. It seems clear that it is possible and meaningful to calculate the cost-effectiveness of road safety programs on the basis of cost per injury consequence. It is also possible to rank programs on the basis of their cost per injury consequence. Cost-effectiveness analysis has the strong advantage that it is more transparent in its decision rule, and easier to communicate to interested parties. It allows the decision maker to adopt a benchmark figure for the value of an injury or fatality and say that a particular program has a cost per injury or fatality prevented less or more than they are prepared to pay. If a program cost more than $5 million per life saved, for example, it may be rejected if the benchmark was $3 million.
Cost-effectiveness analysis seems to provide similar absolute values for programs to cost-benefit analysis, where conventional benchmarks are used. That is to say, programs with a benefit-cost ratio (BCR) in excess of an accepted level of say 3:1, tend to have a cost per life year saved of less than $30,000, which is less than many common health care interventions which have been evaluated. While cost-effectiveness analysis may provide similar guidance to cost-benefit analysis on broad notions of economic worth, it does produce different rankings of programs, especially where minor injury is a large component, and depending on how cost savings are treated in the benefit-cost ratio.
The key limitations on the use of published cost-benefit studies of road safety interventions, are that the underlying data on the effectiveness of the programs is most often weak, and that the available techniques of evaluation are often not rigorously applied. Many cost-benefit studies had flaws in design which were inconsistent with the theory and practice of the cost-benefit approach. There is little use of good experimental designs to establish efficacy and too much evaluation is based on intuitive guesswork, often without any supporting sensitivity analysis. Similarly, the lack of appropriate current data on the predicted outcomes of interventions would limit the use of cost-effectiveness analysis for resource allocation across program areas. Consistency in the application and reporting of economic appraisal methods would improve the quality of information available to assist decision making.
Many of the published economic studies were for a group of initiatives at a particular level of expenditure. This makes it very difficult to evaluate the cost-effectiveness of future initiatives at perhaps different levels of expenditure. We might have evaluated a program to improve road conditions at 40 black spot locations, but this might tell us little about the cost-effectiveness of the 41st or another 20 elsewhere. There is a need for studies to evaluate the marginal cost and outcomes for interventions and to consider the transportability of the results. Only one study which we looked at (Cameron, 1993) had sufficient detail to allow a marginal cost-effectiveness analysis to be carried out. In many ways this study was a good example of an evaluation with a valid study design and containing sufficient detail to allow transportability.
In the short term decisions have to be made and we would continue to recommend the use of economic evaluation methods. In the longer term we would recommend an effort to improve the standard and consistency of evaluation methods. In this context it is important to note that, where the safety budget is constrained, there are dangers in the simple application of rankings from cost-benefit or cost-effectiveness ratios. That is to say a simple league table of interventions ranked by these ratios can be a misleading guide to efficiency. There may be an important role for mathematical programming and associated techniques in determining the optimal profile of programs and the level of each program when utilising cost-benefit analysis or cost-effectiveness analysis results in the presence of a budget constraint.
In principle cost-effectiveness analysis could be used to assist in resource allocation decisions at all levels of decision making in road safety. It could be used to determine the optimal mix of programs within a priority area as well as to determine which broad area warrants extra investment. Data requirements are a barrier to each of these tasks, but particularly the latter. In order to determine the best place for additional investment we need to know the cost-effectiveness of the marginal project in each priority area. We have identified only a small number of published studies overall which can provide reliable evidence on both effectiveness and costs, and a very small number of studies in some areas. It would be dangerous, therefore, to rely too heavily on current economic knowledge to allocate resources between priority areas. On the other hand, given a longer time frame, it should be possible to frame research in this way. In the meantime, cost-effectiveness analysis can be used within some broad program areas where there is sufficient reliable data. This seems to be the case, for example, in road environment programs where unpublished data may be available on the expected costs and outcomes of particular investments. In the absence of perfect data a great deal can be done with the use of informed views of the likely effectiveness of interventions and their likely costs. When used with sensitivity or threshold analysis it is possible to show which programs are likely to be cost effective, and which will not be even under the most optimistic assumptions. In this case the choice of economic analysis type is not likely to be important. In these cases the economic worth of a project is likely to be robust no matter the assumed economic value of intangibles such as lives saved or pain and suffering avoided. Where more detailed marginal choices have to be made, however, for example on program expansion, it is not only necessary to have good knowledge of the marginal effectiveness of a program, but also of the impact of the program on the relevant budget. In other words what safety interventions have to be given up in order to expand a particular program.
The major conclusions of this study are as follows:
- economic evaluation can provide guidance on resource allocation questions in road safety; both cost-benefit analysis and cost-effectiveness analysis of road safety programs can inform the process of priority ranking, decision-making, and policy development;
- where the research question involves the comparison of projects with quite divergent outcomes (e.g. reductions in road trauma, versus reduced travel time, versus projects in rail or air transport, etc.) Cost-benefit analysis should be considered the most favoured method of economic appraisal, but its usefulness is weakened by the debate on the value of statistical life;
- where the research question involves projects, the outcome of which can be encapsulated in a single physical measure (such as lives saved), cost-effectiveness analysis could be the preferred technique, or at least a very useful adjunct to cost-benefit analysis;
- the reliability of both cost-effectiveness analysis and cost-benefit analysis depend, among other things, on the certainty of the benefits predicted, and hence on the precision and accuracy of program effectiveness evaluation. Current practice in this area (as evidenced by the published studies) requires improvement;
- in the absence of good data on the costs and the effectiveness of interventions a great deal can be achieved by modelling the likely impact on safety and costs using best estimates combined with sensitivity analysis. While sensitivity analysis should not be used as an excuse for poor data collection it can be used to determine when a more detailed study is warranted;
- the main advantages of cost-effectiveness analysis in road safety decision making are that:
(1) the value of a project is not influenced by the value of a statistical life;
(2) cost-effectiveness analysis can be adapted to suit a number of policy arenas, e.g. it can be used to compare road safety programs with other types of programs which result in the same benefits, such as health or emergency services, or even air pollution control;
(3) cost-effectiveness analysis enables a transparent, yet systematic approach to program budgeting, that is easier to communicate to decision makers and the general public e.g. cost per crash avoided, cost per fatality prevented.
- Economic evaluation can provide guidance on resource allocation questions in road safety; both cost-benefit analysis and cost-effectiveness analysis of road safety programs can inform the process of priority ranking, decision-making, and policy development.
- Where the decision involves the comparison of projects with quite divergent outcomes (e.g. reductions in road trauma, versus reduced travel time, versus projects in rail or air transport, etc.), cost-benefit analysis should be considered the most favoured method of economic appraisal, but its usefulness is weakened by the debate on the value of statistical life.
- Where the decision involves programs the results of which can be captured in a single measure of outcome, cost-effectiveness analysis is an appropriated technique. The results, which are framed in terms of cost per life year saved or cost per injury avoided, are easier to communicate among decision makers and to the wider community.
- The reliability of both cost-effectiveness analysis and cost-benefit analysis depend, among other things, on the certainty of the benefits predicted, and hence on the precision and accuracy of program effectiveness evaluation. Current practice in this area (as evidenced by the published studies) requires improvement. In order to assist in decision making a greater effort to standardise methods in the evaluation of road safety measures should be made with greater attention to study design and consistent choice of cost and outcome categories.
- Some program areas have few published economic evaluations and it is very difficult on efficiency grounds to determine which road safety program areas should be expanded or contracted.
- It is clear however that some particular projects are cost effective on any measure and therefore if we are deciding how to use extra resources it is possible to save injury at a relatively low cost with the expansions of these programs. For example many of the road engineering , speed and alcohol reduction programs appear to offer value for money.
- There seems to be a publication bias in favour of programs which have been implemented and it is open to question whether new untested programs would be more cost effective. There are for example few studies of cost-effectiveness in the area of education and licensing, or vehicle safety. In the absence of good data on the costs and the effectiveness of interventions a great deal can be achieved by modelling the likely impact on safety and costs using best estimates combined with sensitivity analysis. While sensitivity analysis should not be used as an excuse for poor data collection, it can also be used to determine when a more detailed study is warranted;
If the objective is to minimise injury at least cost then the use of ratio measures (benefit to cost or cost-effectiveness) to rank programs is a good rule of thumb, but a more appropriate technique is the use of simple mathematical programming. This can be done using standard spreadsheet programs on a microcomputer.
Download Complete Document: NRSS_Cost_1 [PDF: 2586 KB]
Type: Research and Analysis Report
Sub Type: Consultant Report
Author(s): A Harris, R Clark, G Verikios, R Carter, D Dunt & S Crowley
Topics: Economic, Methodology
Publication Date: 01/11/95