Analysis of cases

The cases featured various topics and target groups. Climate change campaigns and energy efficiency campaigns addressed the general public or all households, covered numerous topics and used multiple instruments. Energy efficient building programmes targeted either the construction of new buildings or renovation of old ones in a large scale. Household energy use was also addressed by programmes with narrower scope (e.g. changes in the heating system and/or energy use by household appliances or lighting). School programmes, local energy agencies (advice centres), labelling and renewable energy programmes had more focused target groups or more limited number of instruments. In addition, non-energy programmes were included featuring three eco-driving programmes and one health/safety programme.

The cases contained both well-known successful approaches and innovative new ones using new concepts and communication channels. All reviewed cases were either rather or very successful because of the selection criteria used.

The cases were analysed in five phases: context (pre-planning), planning, implementation, monitoring and evaluation. Planning (including contextual analysis) and evaluation were recognised as two of the most critical phases.

Case design usually included certain amount of contextual analysis. Among the most frequently mentioned contextual elements were national and international policies, the institutional setting, market structures, demographic and socio-economic variables and patterns of energy use. However, the exact process and method of setting programme goals remains quite unclear from the case reports. The programme goals were usually clearly defined as such, albeit in a qualitative rather than quantitative way. In quite many cases, there was lack of market segmentation; the goals were not targeted and the programmes tried to offer “everything to everybody”. Because most of the programme goals were not measurable, it was difficult to evaluate if they were challenging, yet achievable. Despite the lack of measurable targets, quantitative evaluations of results were made in some cases. The lack of measurability can be a hindrance for acquiring programme financing when other types of programmes “compete” over the same funds.

Many of the programmes operated with quite formal plans. However, most were not based on scientific theories or evidence. Instead, design seemed to rely more on the experience accumulated to the programme managers and implementing organisations.

A rather limited variety of policy instruments was used. Because of the project objectives, all cases were to contain at least one type of communicative instruments. Regulatory and economic instruments were also used but in a limited number of cases. In terms of communication channels, traditional ones still dominate.

Although the average programme was sizeable at €3.8 million, many small programmes struggled with budget constraints. Also lack of project personnel and time constraints were reported. The average duration of a programme was just under three years. Although some programmes had been running for ten years or longer, there were also numerous short-term programmes.

The results regarding the planning of monitoring and ex-post evaluation were mixed. In many cases ex-post evaluation was planned from the outset of the programme and necessary info rmation was collected during the monitoring process but there were also programmes which had not been subject to comprehensive ex-post evaluation or it was not planned in the planning phase.

In some cases monitoring was used for effective performance management which managed to reveal problems in implementation. The case studies utilised monitoring and gathered feedback by various means. It was very common to monitor the distribution of various materials, number of participants, web-site visitors, subsidy applications, contacts with advisors, market changes and user opinions/satisfaction.

Some level of evaluation was carried out for most of the programmes because it was a selection criterion the case studies. It was most common that evaluation was carried out ex-post but in longer programmes also mid-term evaluations or in permanent programmes periodical evaluations were conducted. No attempts to evaluate the programme impact ex-ante were reported. A multitude of ex-post evaluation methods for programme impacts were reported. These ranged from participant surveys, testing and comparison with control groups to top-down method evaluating the impact of several programmes focusing on the same target group.

Process evaluation (25 cases) was slightly less common than impact evaluation (29 cases). Frequently reported problems included difficulties in the co-operation and motivation of the different implementing parties, problems in the communication channels and messages, and lack of or excessively negative feedback. It was also recognised in some cases that planning should have been better (lacking e.g. background studies and needs assessment) and more time should have been allocated.

Evaluation of the cost-effectiveness of the programmes was a rarity. This is somewhat surprising given the importance of cost-effectiveness in justifying financing decisions. The likely explanation is the difficulty of quantitative impact evaluation.

The hypothesis of the Project was that energy-related behavioural change programmes do not have the high impact they could potentially have, because in general 1) they have little basis in relevant theory, 2) concentrate mostly on motivational factors only, 3) follow a scattergun approach, 4) have rarely a prior diagnosis or evaluation and assessment of behaviour and 5) do not often lead to ongoing activities. The findings were more positive but revealed room for improvement in all areas. Particularly, there is need to increase the programme planners' knowledge on theoretical background, to improve the focus of the programmes (e.g., by market segmentation and target marketing) and to aim at long-term approaches.

		quantitative analysis
		qualitative analysis