From Climate Scepticism

By MARK HODGSON

Today I received an email from my dual fuel energy supplier offering me a £350 discount on a new Worcester Bosch boiler. I was assured that “[u]pgrading to one of [their] A-rated boilers could improve [my] energy efficiency at home – meaning lower energy bills and less [sic] carbon emissions.” They tell me that if I upgrade my boiler I could reduce my energy bills by £840 p.a.

This struck me as strange, because today I read an article in the Guardian under the heading “Heat pumps: more than 80% of households in Great Britain ‘satisfied with system’ – Exclusive: England, Scotland and Wales survey reports similar response to people with gas boilers”. This article referred to a user survey, commissioned by Eunomia, and undertaken by Nesta, which tells me that “[h]eat pumps are an important tool in the decarbonisation of heating as they are powered by electricity, which is increasingly supplied from zero carbon sources, and are highly efficient.” The Guardian article and the survey both stress the need for more heat pumps to be installed, and we are obviously supposed to stop using gas boilers. It seems that my suppliers haven’t yet received the memo.

Eunomia

As it happens, I’m not about to replace my gas boiler, which (fingers crossed, touch wood) is working fine, despite its great age. Instead, I want to discuss the survey which the Guardian assures me demonstrates 80% satisfaction with “system” (presumably a reference to heat pumps, given the rest of the heading and the subject-matter of the survey it is based on).

The first step, whenever a “survey” pushes a line, is to ask who commissioned it (and in the process to wonder why). A visit to Eunomia’s website soon gives you the gist of their worldview. The first page to greet you includes the news that it has appointed a “sustainability change-maker” as its inaugural CEO; a discussion of the EU’s Waste Framework Directive (“An EU Regulatory Framework for a Circular Economy consistent with 1.5 degrees”); and a Pro-bono Carbon Assessment for Devon Air Ambulance. When contemplating Eunomia’s values, we learn that they have a “Carbon Fund” – “to compensate for our footprint from business emissions” – and they have “signed up to the Science-Based Target Initiative business commitment to pursue efforts to limit global warming to 1.5°C, committing to a 50% reduction target by 2030 of absolute scope 1 and scope 2 GHG emissions from a 2018 base-year, and to measure and reduce our scope 3 emissions.”

Enough said.

The Survey

It doesn’t seem grossly unreasonable to speculate that such an organisation might be looking for a particular result from a survey about heat pumps. However, such speculation would be unfair unless a look at the survey backed up such scepticism. Does it do so? I would argue that it very much does. The Guardian puff piece on the survey failed to mention a number of significant pieces of information that the survey offers up.

The first point to note is the extremely limited response to the survey, despite Nesta’s best efforts to persuade heat pump users to take part. Take-up of heat pumps has been painfully slow to date. No doubt that is why surveys like this – gleefully taken up by the likes of the Guardian – are being undertaken, to try to encourage others to follow suit. It’s not too much of a surprise, then, to find that only 169,139 homes were identified as being within the survey’s sampling frame. Postal invitations to participate in the survey were sent to only around 23,500 of those homes. Despite offering a £200 prize draw for High Street vouchers, with a view to increasing the response rate, only 2,792 responses (about a 12% response rate) were received. Sadly, only 2,549 of those were complete, reducing the valid response rate to around 11%. However, since Nesta has decided only to consider the views of owner occupiers, another 154 responses (around 6% of the complete responses) were also excluded, because the respondents didn’t fall into the owner occupier category. Only 2,395 “valid” responses were therefore received, out of a possible 23,500, so the survey actually achieved a qualifying response rate of just over 10%.

To my mind that goes a long way towards invalidating the claim made by the Guardian to the effect that:

More than 80% of households that have replaced their gas boilers with an electric heat pump are satisfied with their new heating system, according to the first major survey of heat pump users.

The Guardian reports that this was a “survey of more than 2,500 domestic heat pump owners”, but it doesn’t tell its readers that the survey actually tried to reach around ten times that number, but valid responses were received from around only one in ten of such households.

Another interesting point is that people who have fitted heat pumps, despite their expense, are likely to be net zero converts who are enthusiasts for the heat pump experiment. This is borne out by the section of the survey dealing with motivations:

Eighty-five percent of heat pump purchases were planned and 13% were ‘distress’ purchases (made out of necessity due to the failure of a previous system)…

…The most popular motivation to install a heat pump was to be more environmentally friendly (84%). Other popular responses were taking advantage of a government subsidy (61%), saving money (54%), improving thermal comfort (29%), and improving health (12%). Not having access to the gas network or dissatisfaction with storing fuel was a motivation for 48 respondents, who indicated this using the open text response option.

It is interesting, then, to contrast the satisfaction levels of those who enthusiastically chose to install a heat pump in their home, and those who bought a house that was already fitted with a heat pump. I think the following statistics are very revealing:

People who installed a heat pump into their own home were the most satisfied (81% as or more satisfied), followed by self-builders (70% similarly satisfied). People who moved into a new build home with a heat pump were less satisfied (64% as or more satisfied), while people who moved into a house in which the previous owner had installed a heat pump were the least satisfied group (52% satisfied).

Even the enthusiasts, who actively chose to install a heat pump, demonstrate satisfaction levels of only 70-81%. More than one-third of those who were presumably agnostic on the question, but who bought a new home with a heat pump, were not satisfied – and this despite the fact that such homes should be the ones most suited to heat pump use, due to having to comply with modern Builiding Regulations standards with regard to such matters as insulation. Finally, almost half of those people who moved into a house which was not a new-build, but which had a heat pump, were not satisfied. That’s a pretty damning level of dissatisfaction.

And it gets worse. In the section of the survey dealing with limitations, it is acknowledged that:

The MCS Installations Database, alongside the EPC register, was used as a sampling frame for the heat pump survey. The database only includes installations by an MCS-accredited installer, or households on the EPC register with heat pumps installed, and does not record all heat pump installations in the UK. An MCS installation is required for homeowners to access government incentives and subsidies, meaning that some records will be for heat pumps that have not been paid for in full by the individual and have been installed by an installer that has shown a commitment to a certain level of quality and compliance. This introduces an element of sampling bias…

You can say that again. Basically the survey samples only those owners of properties with heat pumps who have gone through accredited suppliers, often accessing grant money in the process. These are the people who are least likely to have encountered problems with rogue suppliers, inadequate or problematic insulation, etc. I cannot agree with Nesta’s view that “as many domestic heat pump installations are likely to be MCS accredited, we believe the sampling bias to be small.”

When added to the small (and self-selecting) sample size, I think the survey results are highly questionable and problematic.

Conclusion

Not surprisingly the Guardian has seized on the Nesta survey as being evidence that “[m]ore than 80% of households that have replaced their gas boilers with an electric heat pump are satisfied with their new heating system, according to the first major survey of heat pump users.” I think the figures show nothing of the sort.