The problem isn't unique to Massachusetts. A growing number of energy-efficiency professionals are speaking out about the overly complex and archaic way that energy efficiency is measured. 

"We spend all this money, but our ways to verify if they're saving energy are not working that well," said Tim Guiterman, the director of EM&V solutions at EnergySavvy, which touts itself as a modern alternative to efficiency program management.

At a time when nearly every product, service and behavior can be tracked in real time, the efficiency industry still relies on complicated models and outdated data to verify energy savings, said Guiterman.

In states like California and New York, results for programs weren't delivered unil three years later. "It’s as if you had a speedometer in your car that told you how fast you went an hour ago," he said.

The problem is not limited to any one sector, efficiency company or utility. Rather, say experts, the industry is systematically plagued by an outdated way of measuring performance -- partly because efficiency is hard to track compared to energy generation, and partly because of the industry's inability to modernize.

Utilities in the U.S. now spend more than $7 billion per year on ratepayer-funded energy efficiency programs. Within a decade, ratepayers may be supporting up to $15 billion per year in utility efficiency spending, according to government estimates.

The low cost of delivering energy efficiency cannot be rivaled -- it is by far the cheapest resource. But do utilities actually know how much they're saving? That's debatable. 

Source: Lawrence Berkeley National Laboratory

Economists and analysts tussle over the role of efficiency in nationwide carbon emission reductions, the scope of the rebound effect, and how effective building codes are in changing energy use. With different statistical assumptions, wildly different conclusions can be reached about efficiency.

A similar debate is taking place on the utility level, where different approaches to modeling often bring different results. 

"Jurisdictions calculate and define savings differently, utilize different deemed savings values and baseline assumptions, tend to not report uncertainty in results, and apply different levels of independent review," wrote a group of experts at the DOE's State Energy Efficiency Action Network back in 2011. 

The result: "EM&V is sometimes seen as expensive, not credible, not timely, not transparent, and as a burden, not a benefit."

With billions of dollars at stake, that uncertainty is inexcusable, said efficiency industry veteran Michael Blasnik. He and others have found that traditional modeling approaches can be off by double digits.

"To me, the real scandal is how much utilities spend on consulting firms to determine cost-effectiveness. They don't use energy data; they use projections, models and widgets," said Blasnik, now a senior building scientist at Nest Labs. (Blasnik spoke to GTM before joining Nest, when he ran his own consultancy focused on evaluating energy efficiency.)

"Millions of dollars get spent doing these studies where no one knows the assumptions. You could end up with statistical analysis that disproves the laws of thermodynamics," said Blasnik.

In addition, most utilities rely on self-reporting to determine net savings. After issuing rebates, a utility may hire a consultancy to call customers and ask if they would have retrofitted their home or facility without financial assistance. The practice is a way of separating legitimate projects from "free riders."

There's plenty of dispute over self-reporting. Last October, the Missouri utility Ameren sparred with regulators after it claimed 70 gigawatt-hours of savings more than what independent auditors measured. Its reported savings were based on self-reported data.

"When you give people money and ask if the program that gave you money worked, how do you think they'll respond?" said Energy Savvy's Guiterman.

There is no single answer to the industry's EM&V problem. Because energy efficiency cannot be measured in the same way that energy generation can, there are inherent challenges to measuring behavior change. But new approaches to tracking savings are emerging.

California-based Lime Energy thinks it can track savings on a project-by-project basis better than the status quo.

Lime targets the most difficult customer set: small and medium-sized businesses. The company sets up direct installation programs for utilities, hires local contractors to complete the projects, and then charges the power companies for every kilowatt-hour of efficiency delivered with no administrative fees. The savings are tracked in real time by Lime's software.

"EM&V needs to be continuous and instantaneous. Old methodologies have not gotten us there," said Arjun Saroya, the VP of engineering at Lime. "Utilities have become tired of lack of results from larger administrators. We’ve capitalized on that trend."

Seattle-based EnergySavvy is growing its business based a similar premise. The company has built an end-to-end software platform that helps utilities target efficiency opportunities, manage projects and rebates, and track savings through utility bills and meter data.

"With modern cloud computing and data science it's possible to cheaply analyze actual energy savings for every single project, on a rolling basis, and compare with general energy users to remove other effects. What we do is billing analysis on steroids. And that's never been available before," said Guiterman. "It gets them away from the customer self-reported stuff."

These two companies are among dozens in the "intelligent efficiency" sector using better data loggers, sensors, meters and analytics software to fine-tune reporting of energy efficiency. They're all attacking a different part of a major problem: the slow adoption of IT in measuring efficiency program performance.

"M&V practices have yet to evolve to take advantage of the smart grid infrastructure that allows for increased data collection," wrote DOE efficiency experts back in 2011. Four years later, these companies are just starting to change the way utilities track energy savings.

Efficiency entrepreneur and policy advocate Matt Golden, a frequent contributor at GTM, believes utilities are ignoring their most important piece of infrastructure: the smart meter. 

"Moving to energy-efficiency procurement that pays for efficiency at the meter will unshackle contractors and the broader energy-efficiency industry from the trap of current incentive programs and the stifling regulation that inevitably goes with them," he wrote in a recent op-ed. 

Golden put it much more bluntly in an interview: "I'm looking to fundamentally change the way we think about M&V. If you think about efficiency as a commodity and think about programs in terms of resource acquisition, it's way simpler." 

The problem, he said, is that utilities approach efficiency through a coupon-based approach -- sending out rebates and then calculating savings through complicated models and self-reporting in the hopes that the program will be deemed effective months or years later.

As a senior consultant to the Investor Confidence Project, Golden is applying his passion for standards to metering. He's currently working on the Open EE Meter, an open-source tool designed to standardize the way savings are measured. 

A group of home-performance professionals, led by Nate Adams and Ted Kidd, have also been advocating for a simpler way to measure savings and deliver rebates in the residential efficiency space. Called "One Knob," their proposed program is structured around reading savings at the meter -- delivering incentives simply based on negawatts, not prescriptive retrofits demanded by a program administrator.

"Is paying for a negawatt too simple? Well, isn't that what utilities and public utility commissions want? Incentivizing saved energy directly is the fastest and simplest path there," wrote Adams in a piece at GTM last fall.

Almost everyone agrees that M&V needs to improve. But some are skeptical about the singular obsession with the meter. 

"Reading the meter is helpful, but it doesn’t get to all of the factors that you have to account for. That’s just the starting point," said Glenn Garland, the CEO of CLEAResult, the country's biggest efficiency program administrator. Deeper analysis is needed to account for seasonal changes, building occupancy shifts and de-tuned equipment, he said.

The industry is somewhat divided on the severity of the M&V problem in efficiency. Regulators worried about their reputations and the large consulting firms traditionally responsible for tracking programs are generally hesitant to admit the limitations of modeling. Thus, the most passionate advocates for reform still face a resistance to change.

But rapid changes in technology that make reporting easier may finally break through the inertia.

"The industry is colliding with new technology in a huge way -- and that's going to change it," said Lime Energy's Saroya.