We are living through what you could call the era of the electricity dilemma.

On one hand, electricity remains a driving force of economic development. On the other hand, electrical infrastructure is not only extremely expensive, but its continued reliance on fossil fuel sources is one of the major contributors to carbon pollution. More than 1.3 billion people still aren’t connected to the grid, largely because of the cost and complexity of the process. Nevertheless, electric power generation is expected to grow by 93 percent by 2040, and the vast majority of that growth will occur in non-OECD nations.

We simply can’t meet this anticipated demand the old-fashioned way.

Fortunately, big data and predictive analytics provide a path forward. These technologies, combined with real-time actuation and control of the power system -- what I call the energy internet of things -- will dramatically alter the future of electricity by enabling us to utilize our energy infrastructure in a more effective way.

The needs of the electric power industry are vastly different from those of most other industries. Utilities provide an essential service to everyone within a service territory, one that has a vital impact on public safety and well-being. Instead of revenue and market share, their biggest concerns are quality of service, stability and reliability. If Facebook shuts down for a day, Twitter is abuzz with jokes. If a utility experiences a blackout, people may die and businesses can go bankrupt. The Department of Energy estimates that blackouts and power quality issues cost American businesses more than $100 billion each year.

To achieve these demanding levels of performance, utilities have focused on integrating multiple levels of redundancy and control. Peak power plants cost hundreds of millions of dollars and might only be used 50 hours a year, but utilities build them because they are proven (if inefficient) tools for counteracting temporary spikes in demand. Some of the objections to renewable sources like solar and wind have been based on the variability that they can introduce. Utilities have compensated for this uncertainty through buffering, brute-force engineering, and deliberately circumscribing options for the sake of control and consistency.

An energy internet of things changes this paradigm by providing utilities with real-time feedback and insight for the first time. Simply put, utilities are finally able to know what their customers are doing and what they want, and they are able to make better decisions to serve those needs. Blackouts become less frequent as predictability replaces uncertainty.

Today, forecasting, a fundamental function that drives practically every operational and planning decision at the utility, is done at the system level. With the energy internet of things, forecasts can be issued for millions of customers every few minutes to fine-tune predictions for power consumption across an entire region, in specific geographic areas, or among users along particular distribution branches. Decisions based on these micro-forecasts can be made in order to unobtrusively direct the flow of electrons to improve the quality of service and shave billions in operating expenses. Just to understand the magnitude of improvement, a mere 0.1% improvement in forecasting at a mid-size European utility with 1 million customers can reduce about $3 million a year in operating costs.

Software-based systems also improve as they absorb more data. Over time, the self-learning capability allows these systems to become more precise in how they harvest power. Consumers and businesses won’t know they are saving power until they get a pleasant surprise on their bill. New technologies such as solar, wind, EVs and storage will be integrated safely and more easily when supplemented by software and predictive analytics, and will give their owners a more rapid return on investment.

A virtuous cycle is easy to imagine. Big data and software analytics will help spread the cost of capital investments in power plants and transmission infrastructure over a wider customer base. This will lower the cost of power, which in turn will make it more affordable and practical to bring the power of electrification to more people. The Federal Energy Regulatory Commission has estimated that the U.S. could avoid building 188 gigawatts of power plants, or approximately $400 billion in capital investment, through dynamic peak power controls.

Implementing and integrating data systems will take time. Caution and security must still underpin any changes -- but change is inevitable. Today, we may not be able to imagine all the applications through which utilities and consumers will interact with data, but one thing is certain: we are at the dawn of a new energy revolution driven by software and predictive analytics, and the winners in this new world will be the enterprises that are able to harvest their data most effectively.

In this new world, data is power -- not just figuratively, but literally as well.

***

Amit Narayan is the CEO of AutoGrid Systems, a leading provider of big-data analytics and cloud computing solutions for the energy industry.