ConfUSEd by the ESOO? You’re not alone

I’d rather not add to the number of conspiracy theories in circulation, but I wonder if there’s a conspiracy to make understanding our electricity system in general, and its reliability in particular, as difficult as humanly possible.

There’s no doubt it’s a sophisticated piece of machinery with a lot of complex science and technology behind it.

We then wrap it in a highly regulated market founded in arcane mathematics and towing a boatload of jargon and acronyms, operated and overseen by a plethora of organisations with obscure names and functions, trying to navigate and administer a morass of laws rules regulations procedures codes and other legalese.

Even when trying to measure and communicate the answer to as basic a question as “is there a risk of power cuts this summer?” we make a complete mess of how best to do that.

Case in point being the reporting of AEMO’s 2019 ESOO this week.

From a quick scan of the press and social media you could draw conclusions as diverse as “millions of households and businesses face being plunged into darkness” to “there’s a 99.998% chance of no cuts at all”. You’d be wrong on both counts.

The reasons why include the very obscure way “reliability” is defined and measured in the NEM, the poor job that’s done in communicating it, the apparently unceasing “energy wars“, and an evident decision by AEMO to become a more public player in the policy debate as suggested by the escalating stridency of its messaging efforts.

So perhaps no conspiracy after all.

Here I’m going to stretch (ignore?) Paul’s promise of a “measured walk through what the document says” to instead focus on one key topic – how reliability is defined measured and communicated, and what this year’s ESOO is actually saying on this subject.

A few headline facts

Before getting into the details, a few headline facts distilled from that ESOO, focussing on Victoria this summer since that’s aroused the most commentary:

• AEMO assesses the risk of needing to curtail some Victorian demand this summer as about 1 in 3
  • if one or both of the two large units currently offline for major repairs are delayed in returning to service; and
  • AGL is not granted permission to extend the operation of two of its Torrens Island A gas units in South Australia over the summer.
  • The apparently miniscule “0.002 per cent” or “0.0026 per cent” figures being thrown about in the press or on twitter are NOT probabilities of power cuts.
• They are a very confusing way of expressing the “average” size of any cuts as a proportion of all the energy supplied in a NEM region across an entire year.
• Given that very large denominator (about 45 billion kilowatt hours for Victoria), you would hope to see very very small numbers indeed!
• The form and amount of any demand curtailment would be very dependent on exact circumstances. Talk of “millions of households” being blacked out for hours is shorthand and potentially quite misleading.
• Victoria this summer is the only region and year over AEMO’s 5 year formal assessment which fails the NEM’s reliability threshold.
• AEMO does not believe that the current reliability measure and threshold are fit for purpose.

What are we trying to measure?

“Reliability” has a very commonsense meaning when it comes to electricity – the power stays on 24/7 with no interruptions. But interruptions can happen for many different reasons – most of them to do with local poles and wires – and the narrow type of “reliability” measured in the ESOO ignores almost all of those reasons, and focusses only on there being enough total supply available to meet everyone’s aggregate demand for power, whatever that level of demand happens to be.

This form of “reliability” is also distinct from system-level supply interruptions caused by unexpected trips / sudden failures of major equipment, bushfires etc, which in industry-speak fall into the system security category. The 2016 SA blackout being the canonical example.

If there just isn’t sufficient generation and transmission available, then some demand has to be curtailed to keep the total at or below what can actually be supplied – unlike government or household budgets, electricity systems can’t run deficits.

“Curtailment” can mean anything from contracting with specific very large users – eg aluminum smelters – to temporarily reduce their usage, through to rolling blackouts of entire sections of the distribution network.

Last summer we saw both forms in action in Victoria to cope with very high demands and supply shortfalls on 24 and 25 January.

Assessing this form of “reliability” over future periods obviously requires forecasts – principally of weather which is a huge driver of summer demand peaks, as well as of output from the increasing amounts of wind and solar generation on the system, and can also reduce the capacity of transmission lines to carry power from generators to load centres.

Forecasts of “generation availability” are also required since (despite what some politicians might tell you) no form of generating plant is 100% reliable “24/7”.

Across a large and diverse generating fleet there are inevitable unpredictable breakdowns and other needs to take plant offline for repairs and maintenance.

Off to the casino we go

With neither the weather nor generation performance being predictable, the forecasting required to fully assess system reliability (even in the narrow sense we’re talking about here) is very complicated, and getting more so.

It’s certainly not good enough to take a “best guess” of peak summer demand, and line that up against total nameplate generating capacity, perhaps allowing for one or two large generating units being offline for repairs and taking a stab at how much output wind or solar might be producing at the time.

Instead AEMO has to (metaphorically) go to the casino and model literally hundreds of scenarios which “roll the dice” on temperatures, the level of demand, the output from variable renewable generation, capacity of transmission, and the availability of large generating units.

Various statistical and sampling techniques underly this work but in essence it’s a form of “Monte Carlo modelling“, which yields probabilities of different outcomes, from no reliability issues in some scenarios (where the weather is benign and generators don’t break down) to scenarios where extensive demand reductions would be necessary because of extreme weather and multiple generation failures- just like the real-world case actually experienced this January.

Now, that modelling is necessarily complicated and sophisticated, but communicating what it says shouldn’t be as hard as it seems to be.

Great expectations

Sticking with the casino analogy, if I roll a fair die, the chance of getting any particular face showing is 1 in 6. If the faces are the usual one-spot to six-spot, then the “expected number of spots” is three and a half.

What? There’s no face with 3.5 spots, and in any case a three- or a four- spot is no more likely to appear than a one- or two-spot.

What does three and half spots being “expected” even mean, and what use is the number?

Its meaning is essentially mathematical, and it represents the average of the number of spots I would see if I threw the dice thousands of times, added up all the individual results ranging from one to six, and divided by the number of throws.

More practically, it would also be the “fair” price for a bet on a single roll of the dice where I paid that price in dollars to play, and the casino then paid me a number of dollars equal to the number of spots on the die.

This “mathematical expectation” is how AEMO reports – is required to report – the results of its reliability scenario modelling. It is essentially the average amount of load curtailment across a large number of scenarios many of which will have no curtailment and others which will have widely varying amounts of curtailment (quantity of load interrupted) depending on the specifics of each scenario.

But in a twist that seems inevitably to catch out almost everyone, that expected amount of curtailment – let’s say 900 megawatt hours which is a nice tangible number – gets expressed as a percentage of total annual energy.

The result is the “expected Unserved Energy (USE)” percentage measure which peppers the ESOO and confuses the hell out of those trying to interpret it.

Problem One: If you think about that, the USE measure takes a small sliver of “expected” load interruption, generally amounting to a few percent of maximum demand, which in any one scenario would typically occur only for a few hours on a single extreme day, and then divides it by the total energy supplied over 365 days.

Of course that’s going to yield a tiny percentage that carries very little intrinsic meaning to anyone.

Problem Two: The averaging process crunches into a single value all the underlying richness of information about the likelihood of no curtailment vs some curtailment, and about the range and risk of different amounts of shortfall (including so-called “tail risks” of very extensive shortages).

Balancing or confusing further?

To be fair to AEMO, its recent ESOO reports have tried very hard to expose more of the detail behind the measure and modelling results and methodologies, and AEMO is pushing hard for revisions of the reliability measure (which are bound to be controversial, at least within the rarified atmosphere of optimal electricity market regulation).

But the fact that the measure, and the reliability threshold which triggers actions for AEMO to arrange for reserve resources, are expressed as tiny percentages can give casual observers an entirely misleading impression of supply reliability.

A very common misinterpretation is that the USE percentage is a “probability of blackout”, which at levels of around 0.002% looks astronomically low.

In actual fact, “probability of blackout”, or “loss of load probability” (LoLP) – like “probability of rolling a six-spot” – is a very simple measure to extract and report from AEMO’s reliability modelling. It’s just the proportion of scenarios (assuming these were all equally probable) in which there is any load curtailment at all. For Victoria this summer, AEMO puts that number at about 1 in 3.

Then, perhaps in an attempt to swing the misinterpretation balance back the other way, when discussing possible amounts of load curtailment, AEMO falls into a common habit of expressing megawatt hours of curtailment by reference to average household power usage levels.

So a Victorian USE of 0.0047% in some downside scenarios, which corresponds to about 2,115 megawatt hours of load curtailment, is expressed as “equivalent to” interrupting up to 1.3 million households for up to 4 hours.

This glides over the fact that if, as in past years, AEMO arranges for short term load reduction contracts with large industrial users like aluminium smelters, much of the load curtailment might fall on those users – who are paid for providing this service – and not on actual households.

Surprise finding – reliability has improved??

There’s much more that could be said and picked out of the ESOO, and I owe Paul an apology for perhaps producing more of a rant than an analysis, but I’ll close with a couple of charts that illustrate – counter-intuitively in view of the general commentary – that compared to last year and apart from this summer in Victoria, AEMO is actually forecasting significantly higher reliability almost across the board in this year’s ESOO than in last year’s. Go figure.

Source: WattClarity. Reproduced with permission.