SA blackouts: dud forecasts, lousy software, failing gas plants

The report by the Australian Energy Market Operator into the rolling blackouts or load shedding in South Australia last week reveal a sorry tale of bad management, lousy forecasts, dud software and failing fossil fuel plants.

The System Event Report for February 8, released on Wednesday afternoon, reveals that wind energy was not the culprit, despite the efforts of the Coalition and certain vested interests to make it so. This was a case bad management, and failing technology.

AEMO’s forecasts for temperatures and wind energy in South Australia on the afternoon of February 8 were hopelessly wrong. And when the crunch came after the operator realised its mistake, key fossil fuel infrastructure failed or melted down in the heat.

And then, to make matters worse, the local network operator completely stuffed up the management of the load shedding due to “software” issues – meaning that three times more people lost power than needed.

Wind did not fail – as even The Guardian and the AFR described it – AEMO just botched the forecasts, in the same way it did for temperature.

aemo blackout windIndeed, as the graph above suggests, there was nearly  twice as much wind power generated in South Australia during the afternoon and at the time of the blackout than had been predicted 24 hours earlier, and the market operator saw no problem with that.

But during the afternoon of February 8, AEMO completely misread the weather, changing its forecasts several times to assume ever more wind and falling temperatures, and getting it completely arse-about, because the opposite happened.

What AEMO could not have predicted, though, was the sudden failures of the various gas generators, some of which basically melted in the heat.

We already know that Pelican Point, supposedly the most efficient gas plant in the country, chose not to generate, and AEMO complains it was too slow and inflexible to respond to the change of events.

But another 300MW of gas plant was lost, some of it just minutes before the blackout. (That compares to a gradual loss of 100MW of wind energy below expectations).

aemo blackout gas

AGL’s Torrens Island gas generator, the biggest in the state, lost one 120MW unit on Monday because of a boiler leak. It lost a further 60MW of capacity from another two units just 20 minutes before the blackout because the gas plant could not cope with high temperatures.

Three other gas units also suddenly dropped out of the system just over an hour before the load shedding, one (a Quarantine unit) for no apparent reason, and two other units Lincoln 1 and Lincoln 3, because failed electronics caused the loss of 73MW of power.

aemo blackout generation mixIndeed, it seems the only generation that performed to expectations and forecasts was the state’s significant capacity (705MW) of rooftop solar, another example of the critical role played by distributed energy, and the micro generators installed on the rooftops of consumers.

The South Australian grid cracked at what AEMO says is record demand of 3,085MW. But in fact overall demand was much higher than that for most of the afternoon, but was being met by rooftop solar, which performed exactly as expected during the day.

aemo blackout solar(Much of this output from rooftop solar and the demand from solar households is hidden from the grid because a significant component is consumed within the home).

Rooftop solar provided a peak of 525MW at 1pm and was still providing 175MW of power when the rolling blackouts began soon after 6pm. Its output helped avoid high wholesale prices during most of the afternoon.

How frustrating for solar households that bad management from the grid operator and the network owner should deprive them of power even as their rooftop panels were capable of generating. Another powerful case for battery storage.

Little wonder that AEMO’s press release talked little of the details of the event and pleaded instead for a “unified approach” to energy policy. It is obvious that the tools have to be deployed and managed properly. That may require better policies and an improved culture.

“The complexities and challenges of managing short-notice generation capacity reductions amid high temperatures and increasing electricity consumption are real. And they’re here,” AEMO executive general manager of stakeholders and information, Joe Adamo said

Events do occur. In NSW two days later, load shedding hit the state’s biggest energy consumer, the Tomago aluminium smelter, because two of the biggest coal units went down and two of the biggest gas units also failed, one of them tripping.

In South Australia, there have been two other major load shedding incidents over the past 15 months. One, in December last year, hit BHP the hardest after a network issue in Victoria. More than 300MW was lost in 2015 due to another interstate network issue.

Load shedding occurred in both instances because the fossil fuel plant was too slow to react. In November 2015, it occurred with the Northern coal-fired power generator operating at full pelt, but the gas plants actually made the situation worse and two major units had to be switched off.

This is not about renewables. This is about having good management and the best technology. South Australians, and most of Australia, are paying huge prices for an ageing and dumb grid.

As South Australian energy minister Tom Koutsantonis said in response to the report: This was a litany of errors and the load shedding need not have happened. He is still angry about why Pelican Point was not switched on, an issue barely touched on by the AEMO report.

It’s time to move on and embrace renewables, smart software, battery storage, demand management and energy efficiency – the very things fiercely resisted by incumbents and policy makers in recent years.

Comments

41 responses to “SA blackouts: dud forecasts, lousy software, failing gas plants”

  1. Ray Miller Avatar
    Ray Miller

    Anyone who has been involved in engineering either electrical or mechanical knows of the direct correlation of temperature and reliability. The basic elements of the electrical system the conductors have a temperature coefficient, when they get hot the resistance increases and cooling systems decrease efficiency. The net effect is an ever decreasing output of the energy system directly related to temperature, coupled with the ever increasing probability of the many elements in the system to indeterminacy.

    At the very end of the system this energy is used to drive mostly air conditioning systems pumping out increasing heat from our buildings, at a time when need to pump more, the pumps are performing the worst. This is truly a ‘death spiral’ with major health implication to our population especially to the elderly and sick.

    My main point being we need to acknowledge very quickly, our buildings are the front line of our adaptation defense against ever increasing sever heat events in Australia (as BOM data is showing). Increasing our resilience is very much dependent on retrofitting our buildings and designing our future buildings to such a standard which will robustly protect our population against the current and future climate. Our building energy standards have been crafted from a range of assumptions all of which have been grossly exceeded some time ago. Many assumptions include the cheapness of energy, dependence on air conditioning to avoid hazardous temperatures and a very high dependence on the grid reliability under extreme temperature events.
    It seems no one has studied the physics and thermodynamics of the whole energy system. It really is time for a major rethink. Yes AEMO and building regulations are not up to the task, but then when you are the jockey on a mule this is what you get.

    1. Goldie444 Avatar
      Goldie444

      Energy efficiency will give the quickest gains in energy generation as new renewable generation is built and deployed.

      1. Ray Miller Avatar
        Ray Miller

        Yes agree energy efficiency effort should be top priority. As can be seen by the various NEM load profiles they are very temperature sensitive, so making the NEM load less temperature sensitive would save everyone (use users) $$$ billions. But then again a lot of energy generation and distribution businesses will lose their cash cow.

    2. Greg Hudson Avatar
      Greg Hudson

      Sorry, but to me this sounds like an apology from a gas peaker plant operator, and a very piss poor apology at that. After having gone through the Black Saturday event on 7 Feb 2009 when it was 48 degrees on the balcony of my house (in Melbourne) I can tell you that at no time did the power go off due to the ‘conductors’ getting too hot. That is just pure BS. Adelaide was not in an ‘extreme’ weather event last week, it was just a ‘hot’ summers day, like many in the past, and many to come. The problems may be varied, but blaming ‘hot conductors’ just isn’t going to cut it I’m afraid Ray.

      1. Ray Miller Avatar
        Ray Miller

        Greg, I think you missed the point, our buildings are not resisting heat entry as much as they should be, mainly due to lazy design and construction practices. Even many types of insulation have a temperature derating, we are increasingly using AC to be the ONLY way our buildings are safe, ignoring the fact that the physics of the AC dictate significantly reduced efficiency at elevated temperatures. Our fridges and AC go from 50% duty cycles to 100% and the peak energy draw on the network increases dramatically as AEMO knows.
        From the direct evidence when the NEM is stressed we get the $1400 MWh events and higher probability of failures.
        I think we should be directly using some of the ‘supper profits’ from the extreme prices to feed directly into energy efficiency of our appliances and buildings. The generator business may not like this direction but we all need to try to solve some of the problems as opposed to ignoring them.

        1. Greg Hudson Avatar
          Greg Hudson

          I’m calling you out on your crap statement on conductors becoming too hot on a 42 degree day as being BS. No more.

          BTW, your figures in the msg above are also incorrect (but not BS this time)… your $1400/MWh should actually read $14000/MWh.

          I do however agree with your comment on lazy building construction practices… Having settlement on a brand new house happening today. It is the epitome of piss poor construction, with one exception, the polystyrene wall construction. Everything else sucks. No double glazing, No solar panels, and staggered NE orientation makes solar install extremely difficult and inefficient, instantaneous gas water heating (on the opposite side of the house to where the taps are – what a genius architect this guy was). No shading on North facing glass, no geothermal floor heating/cooling, no solar hot water heating, no linen cupboard for sheets etc, no cable TV connection, no Cat5/6 cabling, no security system, no ducted vacuum, no doorbell, 14kW reverse cycle ducted aircon (heatpump) as the only heating and cooling (I can’t wait to see how much power this baby actually draws – my power meter tells me it is drawing 150 watts on standby!). No sealing around external doors. I’m sure there are more problems to find tomorrow when we move in. I already have a real estate agent booked to come and look at it on 1 March 2018 and hopefully it will sell quickly. I know, you are probably wondering WHY did I buy this HoS (House of Shit)… because my wife loves the look of it. 🙁 Form over function wins, plus happy wife = happy life (supposedly).

          1. Ray Miller Avatar
            Ray Miller

            Sorry about your house, most of the building industry uses no more than certificate 3 training (excepting for surveyors and structural engineers), this is the one industry we need a royal commission but then it will devalue the most of the housing stock by many billions…
            Yes sorry I did mean $14k, I understand the big transmission links are temperature and load dependent, and in a heat wave in Brisbane some years ago a number of transformers got the water sprinkler treatment to prevent them failing. Any number of telecommunications sites are temperature dependent, AC failure is urgently addressed, as electronics stop working, computers need fans cooling for some reason. All to do with the physics of electrical resistance and thermodynamics who would have thought.

  2. Goldie444 Avatar
    Goldie444

    I did read somewhere that AEMO only get and read their weather data from the internet pages of BOM data, just like the rest of us?
    Sorry I can’t find the link and will apologise if I am wrong.

    1. Malcolm M Avatar
      Malcolm M

      The best source of weather data would be BoM’s gridded forecasts, which is what I presume they have used. There are high-resolution models of the major cities, but it appears to have not predicted that the sea breeze that was occurring mid-afternoon would cease by late afternoon. In their report, they acknowledge Weatherzone, who perhaps do some customisation of BoM’s forecast grid.

    2. Rod Avatar
      Rod

      I worked for a state high voltage distribution company once and can confirm back then, we were a client of BoM and received a fax (probably at least a couple of times a day, maybe more).
      I guess from that the controllers had to make a judgement. They also had access to real time lightning strike info. These days with lots of RE this task would be much more critical and hopefully they have a dedicated resource scouring the BoM info from an energy production point of view as well as security.

  3. Brunel Avatar
    Brunel

    Better software should mean using UTC or Greenwich Mean Time. There is no such thing as “NEM Time”.

    And what about the hour that disappears due to fast forwarding the clocks 1 hour?

  4. Don McMillan Avatar
    Don McMillan

    Each time there is a disruption a different element [renewables, AEMO, gas etc etc] is to blame. This is a result of organised bad engineering and a managed market. The most stable systems rely on vibrant market forces where the optimal design can evolve. Businesses reliant of energy need to consider their future.

  5. DevMac Avatar
    DevMac

    “AEMO’s forecasts for temperatures and wind energy in South Australia on the afternoon of February 8 were hopelessly wrong”

    I don’t understand why AEMO needs its own forecasts / forecasters when the BoM exists, and got the forecast almost perfect – the temperature at least anyway, from three or four days out they said it was going to be 41 or 42 and it was.

    Edit: Some discussion of this in other comments.

  6. Chris Fraser Avatar
    Chris Fraser

    So the SA grid got very wobbly when – later in the evening – AEMO realised it could no longer rely on the rooftop output they had been gorging on all day. I’m sure Malcom could be chuffed by the thought he is helping to stabilise the grid by churning out his 14kW – that he supposedly hates to be recklessly reliant on.

    1. stalga Avatar
      stalga

      An ABC article today points out that all bar one wind farm in SA is the direct result of Turnbull’s policy decisions when he was environment minister. More hypocrisy exposed. Labor should try throwing that one at them.

      1. Chris Fraser Avatar
        Chris Fraser

        Indeed they should – they have good facts. But I suspect Labor is overly concerned with not being as able to rant, rave and appear shouty and emotional like the Tomatoheads.

        1. stalga Avatar
          stalga

          I watched a video of Morrison’s speech earlier. He was bellowing like a bull.

          Anne Aly called out Freydenberg for trying to paint $48 billion of investment as a cost to the taxpayer, but I doubt it got any media.

          Bloody Turnbull, what a disappointment.

          Few of us realise yet that a lot of the cost of the transition will come from private investment and homeowners.

          Also. I’ve noticed the last two weeks that every other article Giles or Sophie write appears in the msm a day or two later.
          Great work.

  7. solarguy Avatar
    solarguy

    Perhaps the reason Pelican Point wasn’t switched on is because the Libs and or those who want RE to look incapable. said NO. They have probably been waiting for something to go tits up and when the chance came along………. well the rest is now history.
    Somebody has to start waving the big stick around.

  8. trackdaze Avatar
    trackdaze

    These intermittant gas generators are a worry.

    1. Geremida Avatar
      Geremida

      brilliant!

      1. trackdaze Avatar
        trackdaze

        they’re frightfully expensive to boot?

  9. Alan S Avatar
    Alan S

    The BOM uses satellites, weather radar and supercomputers and correctly forecast 41 C about four day in advance. Obviously the bit of wet seaweed outside AEMO’s back door told them otherwise.

    1. Ian Avatar
      Ian

      Watch the daily forecasts given by AEMO. The day prior forecasts often predict vol events. These rarely come to fruition. You can really only loosely rely on the forecast for the next half hour period. It is very frustrating when you are a manufacturer trying to determine when you need to pause production due to high pool prices.

  10. AllanO Avatar
    AllanO

    For anyone interested there are a couple of articles on WattClarity looking at Wednesday’s events generally (http://www.wattclarity.com.au/2017/02/initial-analysis-sa-load-shedding-wed-8-feb-2017/) and then specifically Pelican Point’s decision not to offer capacity through normal market processes (http://www.wattclarity.com.au/2017/02/examining-why-pelican-point-didnt-bid-full-capacity-into-the-nem-on-wednesday-8-february/).

    There’s an inaccuracy in AEMO’s report about the timing of the loss of 60MW capacity from Torrens Island – 50MW of this loss was rebid at 16:53 not 17:42. Minor in the scheme of things but alarming that AEMO can’t even get this right.

    More broadly it seems implausible that AEMO would rely so rigidly on forecasts of temperature, demand, and wind output that have significant uncertainties, even on the day, and not have at least talked to Pelican Point much earlier to understand their options if things turned sour.

  11. Mike Shackleton Avatar
    Mike Shackleton

    Watching “The Weekly” on the ABC this week the matter of the blackouts was raised (in comedic fashion as is the style of the show) in order to highlight what a non event they were. Despite the failing equipment and the shortfall in capacity, people experienced outages of around 45 minutes. That’s hardly going to destroy the economy and it’s not a massive crisis. For a grid to be 99% reliable, that equates to 87.6 hours a year where power is disrupted. For 99.9%, It’s basically 9 hours a year where you could suffer an outage. Fact is, disruptions are rare.

    1. Giles Avatar

      Too true. But Australians are promised 99.9992 per cent reliability, which means only 11 minutes downtime a year. That’s why the Murdoch media went ape last year when AEMO suggested high renewables might cause this outage to go up to 15 minutes – shock horror – in a high renewables scenario ….. in 2025. No wonder they go bat-shit crazy in a 45 minute outage.

      1. Mike Shackleton Avatar
        Mike Shackleton

        99.9992% sounds like a figure dreamt up by politicians. Who owns anything mechanical or electrical that is that reliable? Especially something 20-30 years old like much of the grid and generation capacity is.

        1. Giles Avatar

          My mistake. It is 99.998%. And it was dreamed up by network owners, who could see an opportunity to use this ridiculous standard to build more poles and wires, gold plate the grid and pass on costs to consumers, who now pay highest grid costs in world. Oh, and the politicians who owned them (at time all networks owned by state governments). Yay!

          1. Andrea Avatar
            Andrea

            The reliability standard relates to energy delivered, not to time. (i.e. 0.002% is the maximum amount of energy not delivered).
            From memory, it was a similar figure under the SECV. There was a review conducted into this about 10 years ago. See http://www.aemc.gov.au/Markets-Reviews-Advice/Comprehensive-Reliability-Review#

        2. Chris Fraser Avatar
          Chris Fraser

          There’s no way they have either 1) performed to this level in the past or 2) have any capacity to do so in future. The economics of home storage start to improve exponentially.

  12. Alan S Avatar
    Alan S

    Does anyone know how long it takes to start a unit at Pelican Point from cold and get it on line? I see various times bandied about without any qualification.
    It’s a CCGT system so can a gas turbine-driven alternator go on line in open cycle mode before the waste heat boiler heats up and its steam turbine-driven alternator can also go on line.
    Please reply only if you know – we have enough guessing on this site.

  13. Alan S Avatar
    Alan S

    Does anyone know how long it takes to start a unit at Pelican Point from cold and get it on line? I see various times bandied about without any qualification.
    It’s a CCGT system so can a gas turbine-driven alternator go on line in open cycle mode before the waste heat boiler heats up and its steam turbine-driven alternator can also go on line.
    Please reply only if you know – we have enough guessing on this site.

  14. Ryan Avatar
    Ryan

    Giles, isn’t Tomago an example of demand management? Not sure why you’ve defined it here as load shedding …

    1. Giles Avatar

      You’re kidding aren’t you? Demand management is voluntary, where the sheddee has an incentive to reduce demand. Tomago was screaming blue murder about the cuts and pleaded for it not to happen. That is load shedding.

      1. Ryan Avatar
        Ryan

        You’re kidding yourself if you think Tomago has entered into a supply agreement that allows curtailment without any commercial benefit to themselves. The fact that they complained when that agreement was executed does not mean it wasn’t a demand management exercise https://www.agl.com.au/about-agl/media-centre/article-list/2017/february/agl-and-tomago-agreement-in-place-to-curtail-electricity

  15. James Ray Avatar

    Local electricity trading makes solar and storage more economically viable, which will help to accelerate their deployment, and thus make the grid more reliable (as well as demand management and energy efficiency). It is also important to reduce the need for air conditioning by designing with climate, using external roller blinds, cool roof paint, etc. I recommend looking at this campaign: http://www.foe.org.au/rase_campaign, and this petition: https://secure.avaaz.org/en/petition/The_AEMC_Reconsider_the_change_request_for_local_electricity_trading/?cKTpNib.

    1. Don McMillan Avatar
      Don McMillan

      Socialists always believe if they are in charge the “trains would run on time”. History has shown different outcomes. The reality is we have an expensive, unreliable energy network. There is huge investment dollars available for renewables which have not been taken up. So here is an opportunity for you, design, cost, development a business plan and go out and raise the money. So forget petitions go out and show us how to do it.

      1. James Ray Avatar

        Yes I realized pretty quickly that petitions weren’t doing much. I’ve been contributing to Ethereum as I think it is the technology that has the most potential to affect positive change, including in the renewable energy industry. I’m still working part-time at Sungrow. Once Ethereum is scalable I’ll probably look into starting a business with a dapp, or buy out incumbent industry players with tokens that have a one-to-one mapping with the company share, with the aim to get a voting majority.

      2. James Ray Avatar

        Yes I agree that history has shown that socialism hasn’t been very successful, but that may be due to greed. But blockchain technology doesn’t rely on trust between individuals, so it could go a way (but not all the way, people ultimately need to be less greedy) to alleviating a lot of problems.

Get up to 3 quotes from pre-vetted solar (and battery) installers.