How the South Australia blackout occurred: what the data tells us

The Adelaide CBD as seen on 7 News Adelaide’s sky cam at about 7pm Wednesday. Yahoo7
The Adelaide CBD as seen on 7 News Adelaide’s sky cam at about 7pm Wednesday. Yahoo7
The Adelaide CBD as seen on 7 News Adelaide’s sky cam at about 7pm Wednesday. Yahoo7

When catastrophic failures occur, people quickly demand explanations and start to point fingers. It takes time to get to the right explanations and point fingers in the right direction. Forensic reviews, done thoroughly, often point to underlying risks which made catastrophic failures more likely.

So, what can we say about the extraordinary “system black” in South Australia 20 hours (at the time of writing this) after it occurred, and after about 75% of demand has since been restored.

At the five minute trading interval starting at 16h20 NEM time (so 15h50 in Adelaide), the system was humming along as normal. The spot price was $60 per MWh, peak demand was 1686 MW of which 535MW was being met from imported power from Victoria (430 MW on Heywood and 105 MW on Murraylink).

Torrens Island was the only South Australian fossil plant producing (from two of its four “B” units, but all four of its “B” units were synchronised. The remainder of the supply was coming from wind farms of which 659 MW in the Northern region and 384 MW in the South East.

If I understand it from the limited information currently available, at around 15h40 Adelaide time, lightning and severe wind knocked out the main transmission lines bringing power from the North, and so dropping about 40% of South Australia’s supply.

The interconnectors to Victoria were already running close to import limits and so could not replace the lost supply by quickly ramping. Torrens Island, which had around 800 MW of synchronised capacity, also presumably could not ramp up quickly enough to replace the lost capacity. Presumably the loss of so much capacity would have quite quickly led to the automatic tripping of the interconnectors and hence the cascading failure of the South Australian power system.

So, what to makes of this? Its much too soon to be certain of the events and even less so what to make of them. But it seems to be the case that the trigger was transmission failure, not generation failure. Specifically, the lines and possibly also key substations bringing power from the Northern region to Adelaide.

It happens to be the case that wind farms were producing the circa 660 MW of power that was knocked off when those lines were incapacitated. But this is not the failure of those wind farms, but of the transmission capacity moving their power. It would have made no difference if the primary source of power knocked out was wind, gas or coal. Playford and Northern, the recently closed coal-plants were also on the “wrong” side of those lines and had they been producing, they too would have been knocked out by the transmission failure.

What could the synchronized gas fired Torrens’ units in the Adelaide region have done to pick up the loss? I don’t know. Was Torrens partially incapacitated in the storm? Presumably with 535 MW of import and knowledge of the lightning and winds, AEMO would have procured spinning reserve from Torrens to cater for contingencies.

We must now work towards a careful and transparent forensic analysis of this event. We must not lose the opportunity presented by this crisis to draw conclusions that reflect a rigorous and unbiased understanding of the power system, its economics and its governance. And, let the sunshine in.

My thanks to Paul McArdle (Global Roam) for his excellent animation of NEM data that has made this early note possible.

 

 

Comments

36 responses to “How the South Australia blackout occurred: what the data tells us”

  1. MG Avatar
    MG

    Agree important not to jump to conclusions and speculate on cause until proper investigation published. Lots of questions to answer about why the procured contingency FCAS wasn’t enough to arrest the frequency fall (contingency FCAS was being procured globally over Heyward at the time, rather than locally within SA), and then why local UFLS within SA wasn’t enough to contain the system. The answer may lie in the ROCOF after the initial transmission loss – we should be very interested in AEMO’s assessment of the ROCOF, and whether the current FCAS products are able to act fast enough in situations of low inertia / high ROCOF.

    1. Farmer Dave Avatar
      Farmer Dave

      Thanks, MG. Have you got the time to re-post what you just said, but this time with all the acronyms explained, please? I know what two of them mean, but the other two mean nothing to me, I’m sorry.

      1. MG Avatar
        MG

        Sure:
        FCAS = Frequency Control Ancillary Services, which Bruce called ‘spinning reserve’ above. These are power stations that offer to, and a paid for, ramping up quickly (in 6 seconds) following a major loss of supply.
        UFLS = Under Frequency Load Shedding, where network feeders progressively trip blocks of load offline in the event that the FCAS response doesn’t adequately bring the falling frequency under control.
        ROCOF = rate of change of frequency. Is related to the size of the sudden supply loss, and the amount of inertia in the system. If the ROCOF is too high/fast, it’s conceivable that the current flavours of FCAS and UFLS won’t be able to react fast enough to prevent a cascading failure like occurred yesterday. Almost all of today’s FCAS is provided by thermal power stations, which ramp relatively slowly. Faster technologies like batteries and loads (demand response) can react quicker, but the current market rules don’t allow them to participate in FCAS.

        The last time SA islanded (1 Nov 2015) AEMO published a report that describes all these mechanisms, and how they worked. Something was clearly different yesterday! https://www.aemo.com.au/media/Load%20shedding%20in%20South%20Australia%20on%20Sunday%201%20November%202015.pdf

        1. Farmer Dave Avatar
          Farmer Dave

          Great, thanks MG. On tonight’s ABC News, Chris Uhlmann made a lot of the “fact” that wind generators are not synchronous, and speculated that it was the lack of synchronous generation on the SA grid which limited its ability to survive the loss of the transmission lines taken out by the storm. This puzzles me, as I understand that it is the rotational inertia of steam, gas and water turbines which helps them control the frequency (yes, provide the FACS), and I would have thought that wind turbines would have a significant amount of rotational inertia.

          Anyway, I seem to remember reading somewhere that wind farms could provide FACS, it is just that the NEM rules do not allow them to. Is that true? If it is true, then once again we are seeing a market tilted firmly towards the fossil fuel incumbents.

          1. Giles Avatar

            Yes, it is true. wind farms can provide synchronous generation and do so in europe. but there is no market for it in australia because that market is controlled by the gas generators. Ullmann’s piece to TV was yet more drivel, resembling closely something that frydenberg’s office/the fossil fuel lobby might say. quite extraordinary really.

          2. MG Avatar
            MG

            Regardless of whether the barriers to entry are rules based or physical capability based, the reality is that none of the wind farms in the NEM seem to be offering FCAS… yet. However the concept of ‘synthetic inertia’ from wind farms is a hot topic that AEMO has been investigating as part of their FPSS initiative, more info here: https://www.aemo.com.au/Electricity/National-Electricity-Market-NEM/Security-and-reliability/-/media/823E457AEA5E43BE83DDD56767126BF2.ashx

          3. George Bush Avatar
            George Bush

            We’ve generally seen limited uptake globally of Wind turbines providing this type of FCAS product, the technology allows it but with the low inertia in the rotatiing mass of the blades there is generally not enough stored energy to ramp up without feathering the blades.

            If feathering the blades is undertaken the cost of providing the service is the lost revenue from energy generation. An interesting challenge without storage.

  2. Paul McArdle Avatar

    Thanks Bruce

    For readers of this article please note that I am still trying to understand what happened, why, and how it might be prevented in future.

    There’s more to this than either a “was due to X” or “wasn’t due to X” simplistic arguments.

    Here’s the first attempt from last night:
    http://www.wattclarity.com.au/2016/09/first-shot-at-trying-to-understand-what-went-on-leading-to-the-sa-blackout-of-wednesday-28th-september/

    Paul

    1. david_fta Avatar
      david_fta

      Power networks can be sensitive to failure of critical components: https://en.wikipedia.org/wiki/1998_Auckland_power_crisis

  3. david_fta Avatar
    david_fta

    So, no transmission redundancy between “the North” and Adelaide.

    1. jmdesp Avatar
      jmdesp

      Seems incredible heard from Europe. We have computers, automated system running scenarios every day, it’s an absolute rule to never run the system with a single point of failure, a single equipment which by going down could cause a large scale blackout. This is all calculated at least one day before so that if the situation is likely to occur, counter measures are taken to avoid it.
      Actually, depending on the occurrence probability, the system is sized to handle two simultaneous point of failure or maybe more.

      1. david_fta Avatar
        david_fta

        Thanks, I daresay Australia’s National Energy Market operators will be looking around for ways of minimising such occurrences hereafter.

    2. Marto Streisand Avatar
      Marto Streisand

      There are 4 transmission lines running from north to south in South Australia. Check this article – 3 were brought down: https://reneweconomy.wpengine.com/2016/coalition-launches-fierce-attack-against-wind-and-solar-after-blackout-93841

      1. david_fta Avatar
        david_fta

        Insufficient redundancy then.

  4. DogzOwn Avatar
    DogzOwn

    How fast can gas peaking plants ramp up? Tests in USA last year showed all big names much slower than claimed. Can anything cope with so many pylons blown over so suddenly? Looking forward to micro grids, home battery and EV storage solutions.

  5. Kenshō Avatar
    Kenshō

    I wish we would stop adding RE to the grid without storage as it’s misleading the mainstream media that RE is necessarily intermittent. Be good if we ceased putting intermittent RE on residential inverter outputs also.

    1. jmdesp Avatar
      jmdesp

      And … where do you find that storage ? Let me tell you, at the moment it just doesn’t exist, at least at the scale required to handle such an event.

      1. Kenshō Avatar
        Kenshō

        To focus more on a new paradigm, we need not begin big. There’s molten salt, hydro and battery manufacturers also assert they are are capable of guaranteeing the supply of large contracts in reasonable time:

        “The competitive process that followed has resulted in Tesla being selected “to provide a 20 MW/80 MWh Powerpack system at the Southern California Edison Mira Loma substation,”

        And:

        “The Gigafactory’s ability to produce at a large scale will allow this system to be manufactured, shipped, installed, and commissioned in three months.”

        https://reneweconomy.wpengine.com/2016/tesla-wins-contract-largest-lithium-ion-battery-storage-installation-world-61917

    2. Brian Avatar
      Brian

      Remember, this was not a RE caused accident. RE does not need storage, but the grid could use storage, RE or not.
      If the grids in this case had 15 minutes worth of battery backup, they would have not shut down, they would have had time to start the reserve generators, and the blackout would not have occurred.

      1. Kenshō Avatar
        Kenshō

        Yes I understand RE did not cause the accident. What I’m raising is the political and social factors in integrating RE to the grid, as well as the technological considerations. In the longer term RE will need storage. At a micro level, a grid-connect inverter needs “grid” or it shuts down. Conversely, my inverter/charger is capable of synchronising its AC output with its external AC input, thus importing/exporting in harmony with a grid. Additionally, the inverter/charger drops the grid if the frequency or voltage is outside the tolerances I programme, while islanding itself and continuing to supply its own local loads. It’s a far superior way to design a whole grid.

        1. Brian Avatar
          Brian

          RE will never need “storage” as most people understand the definition of storage. RE will only need reserve generators just like the grid already has because baseload needs them for load following and peaks. The fuel for those reserve generators will come from wastes converted to fuel, as is happening around the world.

          Maybe it would be a good idea for large solar and wind farms to have their own 15 minute battery backup and fast cold start reserve generators. They could probably contract a higher price for the electricity too. It’s probably too inefficient to run the reserve generators as spinning reserve and they would probably also lose some of their green energy gov breaks.

          I agree. Inverters do not need the grid, many of them are perfectly capable of islanding.
          It’s rather trivial to get one that does not, to do so with a small standalone inverter. Disconnect from the grid, turn on the standalone inverter, and the grid tie inverters will all turn on. Check you local laws before attempting.

          1. Kenshō Avatar
            Kenshō

            I’m suggesting a radically de-centralised grid by beginning at the micro or individual premise level. Inverter/chargers can be built at any level of scale. Ideally, I think city councils need to be able to island themselves from larger grids, especially rural city councils. Rolling out RE/storage at the small level first will popularise and give voters confidence in RE. Such small scale applications on individual sites could be residential, commercial, industrial, agricultural.

          2. Brian Avatar
            Brian

            Yes. agreed. I would add waste to fuels at the city level, I think it’s not appropriate for individuals, but who knows. Govs are moving towards microgrids simply to avoid blackouts. Pretty much all the switch gear is mostly in place, it’s a control issue, which includes markets and politics.

          3. Kenshō Avatar
            Kenshō

            I think the main thing is we need to think about how to simultaneously manage the technology as well as the political turmoil and re-establish voter confidence. Fortunately anyone can take responsibility for their part of a distributed grid, where the leaders exist, whether those leaders are in utilities, city councils or industry. A distributed grid will end vulnerability to large external network disruptions.

  6. Kenshō Avatar
    Kenshō

    As well as understanding what happened, we need to think how to roll out RE from here. It appears the ABC, Frydenberg and the PM have upset confidence in RE. Additionally no grid will cooperate with a distributed paradigm presently. So we need to re-establish confidence in RE by beginning small. Just like individual residences like Redflow’s Simon Hacket’s remained up and running, we need lots of “shining lights” dotted throughout dark environments when this sort of catastrophic failure happens again. RE/storage can be implemented on individual premises such as residential, commercial, industrial, agricultural, that will lead to load defection and that will lead to networks and regulators eventually being forced to cooperate. Then the RE/storage will be willingly and effectively integrated into our mainstream political and regulatory authorities.

    1. Geoff Avatar
      Geoff

      it would be a good idea to know which house holds and businesses were still having power due to solar and battery. Bring this to the media and offer it as a solution. I would be selling this like wildfire if I was in the game

  7. David Osmond Avatar
    David Osmond

    Thanks for the excellent analysis Bruce. When you mentioned that all 4 generators at Torrens Island B were synchronised, does that mean it was offering the same amount of inertia as it would if all 4 generators were actually producing power?

  8. AllanO Avatar
    AllanO

    Good clear article Bruce and thanks also to Paul McArdle for getting his analysis up so promptly.

    A couple of very minor corrections to Bruce’s numbers – there was ~300MW of wind generation in the South East at 16:15 AEST – the other 80 MW was from Ladbroke Grove (gas turbines). Torrens Island was producing 274 MW from three (not two) B units (200 MW capacity each), I’m not clear how Bruce knows the 4th Torrens B unit (TORRB2) was synchronised.

    ElectraNet says that “3 of the 4 transmission powerlines between Adelaide and the north of South Australia” were damaged by the storm – this would refer to the four 275kV lines running from Davenport / Robertstown to Para and Mannum respectively (see Figure 2-2 in https://www.electranet.com.au/wp-content/uploads/report/2016/06/20160630-Report-SouthAustralianTransmissionAnnualPlanningReport.pdf). These lines run on 3 separate corridors almost all the way, so losing three of them in a single event is extraordinary. That said it’s not yet clear where the failures occurred and why that would necessarily lead to the loss of all supply from generation north of Adelaide – we’ll have to wait for fuller information to understand why.

    Given that loss of supply though, and with ~700+ MW already flowing from the South East there is no way the Heywood interconnector and transmission to the South East of Adelaide could have securely picked up the slack, nor could TIPS – or any other plant – ramp anywhere near fast enough to meaningfully address the supply-demand imbalance. The question then is the one raised below by MG as to why the South Australian UFLS scheme did not trigger and reduce load in a more controlled way before the system went black – again we have to wait for a detailed analysis of events before jumping to conclusions.

    Not that that has stopped a bunch of politicians and other highly ill-informed commentators like the ABC’s instant expert on the NEM, Chris Uhlmann, from jumping in with their analysis!

  9. Andrea Avatar
    Andrea

    Great article and discussion. Couple of things:
    – Having the interconnectors trip clearly exacerbated the problem. I had assumed they tripped due to a frequency drop, but Dylan McConnell in The Conversation article suggested it was due to a voltage drop. Or was it due to low frequency oscillations? Any further info on what would cause them to trip?
    – Would it have helped if the interconnectors were both DC? Surely then a frequency disturbance wouldn’t spread to other states.
    – Snowtown wind farms lost about 250MW around 20 minutes before blackout, and this was being picked up by Torrens Island B. Could this have limited the ability of Torrens Island to provide FCAS when transmission was lost?
    – Uhlmann’s point regarding the low amount of synchronous generation in SA deals with the period after SA was separated. So he is not dealing with why transmission loss tripped the interconnector, but what happened after that. He is assuming that SA was no longer able to stabilise the frequency. Do you think he has a point here? It seems to me that importing such a large amount from Vic on such a day when such storms were forecast was quite risky.

    1. Kenshō Avatar
      Kenshō

      Watching Paul’s animation shows the system black happened very quickly. To me this seems consistent with HV going down and generators going offline. Maybe a massive short on the HV resulted in both the frequency and voltage stalling together? Even if there were HV lines that survived the storm, the loads would have probably been far greater than any remaining generation capacity instantly triggering a cascade effect of all generators. With Uhlmann’s comment, even if the wind was configured to potentially stay online if possible, it seems reasonable that any remaining generators would have been part of a cascade effect of all generators going offline, regardless of whether they were “black, brown or green”…

      1. Andrea Avatar
        Andrea

        It’s difficult to tell the sequence of events because the data is only at 5 minute intervals. There actually wasn’t much local gas generation because most of the supply was from the interconnectors and wind farms. If the interconnection had survived, the frequency wouldn’t have dropped sufficiently to trigger low frequency disconnection of other generators. They may have been able to handle it with load shedding. Anyway, I am looking forward to AEMO report, which I am sure will shed some light!

        1. MG Avatar
          MG

          You may enjoy section 5.1 of this AEMO PDF: https://www.aemo.com.au/media/Load%20shedding%20in%20South%20Australia%20on%20Sunday%201%20November%202015.pdf – my interpretation is that because there is so little FCAS contingency that can be sourced locally within SA (partially due to aforementioned rules not allowing DR and batteries to provide FCAS), that the Reliability Panel had told AEMO they can rely on UFLS to remedy imbalances, i.e. shedding load at the network feeder level. I reckon South Australians deserve better…

          1. Andrea Avatar
            Andrea

            There will undoubtedly be changes to the NEM rules to aid SA’s situation after this event, but whether it helps the uptake of more renewables is another matter.

        2. AllanO Avatar
          AllanO

          Without fuller details of the sequence of events at the fine-grained second-to-millisecond time resolution that only AEMO and ElectraNet are likely to have, it’s very hard to draw conclusions. I have heard that flows on the Heywood Interconnector rose to over 1000MW as it naturally attempted to balance the loss of local generation in SA, at which point it tripped on overload protection. If that is what occurred then surely nothing could have saved the SA system from blackout (I find it very hard to believe Greg Hunt’s extraordinary claims to the contrary in the AFR this morning, presented with no hard evidence).

          1. Andrea Avatar
            Andrea

            Well Greg Hunt is calling for more interconnectors, which might have helped. Though that depends on why they tripped. If overload was the issue, then running Heywood at nearly full capacity on a stormy day seems (with the benefit of hindsight) unwise. The question of whether power in the north could have been maintained – well there were four transmission lines and apparently three went down. I am looking forward to hearing how that happened! To paraphrase Oscar Wilde: To lose one transmission line may be regarded as a misfortune; to lose two (or three) looks like carelessness (or at least something that requires an explanation).

  10. MaxG Avatar
    MaxG

    Wasn’t some pollie saying that the wind turbines blew so strong that the power lines got knocked over?
    Na, I made this up; however, I would not have been surprised to hear nonsense like this. Yes, we had it in other forms, which just proves my point, how stupid the leaders of this country are.

Get up to 3 quotes from pre-vetted solar (and battery) installers.