The opacity of distributed, small-scale solar PV output | RenewEconomy

The opacity of distributed, small-scale solar PV output

Print Friendly, PDF & Email

The rise of intermittent generation supplies being injected into the NEM has brought some different versions of the same challenges facing electricity system operators.

Print Friendly, PDF & Email

Watt Clarity

The rise of intermittent generation supplies being injected into Australia’s National Electricity Market (NEM) has brought some different versions of the same challenges that have been facing electricity system operators since the first networks were built 100 years ago.

There are many talking about these issues. For instance, I noticed this article from Matt Rennie at EY about the looming challenges for system operators.  Matt talks about the “invisibility of distributed energy” and the growing challenges that this is likely to present in future.  This is currently an issue with distributed, small-scale solar PV, and will increasingly be a challenge (probably a more difficult one) with injections and withdrawals from distributed storage.

Today I’m posting some comments on the invisibility opacity of distributed solar PV.

1)  Comparing different estimation methods

We should start by noting that no-one actually KNOWS, in aggregate, how much small-scale solar PV systems are injecting into the NEM (or indeed in any grid).

Aggregating data in real time (or even delayed) to provide clarity about injections seems like the type of internet-of-things + Big Data problem that would have technology companies clamouring over themselves to attack.  However, what business problem would they actually be solving?

Today in the NEM there are at least two different methods in use to ESTIMATE aggregate output from the growing number of small-scale PV systems across the NEM:

1a)  The ARENA-funded APVI method

As noted on the “Solar Map” project page, ARENA provided $443,238 of funding (75% of the project total) such that the APVI could develop these very useful solar mapping tools and, particularly of interest to us and others, this view of live solar data.  The first incarnation of this live map showed data by state, with a worthy enhancement being the provision of an added layer of detail – down to 2-digit postcode level.

My understanding of the methodology used by the APVI in the generation of their estimates amounts to the following:
(i)  Sample some of the self-reported small-scale PV output data at (this release implies a sample size of “more than” 5,500 PV systems – which is relatively small, given more than 1,000,000 systems installed Australia-wide, but better than nothing);
(ii)  Aggregated output for a state is then converted to a capacity factor by using the self-reported capacity of the systems being sampled; and
(iii)  This capacity factor is then extrapolated up using what the Clean Energy Regulator knows of the installed capacity of solar PV in a state/region (a somewhat delayed metric, but again better than nothing).

Of course there are a number of assumptions made in this method – as there are in any estimation method.  Whilst others might want to argue the finer points about the validity of the method, we’ve been much keener to be pragmatic and incorporate the data to help increase awareness and understanding, where we can.

Given other recent developments, we grabbed about a week’s worth of history of this data (we’ve been building an internal record for possible* later use) and trended as follows:


As we can see, solar output’s been highest in the middle of the day, is highest in Queensland (most sunshine, and most PV panels), and is also very subject right across the NEM to coincident weather patterns (i.e. the above implies, worryingly, not as much diversity as might have been hoped for – though perhaps this is partly a result of the sampling method?)

Readers should note that we technically don’t have permission* to publish analysis like this, but are doing so in the interests of sharing some observations to assist in broader understanding of this particular challenge that could become more significant moving forwards.

*  the permission we have from the APVI extends only to the inclusion of up to 24 hours of history in products like NEM-Watch (more on this below).

1b)  AEMO’s new method

The mid-year release of AEMO’s MMS contains a number of interesting new data sets which we will be incorporating into a range of our products.  Today we want to highlight the “ROOFTOP_PV_ACTUAL” data field.

Unlike the name implies, unfortunately, it seems this data is not an ACTUAL sum of but is, instead, another ESTIMATE.  We know that it’s created using a method that is slightly different than the APVI method because the quick eyeball comparison we did with the chart included below shows mostly similarities – but also some non-negligible differences (the AEMO method, for instance, seems to produce higher peaks):


A quick scan of the AEMO website did not reveal any details of the method used, but I did pop a quick question into the AEMO.

From what the AEMO explained to me:

1)  There is, indeed, a difference in the method used by AEMO compared to the (competing?) ARENA-funded APVI method.

2)  The AEMO will publish more details on their website later (there is none there now).  When this has happened, if I remember, I will add a relevant link below as a comment – or perhaps one of our readers at the AEMO or elsewhere might do so in case I forget?

1c)  Do you know of other methods in use?

As a company focused on aggregating data, delivering it to clients, and making it understandable, we’re always on the lookout for different sources of data.

It’s possible that there are other reasonable methods in use out there that we don’t know about.  If so, we’d ask that you please let us know?

2)  Accessing this data in our software

We’re pleased to announce that we have now incorporated both of these two different data sets into our software, as follows:

2a)  Real time solar PV estimates, from APVI, in NEM-Watch

Regular WattClarity readers and others will know that we’ve been pleased to be able to incorporate the APVI’s estimates for aggregated small-scale PV output into our NEM-Watch entry-level market dashboard since the release of v10 early in 2015.   Here’s a recent capture highlighting the APVI data:


Around the same time as doing this, we also expanded the breadth of the NEM-Watch service by introducing our first NEM-Watch Widget, the Live Generation-by-Fuel-Type mix hosted on the RenewEconomy website, and at a growing number of other locations as well who’ve chosen to embed the widget at no charge:


This data set, and the widget as well, have certainly struck a chord – with a steady stream of congrats coming in since that time, some of which has been highlighted here.  We also continue to receive a steady stream of feature requests with respect to the widget, some of which we have already addressed.

Some of the requests/suggestions have been to do with historical data – we’ve been (disappointingly) unable to help these people as we would like, by incorporating the historical small-scale solar PV output estimates into our NEM-Review (trending, slicing & dicing) product.

I understand that there were historical reasons why we were not originally granted permission to include historical data in this way –  but, I must admit, I struggle to continue to rationalise this (now a year-and-a-half since we incorporated live data into NEM-Watch, and two-and-a-half years since the APVI map went live), especially in the light of ARENA’s strong knowledge-sharing ethos.

It’s frustrating to know that our diverse range of customers have been denied the opportunity to gain insights into the nature of distributed rooftop PV that we looked to do early in 2015.  It’s also unfortunate to say that we’d basically given up on that idea, given the time-lag.

Surely this was not what ARENA had in mind in provision of such a large funding amount, coupled with their keen interest in knowledge sharing?

2b)  Historical solar PV estimates, from AEMO, in NEM-Review

Thankfully, the recent mid-year release at the AEMO has helped us over that hurdle.

For various reasons to do with the APVI estimate, and the AEMO estimate, we’re not including the AEMO data in our real-time dashboards.  However we are now including this into our NEM-Review historical “slicing and dicing” tool:


Existing clients will see that the data has already been flowing through for a few weeks (AEMO has not yet back-published much historical data, but at least in NEM-Review you’ll be able to start trending data from May onwards).

Clients should select “Metered Generation” not “Target Generation”, as noted on the NEM-Review portal here.

We do still have an interest in incorporating the APVI data into NEM-Review in future (clients will probably want to compare between the two different estimates).  However the opacity of distributed, small-scale solar PV output has been reduced a little bit, thanks to the AEMO, hence the relative priority for our clients (and hence us) would be lower.

2c)  Summing it up, for the NEM

For those who think in pictures, we’ve summed the Australian situation up here:


3)  What about international data?

As I noted here back in April, we’re searching for similar estimates of small-scale solar PV injections for the growing number of grids across the world where we are collecting data.  We’re seeking to add these into the internationally-focused LiveGen™ widget we’ve been developing (here, with the help of RenewEconomy):


If you know of sources for solar generation data (or, indeed, electricity generation data for ANY type of plant, as we want to provide the fullest possible picture) in different parts of the world, please let us know.

Source: Watt Clarity. Reproduced with permission.

Print Friendly, PDF & Email

  1. Mike Dill 4 years ago

    All I know is that my solar PV array reduces my utility bill.

    • Paul McArdle 4 years ago

      Same with us, Mike (in fact it’s been several years since we’ve paid anything courtesy of the very generous & long-lived 44c FIT)

      • Stewart Rogers 4 years ago

        The FITs were a total joke though. I got the FIT purely based on economic benefit and it paid back in 4 years and will keep paying for many more years. Subsidies are a terrible idea and are highly inefficient.

        • Mike Dill 4 years ago

          What we really need is equal treatment, which would include those costs currently ignored in the energy market. I am all for getting rid of solar subsidies, as long as the subsidy for health costs that are caused by coal pollution are placed back on the coal plants and producers.

          Until that happens, solar and wind subsidies should continue to balance the situation.

        • Alastair Leith 4 years ago

          Subsidies primed the Australian market and got soft costs down to well below what they are in USA and many other countries. RE has received less than 1% of the subsidies and academic support fossils and nuclear have enjoyed for six or more decades. If you want to get on your hobby horse start riding over there in gasland and coal country. The point is we have this decade to break the back of the fossil fuel industry (and ag sector comes next with 54% of Australian GHG emissions in that sector), FiTs are a small price to pay to say civilisation and most habitats from carnage.

          That you would take the FiT subsidies while it goes against your principles says quite a bit about you and perhaps more broadly about Liberal Democrats I suppose. I guess you on sold your small scale RE certificate RECs also.

  2. Renate 4 years ago

    Thanks Paul for the endorsement of the APVI mapping tools and recognising the value in the live data. We also like what you are doing with the widget.

    You are right – data transparency is critical for better informed decision making. The APVI mapping tools go a long way towards delivering that and we’d love to do more. The ARENA project was to fund the development of the full set of tools which serve a number of purposes, of which the live data is only a small part. The platform has been very successful, typically generating 10,000 page views each month and serving a very wide range of parties. We’re glad you all like it! The latest CER data will be released in the next week or so… Check it out at –

    The ARENA project concluded in 2015 with the delivery of the platform and now the APVI looks after the ongoing data flows, hosting and maintenance of the full set of tools. We are discussing with a number of parties to include more live data and improve data processing methods to improve the accuracy of the Live PV Map data AND to extend the Solar Potential Tool so that anyone can work out how they’d benefit from rooftop solar. So you can expect more features and functions to come.

    The subscription by APVI members to live data streams provides parties such as Global Roam (thanks!) with access to a wealth of data that is otherwise unavailable. The data is provided to the APVI under agreements that limit what we can release that might be on-sold for commercial purposes. However the APVI is able to offer processed and aggregated time series data and analyses, and has worked with partners and stakeholders such as the NSW government and academic institutions to fund enhancements to the platform that make new data or functionality available for free to everyone. Most prefer anonymity – but wed still liek to say thanks, you know who you are…

    If others are interested in accessing more data for their own internal research and analysis, to contract the APVI to do analysis on the data or to join a number of partners in the roll-out of the Solar Potential Tool you can contact us on at [email protected].

    Renate Egan, Chair Australian PV Institute

    • Paul McArdle 4 years ago

      Thanks for the details, Renate,

      As you know, it’s the “for commercial purposes” that I find puzzling.

      It’s a subjective view – I would hope, for instance, that there are many commercial operators who use the APVI’s great tools for commercial purposes (otherwise it would give cause to question the value of ARENA’s significant investment).

      Thankfully, now that the AEMO has started providing data more openly, meaning we can move onto other ways to serve.


Comments are closed.

Get up to 3 quotes from pre-vetted solar (and battery) installers.