As Australia’s power system becomes increasingly weather-sensitive, forecasting is emerging as one of the most strategically important capabilities for system operators, generators, retailers and market bodies. Climate change is accelerating the frequency and intensity of extreme weather patterns - heatwaves, volatile cloud bands, dust storms, tropical systems and rapidly shifting solar irradiance - pushing traditional forecasting models to their limits.
For a grid undergoing a once-in-a-century transition, the accuracy of both short-term operational forecasts and long-term demand projections is now integral to maintaining system security, designing efficient market operations, planning new transmission infrastructure, and enabling the continued growth of distributed energy resources (DER).
To take a detailed look at how forecasting is evolving, Energy Insights spoke with the Australian Energy Market Operator (AEMO) about its expanding forecasting toolkit, its increasing use of machine learning, and remaining data gaps that are becoming more material as renewable penetration deepens.
AEMO forecasting has shifted from a supplementary support function to a frontline operational requirement.
“Weather forecasting is essential to operating power systems, in addition to other factors such as temperature and potential demand,” a spokesperson told Energy Insights.
“AEMO uses specialised forecasting systems, which have a combination of physical and statistical forecasts to predict energy output from rooftop solar, as well as wind and solar farms.”
AEMO integrates an extensive set of data streams into its operational models. These include:
“All this information flows into AEMO’s forecasting teams every minute of every day and is fed into AEMO’s models, which are becoming more complex with new capabilities,” the AEMO spokesperson added.
As electrification accelerates and variable renewables dominate new investment, AEMO notes that the power system’s sensitivity to weather conditions continues to increase, requiring continuous refinement of forecasting methods to manage risk.
More data - sourced locally, regionally and internationally - is being integrated into algorithms designed to deliver more granular and timely forecasts. This includes new observability initiatives that will improve AEMO’s visibility of behind-the-meter DER behaviour, an area historically plagued by data gaps.
Despite improvements in satellite-based irradiance modelling and cloud-tracking technologies, rooftop PV remains one of the most difficult elements of the system to forecast.
With more than 4.2 million rooftop systems installed nationally and continued annual growth, behind-the-meter generation can swing sharply in response to passing cloud, smoke events, or dust storms - often with little forewarning.
AEMO continues to refine its statistical and physical models to better anticipate DER volatility. According to AEMO, accurate rooftop PV forecasting is critical for maintaining the supply-demand balance, ensuring that other generators or storage assets are on standby to fill rapid shortfalls.
One example illustrates the stakes.
In April 2023, a solar eclipse cast a shadow across parts of central and northern Western Australia, briefly plunging sections of the region into darkness and causing rapid declines in rooftop PV output. While the impact on the National Electricity Market (NEM) was limited, the event had a significant effect on the Wholesale Electricity Market (WEM) in WA.
The eclipse resulted in a 700 - 1,000 MW increase in operational demand over a three-hour period - a substantial shift for the isolated WA grid.
AEMO undertook coordinated preparation with industry, including:
These actions underscored the operational risk associated with large, unexpected changes in PV output - risks that will only grow as DER penetration deepens.
Forecast transparency remains a key priority for market participants seeking to improve operational planning and investment decisions. AEMO currently publishes raw forecasts, providing industry with direct access to underlying inputs and assumptions.
However, complexity continues to increase. As part of a recent rule change, AEMO will begin publishing systematic trends and insights related to Forecast Accuracy starting next year. This will give industry greater visibility into the strengths and limits of operational forecasting models.
AEMO’s most advanced modelling framework is the Forecast Uncertainty Measure (FUM), a machine-learning-based system that incorporates more than one billion individual forecasts. FUM quantifies the expected range of errors associated with variable renewable generation forecasts and is retrained quarterly to reflect evolving system conditions.
By “learning” from historical forecast performance - including the meteorological patterns present at the time of previous inaccuracies - FUM provides probabilistic outputs that help operational teams make more informed decisions under uncertainty.
Cloud formations, dust events, bushfire smoke and rapidly shifting irradiance conditions are all critical features captured by the model.
Utilities and retailers across Australasia are also turning to advanced analytics to improve their forecasting fidelity.
At Mercury NZ, Head of Decision Science & Analytics Paulo Gottgtroy explains that machine learning is used extensively in forecasting wind generation and rainfall-driven hydro inflows -both of which have material impacts on the company’s portfolio optimisation.
“Our machine learning models augment more traditional forecasting models with pattern-matching algorithms and incorporating more complex features that capture relationships between weather variables and generation outputs,” Gottgtroy said.
Mercury NZ does not operate solar generation assets, but the principles remain consistent across resource types: integrating conventional forecasts with data-driven ML models improves accuracy, particularly under volatile conditions.
“The success of integrating conventional forecasts with machine learning techniques lies in the ability to continuously update models using new data sources not embedded in traditional weather forecast models,” he added.
Gottgtroy notes an emerging challenge: as AI-based models become more accurate, they often become less explainable.
“The trade-off between explainability and accuracy is most significant when AI model outputs are used directly as part of a decision-management system. Mercury uses AI-generated forecasts in multiple ways, often in combination with other models, to provide recommendations to human decision-makers such as hydro controllers and geothermal operators. We therefore maintain a strong focus on balancing accuracy, speed, cost, and explainability when developing machine learning models.”
For retail applications, the bar is even higher.
“In the retail context, explainability is even more important than in most generation-forecast use cases - particularly when AI is used to understand customer needs.”
Paulo Gottgtroy is speaking at the upcoming Future Grid Summit, 3-4 December at PARKROYAL Darling Harbour, Sydney.
Australia’s future grid will be shaped by deep uncertainty - but also by unprecedented visibility. As renewable penetration accelerates, electrification reshapes load profiles, and weather volatility increases, forecasting will remain central to maintaining system resilience.
AEMO’s ongoing integration of machine learning, expanded data inputs, probabilistic forecasting tools and DER observability initiatives will be critical to ensuring the NEM and WEM can continue to operate securely through rapid and unpredictable change.
At the same time, industry players are building their own advanced modelling capabilities, shifting to hybrid forecasting architectures that combine physics-based models, ML-driven pattern detection and high-resolution meteorological inputs.
Together, these developments signal a future where forecasting is not simply a support function, but a core operational pillar of a modern, decarbonised power system - one where accuracy, speed and adaptability will increasingly determine grid stability, market efficiency and investment confidence.