The Parliamentary Commissioner for the Environment has
reviewed whether the farm management software Overseer is
fit for purpose in a regulatory context.
The Overseer model is used to manage farm efficiency, nutrient loss and water quality. In his report, Simon Upton hones in on how Overseer is used by regulators to manage nutrient pollution from farms. He recommends the software owners (Ministry for Primary Industries, AgResearch and the Fertiliser Association of New Zealand) do a comprehensive independent review of the model before it becomes New Zealand's official measure. He also suggests that the owners make the software open-source to encourage transparency.
The SMC asked experts to comment on the PCE's report. Feel free to use these comments in your reporting.
Professor Jenny Webster-Brown, Waterways Centre for Freshwater Management, Lincoln University and University of Canterbury, comments:
"There is a saying amongst modellers - 'All models are wrong, but some are useful' (attributed to statistician George Box). There are a great many numerical models used in the management of our natural resources, all of which have their worth as well as their limitations.
"The most useful models are those which
reliably predict events and trends, as confirmed by
observations and actual measurements of conditions in the
environments that have been modelled (a process known as
validation of the model). For modellers to have confidence
in their predictions, they also need to be sure they are
using the model for the purpose it was intended, and know
how the model works.
"Models can rarely be used in “black box” mode, if a reliable, defensible prediction is required. Without a working knowledge of how the model works and processes input data, the modeller cannot know how assumptions will affect the result, or the limitations of the predictions made.
"Overseer is an established, accessible, well-supported numerical model. However, it is increasingly being used for purposes other than those it was designed and validated for, mainly by modellers who do not have access to details of the modelling processes and their limitations.
"If Overseer is to be the model of choice for national and regional scale nutrient loss and load modelling, and widely accepted and adopted across multiple sectors, it needs to be openly reviewed and refined for its use in this context. The modelling process also needs to become transparent. The PCE’s report recommends such a review, as well as a role for central Government to provide national consistency and support for a comprehensive, independent review process."
Conflict of interest statement: None. We use Overseer in our teaching, so that our students (in Water Resource Management) can see how nutrient load limits are currently being calculated in most cases, and how a farmer can see which farm practices might be changed to meet such limits. We convey the limitations of using this model in this way, but also the reasons for its common use for this purpose.
Dr Julie Everett-Hincks, Legal and Scientific Researcher, University of Otago, comments:
"Who Oversees Overseer?
"Aotearoa/New Zealand is facing significant issues in degraded water quality. The National Policy Statement for Freshwater prompted some regional councils to adopt Overseer as a tool to encourage compliance and enforce nitrogen limits on farmers.
"The PCE’s investigation 'identified some important gaps and shortcomings in transparency, peer review, corroboration, uncertainty and sensitivity analyses, and the way the model has been documented'.
"Overseer would not likely withstand legal challenge, but more importantly is it right to burden farmers with regulatory compliance when the tool used cannot reasonably measure nutrient losses? In its current form and governance structure, Overseer is not fit to be a regulatory tool.
"Overseer was never designed to estimate limits and enforce compliance. Its original design was a nutrient budgeting tool for farmers. Overseer has been adapted over the past four decades, both in its governance structure and its capacity to meet regional council demands.
"Overseer Ltd’s strategy is making Overseer 'the trusted on-farm strategic management tool for achieving optimal nutrient use for increased profitability and managing within environmental limits'. However, to be trusted it must first become transparent.
"The PCE recommends a comprehensive and well-resourced evaluation of Overseer for it to be considered a regulatory tool. The PCE also recommends that Overseer adopts an open-source model, which conflicts with the current business model.
"To regulate water quality in Aotearoa, I believe that a purpose-built model is required."
No conflict of interest:
Professor Troy Baisden, BOPRC Chair in Lake and Freshwater Science, University of Waikato, comments:
"Almost any Kiwi involved with farming or nutrient management will have heard of the Overseer model. It was and is intended to be New Zealand's main computer tool developed to support farmers’ efforts to minimise nutrient losses. Over a dozen years, use has shifted beyond this intention and the model is now required by many regional councils setting limits on nitrogen losses from farming to maintain water quality.
"Overseer’s path from helpful calculator to regulatory tool has been bumpy for both councils and farmers, and that is the issue the Parliamentary Commissioner for the Environment (PCE) has investigated.
"On the upside, Overseer is well used and reflects some of our farming systems well. That would be perfect if Overseer was still mainly a calculator to improve farm nutrient management. But, when used to enforce regulation, Overseer lacks the openness and transparency needed for scientists to review model results or develop improvements.
"The report concludes that Overseer has achieved a safe monopoly on regulatory use. Essentially, Overseer is the best model we have, because it is the only model we have. As a result, there are reasons to recommend the Government address issues of openness, enabling the science community to do more to check and improve Overseer."
No significant conflicts of interest.
Dr Suzi Kerr, Senior Fellow, Motu Economic and Public Policy Research, comments:
"This is a really thorough, useful and long overdue evaluation of the OVERSEER model as a regulatory tool.
"It identifies and explores key fundamental issues such as governance, transparency and quality control; addressing these with some of the suggestions in the report could significantly improve the quality and acceptability of the model.
"At the same time it makes clear the inevitable limitations of any model, and hence our need to accept inaccuracy if we want to have flexible policy.
"Reassuringly, many of the modelling challenges it identifies are specific to water quality where, for example, leaching rates depend heavily on local farm conditions so are data heavy and where the importance of leaching from a specific farm for water quality goals depends on catchment-wide processes.
"Creating a strong version of OVERSEER, or a similar tool, to estimate greenhouse gases is a significantly easier task though one that is still in progress."
No conflicts of interest.
Professor Richard McDowell, Chief scientist, Our Land and Water National Science Challenge comments:
"Catchment water quality is driven by action at the farm scale. Overseer is ‘good enough’ to model most complex farm systems and estimate nitrogen and phosphorus loss in response to farm practices. While parts of the model can be improved there’s a risk that focusing on uncertainty delays action on farms.
"My opinion is that the best use of Overseer is to connect it to farm environment plans, and I would like to have seen this included as a recommendation. Farm environment plans should fall under the same national guidance recommended by the Commissioner for Overseer, to ensure plans are effective. Currently there is potential for significant variation in quality between 16 regional councils.
"Beyond this connection to farm environment plans, Overseer needs to evolve so it is able to map and target critical source areas of nutrient loss within a farm (spatial variation) and measure the impact of day-to-day decisions on the farm (temporal variation). The next step would see Overseer and other models used to help farmers and growers identify the most suitable land uses for areas most prone to nutrient loss.
"The Commissioner’s recommendation to make the Overseer model open source could improve transparency and confidence in the outputs. This would allow components of the model that need improvement to be worked on – an uncertainty and sensitivity analysis for many of the model’s components would be helpful."
Conflict of interest: Richard was one of the reviewers of the report and also a contributor to the model’s development.
Professor Louis Schipper, University of Waikato, comments:
"Overseer is a computer model that, in part, provides information on nutrient losses from agricultural practices below the root zone that can pollute waterways. The model can be used in a regulatory fashion, where Overseer predicts nutrient losses based on farmer practices. Consequently, Overseer can create operating boundaries for farmers and there is considerable interest in the model’s accuracy and transparency, which is the focus of the PCE report. Other countries are struggling with similar issues and have regulated inputs which, as the report suggests, may not be welcomed by NZ farmers as being more constraining.
"An important conclusion is that Overseer is not open enough to external inspection and critique and that the level of uncertainty of the model outputs is not well understood. The PCE concludes that the Government needs to state whether and how it wishes Overseer to be used in a regulatory fashion. If so, the PCE then recommends a comprehensive external review of the model, that the model become open source, and that a long-term plan is developed to fund ongoing support for Overseer.
"In my opinion, the PCE’s overarching conclusions and recommendations provide a clear path forward and need to be addressed comprehensively and with urgency. It is critical that any review is carried out in a constructive fashion identifying strengths and weaknesses of Overseer and identifies a way to enhance its performance so that farmers, regulators, and the public have confidence in the use of the model to protect water bodies while also providing long-term certainty for land managers. A major challenge is that as the model continues to improve, users might end up with different predictions for the same management practices. The community will need a degree of understanding and common agreement on how these changes are then used when making decisions."
No conflicts of interest.