AI Tools Can Interpret NZ Privacy Policies With Up To 85% Accuracy

Would you read an online store’s privacy policy before handing over your address or credit card details?

If you answered no, you’re not alone.

While the Government’s Consumer Protection website recommends it, in 2020 the Herald reported that only 2 percent of New Zealanders always read privacy policies on the websites they visit.

But new research from Auckland University of Technology (AUT) has found generative AI tools like Chat GPT, Claude AI and Gemini may be able to help.

Researchers Sumedha Mukherjee and Professor Jairo Gutierrez from AUT’s School of Engineering, Computer and Mathematical Sciences, found the tools can read and interpret privacy statements with up to 85 percent accuracy.

By comparing human interpretations of statements from 15 high traffic New Zealand websites with interpretations by the AI chatbots, Mukherjee and Gutierrez found AI agreed with humans most of the time.

“Across websites from the finance, education, government, entertainment and healthcare sectors, all three tools were on average about 75 percent in agreement with the human findings,” said Mukherjee, who completed her Masters in Cybersecurity and Digital Forensics at AUT.

“Anthropic’s Claude model was the best performer, with more than 85 percent accuracy overall.

“This means there’s perceived to be a general low chance for error if a user asked one of these chatbots to assess how a privacy statement on a website aligns with New Zealand’s Privacy Act.”

By law, New Zealand websites must have a privacy statement that tells users what personal information is collected, how long it’s kept, why it’s used, and who it’s shared with. The Privacy Act 2020 governs how this information is used.

Though the accuracy of the results found in this study are promising, Professor Gutierrez warns that users must remain cautious when using these tools.

“Gen AI tools are improving, but reports of biases and ‘hallucinations’ are a daily occurrence,” he says.

“A probing question on a particular privacy issue may provide unreliable results: for example, wrong information about the fines associated with violations of the Privacy Act.”

Professor Gutierrez also warns that using a personal account with a generative AI tool has personal privacy implications.

These include how your inputs may be stored, reused, or used for training; that your data may be transferred overseas beyond New Zealand’s privacy framework; and that you may have limited control over access and correction rights.

The research paper, Harnessing Generative AI for Enhancing User Privacy Awareness: An Examination of Privacy Statements and GenAI Systems, was presented at the Americas Conference on Information Systems in Montreal, Canada.

© Scoop Media