Researchers from the ARC Centre of Excellence for Automated Decision-Making and Society (ADM+S), the ARC Centre of Excellence for the Digital Child and the QUT Gen AI Lab have developed an interactive voice-based chatbot aimed at helping children assess the trustworthiness of AI responses.
The tool allows users to switch between two modes: an “always agree” setting and a second setting designed to respond more cautiously. The experience is intended to highlight “AI sycophancy”, a problem in which chatbots prioritise agreement or flattery over accuracy and judgement.
The project is positioned against growing use of AI systems by children for information and personal advice. The researchers pointed to reporting on research involving Adelaide University and the Australian Federal Police which found 35 per cent of children who experienced online sexual victimisation either sought help from an AI chatbot or told no one.
Dr Henry Fraser, a QUT researcher and ADM+S Associate Investigator, said the work supports “bottom-up approaches” to AI safety and governance, alongside regulation.
“Children’s critical AI capabilities are an essential part of AI safety and governance. Regulation, like Australia’s proposed digital duty of care, is important, but it can’t fix everything. What we can do is enhance children’s preparedness for and resilience to AI risks in a way that takes their experiences and perspectives seriously,” Dr Fraser said.
The researchers said voice interfaces are increasingly being built into toys and other consumer devices used by children and families, which makes AI literacy a growing consideration for schools and households.
The interactive experience prompts users to compare the differences between chatbot responses and consider whether agreement should be interpreted as reliable advice, as well as why a system might be designed to align with a user’s views.
The research underpinning the chatbot is due to be presented at the Association of Internet Researchers (AoIR) Annual Conference in Mexico in October. The team said it is intended to contribute to international discussions about children’s experiences with AI and early AI literacy.
The chatbot is available via the GenAI Arcade website and is intended for supervised educational use. The researchers said children should ask a trusted adult before using it and should not follow chatbot advice without checking with a trusted adult.
Further details and a related research paper, “Should I hide my mum’s slippers? A design-based pilot study of children’s critical engagement with AI sycophancy using a custom AI chatbot”, are available online.
