Black Box AI: Can We Trust What We Don't Understand?

Responsible AI Australia hosted Black Box AI: Can We Trust What We Don't Understand? at The Precinct on 24 September 2025. The evening brought together developers, researchers, founders and policy thinkers for an in person conversation about what trust in opaque AI systems actually requires.

A social experiment to open the night

Nathan Chung opened the evening with a short social experiment. Each attendee was asked to provide a brief response to an AI related question, alongside a response generated by ChatGPT. The room then voted on which answer came from the human. Most votes correctly identified the model generated response within seconds.

The exercise sparked a longer discussion about how language patterns reveal cognition, and where the gap between fluent text and human reasoning still sits. It also set the tone for the rest of the evening, which focused on the practical limits of how much we can trust AI output that looks correct on the surface.

John Crook on bias mitigation in AI models

John Crook followed with a walkthrough of the influential paper Why Should I Trust You? Explaining the Predictions of Any Classifier (Ribeiro, Singh and Guestrin, 2016). His presentation focused on bias mitigation in AI models and the techniques practitioners can deploy today.

The methods he highlighted included:

• Human in the loop review at critical decision points
• Curated training data with explicit attention to representation
• Refined system prompts that reduce drift in production
• Model ensembling to surface disagreement and uncertainty
• Audit tooling that records decisions and inputs over time
• Explainability methods such as LIME and SHAP for individual predictions

The talk made the case that opacity is a hard problem, not an unsolvable one, and that responsible teams should treat it as engineering work rather than a topic for marketing copy.

Syed Mosawi on Black Box AI in practice

Founder Syed Mosawi closed the evening with a presentation on Black Box AI in practice. He explored the different forms opacity takes in real systems, including model level opacity in deep neural networks, organisational secrecy around training data and model architecture, and contractual opacity that appears when systems are accessed through third party APIs.

For each form he examined what could realistically be remedied through technical or governance means, and what was structural. The session also covered the philosophical question that gives the topic its weight: what does it mean to trust a decision when the reasoning behind it cannot be inspected? That question is becoming a live concern for boards, regulators and end users alike.

A night that continued in the room

The evening closed with continued conversations between attendees on what responsible AI looks like in their own organisations. Many of those conversations have continued in the months since, and several attendees have since engaged with Responsible AI Australia on certification and governance work.