Abstract:

As artificial intelligence becomes embedded in critical domains—from healthcare to cybersecurity—ensuring that AI-enabled systems (a.k.a. AI-Software) meet responsible AI requirements is an urgent challenge. Traditional debugging techniques, which focus on identifying defects in single executions, are fundamentally limited when applied to modern AI-Software, whose behavior is often opaque and data dependent.

This talk introduces metamorphic debugging, a new approach that shifts debugging from analyzing individual executions to reasoning about relationships across multiple executions. By specifying how outputs should change in response to controlled input transformations, metamorphic debugging enables systematic detection, explanation, and mitigation of ethical, legal, and safety issues in AI-Software.

We illustrate this approach through a real-world agentic system developed for legal-critical tax software, where an orchestrating agent integrates natural language processing, retrieval-augmented generation, and symbolic tax computation. While highly expressive, such systems are difficult to verify and trust. Metamorphic debugging addresses this limitation through enhancing metamorphic testing across three dimensions: (1) causality to specify cause-effect relationships, (2) information theory to scale the analysis in the presence of higher-order relationships, and (3) extreme value theory to provide statistical guarantees on the absence of rare but critical failures. 

These methods have led to the discovery of performance bugs in widely used machine learning libraries, fairness issues in training pipelines, and privacy vulnerabilities in critical software systems. The work has also informed collaborations with the U.S. IRS and educational initiatives in AI ethics for K-12 educators.

Bio: Saeid Tizpaz-Niari is an Assistant Professor of Computer Science at the University of Illinois Chicago. He received his PhD in Computer Engineering from University of Colorado Boulder in 2020. His research interests are at the intersection of SE, AI, and cybersecurity. His research group builds debugging tools and techniques for AI-enabled software systems. Tizpaz-Niari is an NSF CAREER awardee of 2025 and has also received two other NSF awards from Secure and Trustworthy Cyberspace (SaTC) and Designing Accountable Software Systems (DASS) programs. During his PhD, he received a Gold Research Award from the ECEE department at CU Boulder, and the second prize for his submission to the First Microsoft Open-Source Challenge.