Audits of artificial intelligence (AI) systems

The most discussed topic regarding AI tools is the so called blackbox problem – the fact that humans are often not able to understand how AI generates results. The systems are characterised by great individuality, and therefore there is the risk that automatically performed tasks will be difficult to comprehend and to control. So the question of how to assure the quality of highly complex AI systems is of great importance.

To address this concern the Institut der Wirtschaftsprüfer (Institute of Public Auditors) in Germany has issued a draft auditing standard dealing with auditing AI systems (IDW EPS 861). There is no commonly used definition of AI due to the diverse and constantly changing possibilities of application, so the Institut der Wirtschaftsprüfer has defined AI in narrow limits for the purpose of IDW EPS 861. In concrete terms this means that methods of classical statistics are excluded, as well as programmed routines like, for example, robotic process automation.

In general, “AI” is an umbrella term for different computer-based activities that imitate human thinking and mimic human thought and actions to make decisions and solve problems. It is often used to handle mass data and automate decision making. AI can be further divided in the two areas: machine learning and deep learning. Decisions made with the help of machine learning are based on previously learned techniques. Machine learning distinguishes different approaches to learning such as supervised learning and unsupervised learning.

The purpose of supervised learning is to recognise patterns based on labelled training data. In contrast, unsupervised learning tries to find patterns in data sets without a prior labelled- data learning process. A typical example for using machine learning is to categorise contracts and to put them into the right field of law in a contract management system.

Deep learning on the other hand uses artificial, deep neural networks which autonomously can make complex decisions. Examples of technologies that are based on deep learning include Natural Language Processing like “Alexa”, or Computer Vision that can be used for image analysis so that a complete invoice is recognised and automatically posted.

As AI becomes increasingly important, the above-mentioned auditing standard contains the criteria for performing an audit of an AI system distinct from the auditing of the annual financial statements.

The basis of an audit of an AI system is the detailed description provided by the management of the AI system used in the company. The managers are responsible for the implementation and supervision of the system. When checking the design and "up-to-date”-ness of the AI system description, particular attention should be paid to checking the completeness of the company’s description. Only if the contents are described clearly, unambiguously, and in a way that can be understood by a knowledgeable third party, can it be guaranteed that AI delivers appropriate and effective results.

As you can see, the audit of an AI system is a very special task and will be subject to a lot of future adjustments and discussions.

---

Photo: phonlamaiphoto - stock.adobe.com