DOKK Library

Taxonomy: AI audit, assurance & assesment

Authors Ryan Carrier Shea Brown

License CC-BY-NC-ND-4.0



The authors would like to thank Jacquie Hughes, Mark Potkewitz, and Adam Leon Smith for their
invaluable feedback on earlier drafts of this report.


© 2021 by ForHumanity. This work is licensed under a Attribution-NonCommercial-NoDerivatives 4.0
International license.

To view a copy of this license, visit:
Ryan Carrier, CFA
Ryan is Founder, Executive Director, and Chairman of the Board of Directors of ForHumanity. |

Shea Brown, Ph.D.
Shea is the CEO of BABL AI Inc. and a ForHumanity Fellow. |
  FORHUMANITY                                              PAGE | 01


                    LEADS TO

                   In the growing field of AI Ethics and
                   Governance, the call for “Algorithm Audits” as a
                   means of mitigating the risk posed by the rapid
                   integration of AI into our everyday lives has
                   been getting louder. However, ambiguity over
                   what constitutes an AI audit has led to
                   confusion and uncertainty, and as the rhetoric
                   around recent “audits'' ramps up, the industry
                   is in danger of misusing terms that have well-
                   established meanings in the world of financial
                   auditing. Casual use of the term audit simply
                   means “to do a deep dive into fairness,
                   accountability, and transparency of the
                   algorithm”, an audit with a small ‘a’, as it were.
                   Audit and Assurance, with a capital “A”, is
                   different, and well understood to practitioners
                   to refer to a robust, long-established, set of
                   principles and approach first developed in the
                   financial services sector. We watched as the AI
                   Ethics industry has mistakenly deployed these
                   terms creating harmful confusion for the public
                   and for the owners of algorithms.

                   This taxonomy is critical to guiding the

What it means to
                   contractual relationships and the descriptions
                   of those services.

 PAGE | 02                                                                      FORHUMANITY

  ForHumanity has adapted an established taxonomy to distinguish between audit, assurance, and
 assessments which makes clear the important roles that independence and liability must play in this
 burgeoning industry. Below we lay out the features and benefits which define and explain the
 differences of five different services that are routinely misused.

Our AI Audit
   FORHUMANITY                                        PAGE | 03


                 LEADS TO

                 Key words and phrases in this system are well-
                 defined and understood:

                 Certified Practitioners Required? - this
                 distinguishes who will execute the service for
                 Target of Evaluation (ToE).

                 Certified Practitioners - means that the
                 contracted party must have certified experts;
                 employees who have submitted to a
                 certification process (which includes exams to
                 prove their knowledge base and expertise
                 related to the specific service being offered).
                 Certified practitioners are often governed by a
                 local or international body that provides
                 training, continuing education, and exam-
                 based certification. Frequently, certified
                 practitioners are required to have some
                 amount of practical work with certified firms.

                 Assessments and consulting do not require
                 certification but those conducting these
                 services are not precluded from being certified.

         R H0U
             4MANITY                                                                FORHUMANITY

Definitions cont.

Objective/Subjective - the work for the ToE is either:

   Objective - expressing or dealing with facts or conditions as perceived without distortion by
   personal feelings, prejudices, or interpretations

   Subjective - based on personal opinions and feelings rather than on facts

There are often many ways to solve a problem, ( the ‘how’) and the process can be characterized as
subjective, even if the requirement is objective. For example, if the requirement calls for all
employees to be trained on a curriculum, the “all” is objective, and the fact that it has to happen is
objective. But the process could be accomplished in many ways such as:

   All at once - physically in person
   All online with a completion time period
   A mixture of in-person classes and online learning

The ‘how’ is unspecified and therefore subjective.

Internal Audit, Audit, and Assurance examine an end result against an objective benchmark of
compliance or non-compliance. Assessment or consultation may instead propose “how” to achieve
compliance—the process and procedures deemed necessary to achieve the end result to be
measured by the auditors. Consultants and Assessors have been requested to either help the ToE
prepare for an audit or assurance or because the ToE wants to improve a process.

Independent - this word is defined by The Sarbanes-Oxley Act of 2001 and explains that an entity
(an audit provider) must receive no other remuneration from a ToE than the audit fees. ForHumanity
further stipulates in its license agreements that a licensee holder cannot be an auditor and an
Assessor/Consultant (or provide any other form of service) to the same ToE in a 12-month period.

This analysis designates Audit and Assurance as requiring Independence based on its function as a
proxy for society, and its responsibility to objectively review compliance for public production. We
deem Internal Audit to be Independent because Internal Audit is subject to these rules, as
demonstrated in the Journal of Accountancy - Internal Auditor Independence.
    FORHUMANITY                                                                                  PAGE | 05

Definitions cont.

Known third-party, transparent, binary rules or laws - Generally Accepted Accounting
Principles (GAAP) and International Financial Reporting Standards (IFRS) are examples of third-party
accounting rules. They are transparent and the set of rules have been adopted into law by most
jurisdictions around the world. We define ‘binary’ to mean that a determination can be made
conclusively that a ToE is compliant or non-compliant with respect to the rules and/or laws.

ForHumanity has dedicated itself to creating this taxonomy and supporting this taxonomy with third-
party, binary, transparent rules. Additionally, we have laid the groundwork for translating and
transferring the ‘trust mechanisms’ established in financial accounting, to AI and Autonomous
Systems. ForHumanity is working to create an #infrastructureoftrust, described in greater detail here.

"Audit is a form of Assurance that uses Rules and Laws. Assurance is a slightly softer
version of the same service using rules, guidelines, and standards that have slightly less
objectivity and often are not codified in law. Audit is a specialized subset of Assurance.
Assurance does not necessarily mean 'audit'."

During assessments and consulting, the service provider might be building systems or procedures to
produce compliance, but the variance in how compliance is achieved does not present conflict with
rules or laws which rarely strictly codify the “how”. Many assessors and consultants abide by their
own frameworks, guidance, and toolkits which are key differentiators in their service offerings.

Known third-party, non-binary, transparent, rules, guidelines, and standards - These may be
assured, but the binary/non-binary characterization distinguishes the difference between audit and
assurance. An example of non-binary guidelines is the OECD Principles for responsible stewardship of
trustworthy AI.

Another example is the UK’s Information Commissioners Office (ICO) Children’s Code; a non-binary
set of guidelines for which ForHumanity is currently drafting binary rules to enhance audit capability.
PAGE | 06                                                                            FORHUMANITY

Definitions cont.

Service provided for? - In all audit and assurance contracts, a three-party system is required.

A three-party relationship involves a practitioner, a responsible party, and intended users. “Intended
users” often means society-at-large, the public, or at least a body that exists outside of the ToE and
not under its control. Audit and Assurance contracts require the auditor or assurer to act on behalf
of the intended user and not the ToE. Assessments and consulting contracts are two-party contracts
designed to serve the needs of the ToE exclusively.

Feedback Loop with the Company, iterative problem solving, teaching, tailoring - Assessors
and consultants focus on offering a bespoke service of problem-solving, staff education, and general
expertise to the ToE. There is no public report or third-party responsibility. Depending on the nature
of the contract, there may be an application of rules, standards. Here is a noteworthy aspect of The
Treadway Commission of Sponsoring Organizations (COSO) Internal Audit must not create
recommendations for the ToE, and must only submit its recommendations to the Board of Directors,
or officers as directed by corporate governance and their applications of COSO.

Consequences for false assertions of compliance - The consequences of false assertions for
auditors and assurers are clear - liability. Given assessors and consultants do not make assertions,
but rather highlight to the ToE that compliance is not achieved and suggest remediation - there are
no comparable liabilities or consequences.

For Assurance and Audit, both entities carry liability to intended users. For a further exposition of
this point, a study of the fate of Arthur Anderson (AA) and its audit/assurance responsibilities for
Enron and WorldCom would be instructive. AA’s liability was so great it effectively bankrupted the
firm. Responsibility and liability are crucial elements in building an infrastructure of trust for the
public that relies on the outputs of auditors and assurers.

Written report produced for the public - this is the tangible evidence of “who is the client for a
service contract”. Intent is baked in here. Does the ToE want the review of its work? Or for public
demonstration, transparency, and disclosure?. Audit and Assurance work for the intended users, for
the public-at-large.
    FORHUMANITY                                                                             PAGE | 07



Referenced against this clear and robust taxonomy, ForHumanity believes that most services
examining AI and autonomous systems to-date, regardless of how labeled, do not meet the exacting
standards of ‘Audit’. At best, they reach the standard of assessment or consultation. As explained
above, Audit and Assurance require higher standards:

   Certified practitioners
   Services being rendered FOR the public, not just shown to the public
   Third-party rules
   Liability for auditors and assurers
   Written reports produced FOR the public
   Genuine independence

We believe that rigorous adherence to the well-established principles and standards of Audit, applied
to AI and Autonomous systems will go a long way towards building a much-needed
#infrastructureoftrust around their deployment. To enable and promote progress here, we are:

   Drafting audit rules on behalf of humans impacted by AI in the areas of Ethics, Bias, Privacy, Trust,
   and Cybersecurity. We do this in a transparent, inclusive, crowdsourced fashion where all are
   Educating and certifying practitioners
   Licensing entities that will be required to maintain Independence, bear liability, and be certified by
   appropriate certification authorities
   Working with lawmakers to enact laws that mimic the trustworthy framework of financial
   accounting where we have a proven track record of trust (50 years of experience)

All are welcome in ForHumanity, we hope to do some good for people, maybe you can help too?
PAGE | 08                                                                                  FORHUMANITY

ForHumanity is a 501(c)(3) tax-exempt public charity formed to examine and analyze the downside risks
associated with the ubiquitous advance of AI and automation. To this end, we engage in risk control and
mitigation and deploy the lens and filter of Ethics, Bias, Privacy, Trust, and Cybersecurity to ensure the
optimal outcome…ForHumanity.

ForHumanity is an interdisciplinary group of dedicated expert volunteers, with over 220 contributors and 30
Fellows, Its collective expertise spans the AI field, ranging from ethics to algorithmic risk and to security. Our
team is drawn from the academic, legal, policy, corporate, and public sectors of over 36 countries around the
world. Our mission is to help create an ‘infrastructure of trust’ for all autonomous systems that directly
impact humans.

ForHumanity drafts comprehensive, pragmatic and implementable audit rules and standards for
autonomous systems in every corner of the economy. Our experts collaborate with industry practitioners to
ensure these audits achieve our mission of mitigating AI risk to humans. This system of audit rules and
standards - adapted to local jurisdictional laws and regulations - is called Independent Audit of AI Systems