Plaintext
ForHumanity
TAXONOMY:
AI AUDIT, ASSURANCE &
ASSESMENT
FEB 2021 // PREPARED BY RYAN CARRIER & SHEA BROWN
ACKNOWLEDGMENTS
The authors would like to thank Jacquie Hughes, Mark Potkewitz, and Adam Leon Smith for their
invaluable feedback on earlier drafts of this report.
PRINT AND ELECTRONIC DISTRIBUTION RIGHTS
© 2021 by ForHumanity. This work is licensed under a Attribution-NonCommercial-NoDerivatives 4.0
International license.
To view a copy of this license, visit: https://creativecommons.org/licenses/by-nc-nd/4.0/
Ryan Carrier, CFA
Ryan is Founder, Executive Director, and Chairman of the Board of Directors of ForHumanity.
Ryan@Forhumanity.center | https://forhumanity.center/
Shea Brown, Ph.D.
Shea is the CEO of BABL AI Inc. and a ForHumanity Fellow.
sheabrown@bablai.com | https://babl.ai/
FORHUMANITY PAGE | 01
Introduction
AMBIGUITY
LEADS TO
CONFUSION
In the growing field of AI Ethics and
Governance, the call for “Algorithm Audits” as a
means of mitigating the risk posed by the rapid
integration of AI into our everyday lives has
been getting louder. However, ambiguity over
what constitutes an AI audit has led to
confusion and uncertainty, and as the rhetoric
around recent “audits'' ramps up, the industry
is in danger of misusing terms that have well-
established meanings in the world of financial
auditing. Casual use of the term audit simply
means “to do a deep dive into fairness,
accountability, and transparency of the
algorithm”, an audit with a small ‘a’, as it were.
Audit and Assurance, with a capital “A”, is
different, and well understood to practitioners
to refer to a robust, long-established, set of
principles and approach first developed in the
financial services sector. We watched as the AI
Ethics industry has mistakenly deployed these
terms creating harmful confusion for the public
and for the owners of algorithms.
This taxonomy is critical to guiding the
What it means to
contractual relationships and the descriptions
of those services.
AUDIT
PAGE | 02 FORHUMANITY
ForHumanity has adapted an established taxonomy to distinguish between audit, assurance, and
assessments which makes clear the important roles that independence and liability must play in this
burgeoning industry. Below we lay out the features and benefits which define and explain the
differences of five different services that are routinely misused.
Our AI Audit
TAXONOMY
FORHUMANITY PAGE | 03
Definitions
CONFUSION
LEADS TO
LIABILITY
Key words and phrases in this system are well-
defined and understood:
Certified Practitioners Required? - this
distinguishes who will execute the service for
Target of Evaluation (ToE).
Certified Practitioners - means that the
contracted party must have certified experts;
employees who have submitted to a
certification process (which includes exams to
prove their knowledge base and expertise
related to the specific service being offered).
Certified practitioners are often governed by a
local or international body that provides
training, continuing education, and exam-
based certification. Frequently, certified
practitioners are required to have some
amount of practical work with certified firms.
Assessments and consulting do not require
certification but those conducting these
services are not precluded from being certified.
Professional
Standards
P A GFEO |
R H0U
4MANITY FORHUMANITY
Definitions cont.
Objective/Subjective - the work for the ToE is either:
Objective - expressing or dealing with facts or conditions as perceived without distortion by
personal feelings, prejudices, or interpretations
Subjective - based on personal opinions and feelings rather than on facts
There are often many ways to solve a problem, ( the ‘how’) and the process can be characterized as
subjective, even if the requirement is objective. For example, if the requirement calls for all
employees to be trained on a curriculum, the “all” is objective, and the fact that it has to happen is
objective. But the process could be accomplished in many ways such as:
All at once - physically in person
All online with a completion time period
A mixture of in-person classes and online learning
The ‘how’ is unspecified and therefore subjective.
Internal Audit, Audit, and Assurance examine an end result against an objective benchmark of
compliance or non-compliance. Assessment or consultation may instead propose “how” to achieve
compliance—the process and procedures deemed necessary to achieve the end result to be
measured by the auditors. Consultants and Assessors have been requested to either help the ToE
prepare for an audit or assurance or because the ToE wants to improve a process.
Independent - this word is defined by The Sarbanes-Oxley Act of 2001 and explains that an entity
(an audit provider) must receive no other remuneration from a ToE than the audit fees. ForHumanity
further stipulates in its license agreements that a licensee holder cannot be an auditor and an
Assessor/Consultant (or provide any other form of service) to the same ToE in a 12-month period.
This analysis designates Audit and Assurance as requiring Independence based on its function as a
proxy for society, and its responsibility to objectively review compliance for public production. We
deem Internal Audit to be Independent because Internal Audit is subject to these rules, as
demonstrated in the Journal of Accountancy - Internal Auditor Independence.
FORHUMANITY PAGE | 05
Definitions cont.
Known third-party, transparent, binary rules or laws - Generally Accepted Accounting
Principles (GAAP) and International Financial Reporting Standards (IFRS) are examples of third-party
accounting rules. They are transparent and the set of rules have been adopted into law by most
jurisdictions around the world. We define ‘binary’ to mean that a determination can be made
conclusively that a ToE is compliant or non-compliant with respect to the rules and/or laws.
ForHumanity has dedicated itself to creating this taxonomy and supporting this taxonomy with third-
party, binary, transparent rules. Additionally, we have laid the groundwork for translating and
transferring the ‘trust mechanisms’ established in financial accounting, to AI and Autonomous
Systems. ForHumanity is working to create an #infrastructureoftrust, described in greater detail here.
https://forhumanity.center/blog/auditing-ai-and-autonomous-systems-building-an-infrastructureoftrust
"Audit is a form of Assurance that uses Rules and Laws. Assurance is a slightly softer
version of the same service using rules, guidelines, and standards that have slightly less
objectivity and often are not codified in law. Audit is a specialized subset of Assurance.
Assurance does not necessarily mean 'audit'."
During assessments and consulting, the service provider might be building systems or procedures to
produce compliance, but the variance in how compliance is achieved does not present conflict with
rules or laws which rarely strictly codify the “how”. Many assessors and consultants abide by their
own frameworks, guidance, and toolkits which are key differentiators in their service offerings.
Known third-party, non-binary, transparent, rules, guidelines, and standards - These may be
assured, but the binary/non-binary characterization distinguishes the difference between audit and
assurance. An example of non-binary guidelines is the OECD Principles for responsible stewardship of
trustworthy AI.
Another example is the UK’s Information Commissioners Office (ICO) Children’s Code; a non-binary
set of guidelines for which ForHumanity is currently drafting binary rules to enhance audit capability.
PAGE | 06 FORHUMANITY
Definitions cont.
Service provided for? - In all audit and assurance contracts, a three-party system is required.
A three-party relationship involves a practitioner, a responsible party, and intended users. “Intended
users” often means society-at-large, the public, or at least a body that exists outside of the ToE and
not under its control. Audit and Assurance contracts require the auditor or assurer to act on behalf
of the intended user and not the ToE. Assessments and consulting contracts are two-party contracts
designed to serve the needs of the ToE exclusively.
Feedback Loop with the Company, iterative problem solving, teaching, tailoring - Assessors
and consultants focus on offering a bespoke service of problem-solving, staff education, and general
expertise to the ToE. There is no public report or third-party responsibility. Depending on the nature
of the contract, there may be an application of rules, standards. Here is a noteworthy aspect of The
Treadway Commission of Sponsoring Organizations (COSO) Internal Audit must not create
recommendations for the ToE, and must only submit its recommendations to the Board of Directors,
or officers as directed by corporate governance and their applications of COSO.
Consequences for false assertions of compliance - The consequences of false assertions for
auditors and assurers are clear - liability. Given assessors and consultants do not make assertions,
but rather highlight to the ToE that compliance is not achieved and suggest remediation - there are
no comparable liabilities or consequences.
For Assurance and Audit, both entities carry liability to intended users. For a further exposition of
this point, a study of the fate of Arthur Anderson (AA) and its audit/assurance responsibilities for
Enron and WorldCom would be instructive. AA’s liability was so great it effectively bankrupted the
firm. Responsibility and liability are crucial elements in building an infrastructure of trust for the
public that relies on the outputs of auditors and assurers.
Written report produced for the public - this is the tangible evidence of “who is the client for a
service contract”. Intent is baked in here. Does the ToE want the review of its work? Or for public
demonstration, transparency, and disclosure?. Audit and Assurance work for the intended users, for
the public-at-large.
FORHUMANITY PAGE | 07
Conclusion
CLARITY
LEADS TO
TRUST
Referenced against this clear and robust taxonomy, ForHumanity believes that most services
examining AI and autonomous systems to-date, regardless of how labeled, do not meet the exacting
standards of ‘Audit’. At best, they reach the standard of assessment or consultation. As explained
above, Audit and Assurance require higher standards:
Certified practitioners
Services being rendered FOR the public, not just shown to the public
Third-party rules
Liability for auditors and assurers
Written reports produced FOR the public
Genuine independence
We believe that rigorous adherence to the well-established principles and standards of Audit, applied
to AI and Autonomous systems will go a long way towards building a much-needed
#infrastructureoftrust around their deployment. To enable and promote progress here, we are:
Drafting audit rules on behalf of humans impacted by AI in the areas of Ethics, Bias, Privacy, Trust,
and Cybersecurity. We do this in a transparent, inclusive, crowdsourced fashion where all are
welcome
Educating and certifying practitioners
Licensing entities that will be required to maintain Independence, bear liability, and be certified by
appropriate certification authorities
Working with lawmakers to enact laws that mimic the trustworthy framework of financial
accounting where we have a proven track record of trust (50 years of experience)
All are welcome in ForHumanity, we hope to do some good for people, maybe you can help too?
PAGE | 08 FORHUMANITY
ForHumanity
ForHumanity is a 501(c)(3) tax-exempt public charity formed to examine and analyze the downside risks
associated with the ubiquitous advance of AI and automation. To this end, we engage in risk control and
mitigation and deploy the lens and filter of Ethics, Bias, Privacy, Trust, and Cybersecurity to ensure the
optimal outcome…ForHumanity.
ForHumanity is an interdisciplinary group of dedicated expert volunteers, with over 220 contributors and 30
Fellows, Its collective expertise spans the AI field, ranging from ethics to algorithmic risk and to security. Our
team is drawn from the academic, legal, policy, corporate, and public sectors of over 36 countries around the
world. Our mission is to help create an ‘infrastructure of trust’ for all autonomous systems that directly
impact humans.
ForHumanity drafts comprehensive, pragmatic and implementable audit rules and standards for
autonomous systems in every corner of the economy. Our experts collaborate with industry practitioners to
ensure these audits achieve our mission of mitigating AI risk to humans. This system of audit rules and
standards - adapted to local jurisdictional laws and regulations - is called Independent Audit of AI Systems
(IAAIS).
FORHUMANITY.CENTER | RYAN@FORHUMANITY.CENTER
ForHumanity