The Importance of Transparency in Machine Learning Models

Digital systems are only useful if they can be trusted to do their job

Digital systems are only useful if they can be trusted to do their job. With traditional deterministic systems, we could derive a degree of certainty that a system was working correctly through processes such as unit tests, end-user testing, code reviews, and design documentation.

AI is inherently non-deterministic in nature, especially with approaches like machine learning which continually evolve, update, and retrain, over their life cycle. As AI is increasingly adopted to make more important decisions than ever before, users need to be able to establish the same level of trust in AI as their deterministic predecessors once provided. Most notably, today’s AI systems need to ensure fairness and reduce or eliminate bias, as well as provide accountability (e.g., a way to audit/review how the system derives results) and justification for why a system outputs specific results.

Users need to be able to establish the same level of trust in AI as their deterministic predecessors once provided

Deriving the same degree of trust in AI systems as we had with traditional deterministic systems, has led to a greater need to remove the “black box” and increase transparency of how the system was built and ultimately how it works.

Increasing transparency, can benefit stakeholders in different ways:

• End users: can better understand why certain results are being generated.
• Developers: can more easily debug, tune, and optimize machine learning models.
• Project managers: can better comprehend how an otherwise technical project works

To formalize the problem of establishing trust, organizations are starting to put together different paradigms, tools, and frameworks which approach the problem in different ways. For example, IBM’s Four Pillars paradigm suggests that AI systems must fulfill the goals of being fair, robust, explainable, and auditable, while Google has developed its Model Card recommendations on how to best document data and models. Google has also provided their Explainable AI toolset that allows you to better understand machine learning models.

PerceptiLabs’ visual modeling tool was built from the ground up with transparency in mind, to provide transparency to users in three key ways:

1. Visualizations of Components and Variables.
2. Fully Transparent and Editable Code.
3. Visual Model Architecture.

We’ll show you how each of these three areas helps increase transparency.

Visualizations of Components and Variables

PerceptiLabs’ visual modeling tool, with its drag and drop interface, means that users control every component that goes into a model and every connection between those components. Most notably, PerceptiLabs provides previews for every variable in each component so you can see and control exactly how they are configured. Likewise, PerceptiLabs provides previews of the output from each component (e.g., a bitmap generated from image data).

This is particularly useful for developers who need to debug and understand machine learning models, but also benefits other users like data scientists who need to understand how the model is processing their data. So, not only can you see the output, but you can also run separate datasets through the same model and compare the results. For example, a data scientist can run the model on one dataset where they know the results/distribution are fair, and then run the model on other test datasets to compare the results. Similarly, you can tweak both code and parameters during this iterative process to further enhance testing.

Fully Transparent — Editable and Auditable Code

Behind the scenes in PerceptiLabs’ modeling tool, each component in the model results in the generation of TensorFlow code (with other frameworks soon to be supported) providing full transparency to programmers about the code that is being run. Developers can not only view the code for each component but also directly modify and customize that code. This makes the code auditable, so you can see what your code is doing at each phase of the model, and also see where/when the code is running.

Dividing the model up in this way, also helps users build a case to state their assumptions at various points and justify (or audit) why the system, or a given component in the architecture, is producing certain results. Moreover, this allows you to assess the robustness of your data models.

Visual Model Architecture

With the data model shown visually in PerceptiLabs’ modeling tool, users can immediately see both the objects in the model and the connections between them, which provides instant visualization as to how the model works. You can view the properties and previews as they trace forwards and backwards to see how each layer of the model is deriving output, which further contributes to auditability.

Safety and security are becoming big concerns, as the data that is fed into a machine learning system may have been tampered with to produce biases. PerceptiLabs’ includes a number of statistics on prediction and accuracy that can be used to better assess how the machine learning model is performing on different datasets. Again, this can be used during iterations of tweaking the model, retraining it, and comparing the results, while looking for biases both in the trained data and during inference.

Conclusion

Achieving transparency with AI systems is critical as their adoption grows, as these inherently non-deterministic systems must be trustworthy when used for inference in the real world.

PerceptiLabs, and its visual approach, was built from the ground up with transparency in mind and provides a good starting point on which to base checks and balances in order to establish trust with your model and advance explainability in machine learning.

Originally published at https://blog.perceptilabs.com.