Why Are Neural Networks a “Black-Box”
A math perspective
Ah, the black box problem. For those of you unfamiliar with the term, a black box scenario is one in which we have a working instrument without a decent understanding of how all the internal gears and levers fit together to produce the results we’re seeing. In the case of neural networks, the gears and levers are layers of large-scale matrix multiplication. We know how matrices multiply. We can even inspect the numbers in said matrices at any given layer, but a larger “ah, that’s how that works” moment, has been frustratingly elusive.
As the theory and methods behind neural networks lie solely within the field of geometry, it’s not at all surprising the AI community is having such difficulties in explaining these models. At its core, the question of “how does my model make predictions?” is akin to answering the geometry question of “can I find a global parameterization for the surface I’m looking at.” In less mathy terms, it’s the question of whether we can find a prescribed way to get correct solutions for any input sample.
The answer to this question is no. Not only is the answer no, but in geometry, finding explicit formulas that parameterize high dimensional surfaces globally, is virtually impossible.
Why would this be? To give you an idea let’s run an experiment. The best way to think of parameterizations are with the surfaces we can see and touch in the real world. The outside of a globe is one such surface. To bring this back into the modeling perspective, let’s say we have two continuous and independent input variables and want a regression model that will predictably land these input variables on our globe such that the entire globe is covered. A great way to simulate this is with a piece of paper. If we take one of our variables to be the vertical component, and the other to be the horizontal component of our paper, then our answer lies in our ability to use our paper to cover the outside of the globe. In only a few seconds of this exercise, you’ll realize that this cannot be done without folding, ripping or stretching the paper. But wait! Remember that the height and width of the paper are our input features! So by folding the paper, you’re not really dealing with the original input variables, rather some transformed version of them.
“Ok get to the point” — Random Reader
The point is that having a model (read function) that is able to take our paper and predictably put it on a sphere in a well-defined way such that for any point on the sphere, we can clearly relate that point back to our input variables is not straightforward. Oh yes, it can be done in a formulaic way, (FYI this will always take at least two sheets of paper to do!) but this was a simple example. Not to mention, we couldn’t use our original variables, we had to “fold” them. Is it as simple to interpret the folded variables as the original variables? Most likely not. Again, this is an extremely simple example and we’ve already run into complications.
Welcome back to the real world! Now we have a hundred input variables. Our model will take these hundred variables and put them in our solution surface.
Here we hit another snag…
What is our solution surface?
In our sphere example, we already knew the shape we were trying to cover! When creating neural networks, the whole point of the model is to search out what this solution shape is. And Ultimately, what the network finds is a numerical approximation of the solution surface. The nature of approximations further complicates our understanding of this process. Approximations chase solutions through calculus rather than attempting to define the surface in a functional manner. When the process is complete, we have a great approximation, but almost no intuition of how we got there.
Stepping back, let’s take stock of our problem. We want to understand the following statement. For any example of our dependant variable, how can we trace back to the combination of independent input variables that led us there. If we knew what our solution surface elooked like, we might have a chance at deriving some formulaic way to satisfy our question. But, we don’t know what this surface looks like. Hence, we’ve settled on the idea of using neural networks to approximate this surface. The approximation uses derivatives to “feel” out small portions of the solution space to find the lowest point of our solution space in the area. The approximation yields highly accurate (or high-level whatever metric you’re using to grade the NN) results but leaves much to be desired when it comes to getting a larger formulaic sense of how each of our input variables contributes to the approximation.
Ok, so where do we go from here?
There are two general approaches of how to solve this problem.
- See if you can patch together a bunch of small areas in which you understand what the model is doing into a larger, more complete solution.
- Try to understand a simplified version of your problem and use conclusions from the simplified problem to extrapolate something from the original problem.
The premise of the first method should sound familiar if you’ve dealt with machine learning before. Packages such as LIME and shap explore the local regions around individual data points and give estimates as to what each input variable is contributing to the prediction at that data point. It should be noted that Shap goes one step further and aggregates all the local explanations into a global one! The process of how this is done is best left to the docs, but it represents the most comprehensive and digestible form of global explanations that I’ve come across to date. Again though, shap values are estimations of input variable effects, not an exact solution.
More estimations huh? Can we skip the games and use the small local regions around data points to reconstruct the solution surface in an exact form? Again, frustratingly, the answer is no (at least with the current progression of mathematical research). To explain why this can’t be done let’s consider what we’re asking. If we pick an example prediction, then we’ve arrived at where we are on the solution surface through some version of gradient descent. The process of gradient descent does tell us a lot! For instance, we know the slopes of the area around us in all directions. So, what we need to do is use this slope information, and our final resting coordinates to reconstruct the solution space around us. In other words, we have an expression of where we are globally in terms of a bunch of derivative information and we want to find the parent function whose derivative matches what we’re looking at. Anyone with some experience in math should gulp right about now. Welcome to the world of differential equations. More specifically, welcome to the world of partial differential equations (PDEs). In short, getting exact formulaic solutions to a PDE is not possible in general. The typical answer after lots of work is something like, “The solution we are looking for is part of this very large family of functions.” When looking for answers that give us an explicit understanding of variable contributions, that’s not that helpful.
I personally find the second approach to be the best answer. Answering simplified versions of problems is a hallmark of math. In fact, historically, entire fields of math have been explored in an attempt to find alternative approaches to difficult geometry questions. So what does this look like for the neural network case? I have yet to hear of an official proposed methodology but I imagine it will involve many of the tools we’ve already become accustomed to using. For example, tree-based models are among those with relatively high explainability. So what about a comparison between a trained tree model and a trained neural network? By comparing the predictions and scoring metrics, we may be able to explain those predictions which overlap using the more easily understood tree model and focus our investigation on the harder cases in which only the neural network guessed correctly. Another thought is simplifying the discovered solution space with something more easily understood. For example, replacing the solution space with the closest possible convex version of the original. Or slightly more complex, replace the solution space with a concave version that has some minimum distance for each side. While such decisions would degrade the model’s performance, it’s a huge step towards explaining why the model acts the way that it does. And remember from earlier, finding a perfect solution space may not even be possible.
Conclusion
As disappointing as all this may sound, I would have you keep in mind that we’re not dealing with an easy problem. There are mathematicians whose entire academic careers have been spent dealing with specific types of PDEs and geometric surfaces. It would be incredibly naive to think that solving the black box of neural networks is an easy task. However, this isn’t the same as giving up! We have many good tools — shap, less powerful models, math techniques, new and future technologies — that should be expanded upon to create the best understanding possible. Given the power of neural networks, I find it very likely that they will only become more pervasive in everyday life. given this, it’s extremely important that we develop reasonably good explanation tools.
References
- Hurley, Samantha. Black And White Gift Box Photo. Unknown,
burst.shopify.com, Online, https://burst.shopify.com/photos/black-and-white-gift-box
