We’ve had a little breakthrough at Sense6 recently, solving a problem which was taking our attention for a few months. Ready for a small technical rant? We’ll keep things as simple as possible and start with an example.

Suppose you’re developing your own OCR engine and have trained 3 neural networks with different specializations. The first is trained for dealing with black-and-white documents, the second for very colorful documents, the third is a jack-of-all-trades performing reasonably well whatever the situation.

Let’s further assume that the best overall performance is from the “jack of all trades” with an accuracy of 75%. However in the 25% cases it gets wrong, you’ve observed that at least one of the specialized models gets the right result. If you were able to always select the best result from your 3 models, your accuracy would skyrocket.

Programming static rules atop 3 neural networks is a recipe for disaster. Therefore you decide to create a second type of neural network whose goal is to triangulate the best result from the output of the 3 underlying models. You set everything up and excitedly run your first test-cases, waiting for accuracy bliss and … nothing happens. Still stuck at 75%. Ensue weeks of debugging 2 layers of neural networks, an activity best described as searching for a needle in a haystack... in the dark... with mittens on.

We’ll pause here to let the reader wonder about the solution for a few seconds.



There’s an important step whenever training classification models - “class imbalance” - a step we were taking into account for the first layer models but had forgotten in the second layer, mistakenly thinking “it had already been dealt with”. What is it? Let’s have another example.

Suppose you wanted to train a neural network to “turn left” or “turn right”, but 99% of your training data pertained to a “turn left” scenario. Your training data is “imbalanced” and if you trained a model without any changes, it would quickly learn that the best strategy is to simply turn left all the time: 1% error rate isn’t bad. BTW, this is typical when working on “fraud” use-cases where fraudulent banking transactions or fraudulent insurance claims are a tiny fraction of the total data volume.

To obtain the correct behavior you need to rebalance your data, i.e. transform your training data until “turn left” versus “turn right” are about 50:50. There are multiple ways to achieve this and this is a separate topic altogether; a basic approach is to randomly delete unwanted data until you reach the desired equilibrium.

So back to our original OCR example: what was happening? Well, the triangulation model had learned that one of the underlying OCR models had a better performance and that the best strategy was simply to stick to it and ignore the others. To remedy this, all you need to do is to rebalance the data, thereby removing the incentive to always default to the “jack of all trades” output (33% accuracy from a rebalanced scenario isn’t that great), and hey presto: performance improvement.

The more experienced of you could point out: "Why bother with 2 layers of neural network for OCR? Why not just create a bigger deeper neural network and let it deal automatically with jack-of-all-trades versus specialized internal configurations?”
That is a valid criticism, and that’s why OCR is a simplistic example. However there are use-cases where (we think) a second neural network layer makes sense, mostly because they have access to more data (features) than were available in a “first run” prediction. For example:
  • Data extraction from financial statements, where the relationship between financial metrics can be codified as features in a second layer (e.g. total assets = liquid assets + non-liquid assets)
  • Decision-making in self-driving cars, where the master model receives as input the predictions from the various car sensors (layer 1 neural networks). We’re less sure of this one though, having never worked on self-driving neural networks

In closing, every time we describe something technical we seem to go on a rant and write long posts. Apologies for this, we’re trying to break down complex problems into small digestible parts, unfortunately that takes up space.