Over the past few months, I have shown you why machine learning at the network edge is so essential. In this final article, we look back at why machine learning is needed and look at some of the real-world cases where it has already been transformative.
Why is Machine Learning needed at the network edge?
Machine learning is the core technology for smart homes, autonomous vehicles, and intelligent automation. All these applications take place at (or beyond) the network edge. However, most ML models need the sort of computing power you can only get in the cloud.
This is especially true when it comes to training new models. But what happens when you can’t run your model in a data centre? Then you need to rely on running ML models on edge devices.
There are several reasons why you can’t run a model in the cloud. The three that matter the most are explained here.
Real-time performance
Many edge ML applications rely on working in real-time. That is, they need the ML model to give a result almost instantaneously. Two key examples of this are:
- Self-driving cars. Clearly, self-driving cars need to react as fast as possible to the world around them. They will rely on a range of different sensors to gather data about current driving conditions. These include cameras, LiDAR, radar, and time-of-flight cameras. These sensors produce a range of inputs that need to be analysed and combined in real-time.
- Safety systems in factories. Automation has allowed factories to become far safer in recent years. However, there are still potential dangers, not least because production lines are often monitored by just one person. Increasingly, factories rely on sensors to understand what is happening in real-time. These perform various tasks, such as detecting unexpected objects and identifying broken tools.
Security and privacy
Some applications of ML handle sensitive data. For instance, medical data related to AI for patient diagnosis. This data is viewed as especially sensitive in many countries. For instance, in the EU it receives extra protections under GDPR. Sending this data across the network to a remote data centre may be undesirable.
Likewise, some security applications may require local processing. For instance, smart door entry systems at government sensitive sites. Here, it is probably best not to send photos of every visitor across the network to verify them against a central database.
Network connectivity
There are many locations where edge ML is the only option due to the lack of network connectivity. This can be due to two reasons. Either, there simply is no network due to the remote location. Or, local network policies forbid connections to the Internet.
The former happens regularly in heavy industry, mining, and agriculture. Take modern farming for instance. Nowadays, agriculture is driven by big data and edge ML is used to boost yield and cut the environmental impact.
Combine harvesters monitor yield in fields to the nearest square metre. Fertiliser sprayers can adapt the amount of fertiliser they deliver based on monitoring the crop using cameras.
How do we enable edge Machine Learning?
Machine learning at the edge has become feasible thanks to two developments. Firstly, low-cost MCUs now have sufficient processing power to power machine learning models. Secondly, a number of frameworks have been developed specifically for edge ML.
What hardware is available?
We already saw the sort of affordable hardware that can power edge ML. Mostly, this will be some form of MCU with an embedded ARM Cortex core. The following table shows some of the more popular options.
| Board | Processor | Core | Speed |
| NXP i.MX RT1050 EVK | MIMXRT1052DVL6A | Cortex-M7 | Up to 600MHz |
| Microchip SAM E54 Xplained Pro | ATSAME54P20A | Cortex-M4 | Up to 120MHz |
| Infineon XMC4700 Relax Kit | XMC4700-F144 | Cortex-M4 | 144MHz |
| SiLabs SLSTK3701A starter kit | EFM32 Giant Gecko 11 | Cortex-M4 | 72MHz |
| Arduino Nano 33 BLE Sense | nRF52840 | Cortex-M4 | 64MHz |
In future, we will see more specialised system-on-a-chip or SoC solutions. SoCs deliver the fast processing, graphics, sensing, network connectivity, and other features we expect in a modern smartphone. The latest, such as Apple’s A13, now include specialised hardware specifically for running neural network machine learning models.
How about software support?
Most machine learning models are typically created using a framework or set of libraries. Two of the most popular are:
- TensorFlow. This was developed by Google but released under opensource license. This framework (officially a platform) makes it easy to develop ML models anywhere.
- PyTorch. This is Facebook’s extension to the Torch ML library. PyTorch simplifies the creation of ML models in Python. It leverages NumPy to offer high performance and accuracy.
Both of these frameworks offer versions that are optimised for running at the network edge. TensorFlow Lite allows you to export any TensorFlow model and run it on a variety of edge devices. PyTorch Mobile allows PyTorch models to be run on SoC-based systems.
In a previous article, I showed you how to convert a TensorFLow model to TensorFlow Lite. As I explained then, there are some limitations that mean you can’t simply convert every existing model.
However, each new release of TensorFlow Lite reduces these limitations further. Currently, the greatest limitation relates to data types. While tf.float32, tf.int8 and tf.uint8 are always supported, some operations don’t support tf.float16 or tf.string.
Real-world examples of edge ML
Over this series, I showed you several practical examples of edge ML, such as gesture recognition, computer vision, and voice control. All these examples relied on existing ML models and were relatively simple. But what about more realistic applications of edge ML?
Real-time facial recognition
Police forces and security services need to be able to spot trouble-makers at demonstrations or sporting events. For years, they have employed so-called super-recognisers—people with an almost superhuman ability to recognise faces. These people look through thousands of police mugshots and learn to identify known hooligans or extremists.
Then, at the event, they scan the crowd looking for people they recognise. Unfortunately, this super-recogniser skill is rare. If you want to find out if you have this skill, you can check out the Greenwich University Super-Recogniser test.
Fortunately, facial recognition systems are increasingly able to perform as well as humans (albeit, with some teething issues relating to bias). Usually, this involves uploading footage for analysis in the cloud. However, in a real-world situation, the police need to be able to recognise people in real-time.
Often, there will only be limited network connectivity too. Thus, they need to rely on edge-ML to achieve this. Currently, most of these solutions rely on CPUs, rather than MCUs. One example is Advantech’s Video AI Facial Recognition Edge.
Smart headphones and speakers
Modern headphones and Bluetooth speakers are no longer just a way to listen to music on the go. Nowadays, they routinely include smart features like noice cancellation and digital assistants. But how is that possible in a device that is sometimes small enough to fit inside your ear? The answer is by using highly-specialised SoCs.
One good example is NXP’s MCU-based solutions for Alexa Voice. These are specifically designed for integration into smart devices all over the home. These are based on the i.MX RT1060 high-performance MCU with an embedded Cortex-M7 ARM core. They come with integrated Bluetooth and WiFi for maximum connectivity.
Currently, they are able to do always-on voice recognition, listening for and reacting to “wake words”. In future, these will likely become powerful enough to run complete voice control ML models.
Self-driving cars
I already mentioned that self-driving cars rely on edge-ML to allow them to operate safely. SAE International identifies 5 levels of automation for self-driving vehicles. These are show in the table below.
Most modern cars achieve some level of automation. For instance, lane centering with adaptive cruise control is a common feature on many cars. This system will ensure the car remains between the lines on the highway and will prevent it getting too close to the vehicle in front. Systems like this rely heavily on embedded MCUs running simple edge ML models.
As you move up to SAE level 4 or 5, you need to use far more complex models. At this level, you start to rely on highly specialised single-board computers, such as the NVIDIA DRIVE AGX Pegasus.
This is described as an embedded supercomputing platform, and is capable of 320 trillion operations per second. For comparison, that is approximately the same as the latest generation of AMD Ryzen desktop CPU. However, the NVIDIA system is specifically optimised for machine learning with two embedded GPUs.
Before long, we will see systems that effectively integrate the GPU and processing cores into a single SoC.
What’s next for edge ML?
In many ways, we are still in the infancy of edge ML. But there is no doubt in my mind that it is going to grow and grow as a field. So, what applications might we see in the future? Last time, I looked at how the increasing performance of MCUs will allow new forms of ML at the edge. Specifically, I identified three areas:
- Unsupervised learning. This is where a computer builds its own model from data that has not been labelled or processed in advance. Often, it is used in combination with supervised learning through a process known as boosting.
- Deep learning. Currently, this is the nearest computers can come to true human-like intelligence. Deep learning involves the use of enormous deep neural networks to tackle problems. Google DeepMind is probably the world-leader here. It has demonstrated some remarkable feats, including solving the protein folding problem.
- Continuous learning. Supervised learning models often become outdated as soon as they are created. For instance, stock predictions for fashion retailers don’t just depend on the current weather and season. They also change rapidly as fashions come and go. Continuous learning solves this by constantly updating the model based on the latest data.
Future edge Machine Learning applications
The above advances will open up some important new applications for edge ML. Here are my three predictions for the near future.
True intelligent sensors
The Internet of Things (IoT) has been much-hyped in recent years. But so far, it has failed to deliver its full potential. This is because it relies on simple sensors. These sensors are then connected together and feed their data to a central system for processing. This creates two drawbacks.
Firstly, the systems only work when three is really good connectivity. In fact, IoT is one of the key drivers behind the rollout of 5G technology. Secondly, this need for constant connectivity greatly increases the power drawn.
In the near future, we will see intelligent sensors that use an on-board MCU to do much of the processing. They will only need to connect to the central system if they identify a problem. Potentially, they may even be able to respond directly, for instance cutting power to a system that is failing.
Autonomous digital assistants
Digital assistants like Alexa, Siri, and Google Assistant are all round us. However, all these systems rely on being permanently connected to the cloud. This is needed because accurate speech recognition and voice control still need quite significant processing power. This is a definite limitation, as can be seen in the decidedly unreliable voice control systems found in cars.
Soon MCUs will be able to run deep neural networks. This will allow them to run truly autonomous voice assistants that don’t rely on network connectivity. Finally, we will have car voice control that doesn’t rely on us speaking slowly, clearly, and with careful annunciation!
Intelligent security systems
My third prediction relates to a key problem in the modern world—security. Sadly, we live in a world where there are constant terror threats. We are always asked to watch out for unattended bags or suspicious behaviour. But what if we could rely on artificial intelligence to be our eyes and ears?
Well, if you search the academic literature, you can find plenty of published articles looking at using deep learning for this. For instance, i-SAFE, which uses fuzzy decision making and deep learning to spot suspicious activity in a video feed. With the increased capability of edge-ML, this feature will soon be embedded in CCTV systems and even home security systems.
Conclusions
Over the past months, I have talked a lot about edge-Machine Learning. You have seen how it is enabling artificial intelligence applications right at the edge of the network. I gave you hands-on examples of voice control, face recognition, and gesture detection. Hopefully, these have shown you something of the requirements and capabilities of edge-ML.
As you have seen, this is a rapidly evolving field with more and more applications. We have touched on a few of these, and I am sure you will hear lots about them in future. I hope you have been inspired to try out some of the practical examples. Most importantly, I hope you have enjoyed learning about a truly cutting-edge area of machine learning.
