Key Considerations for Software Updates for Embedded Linux and IoT

The Mirai botnet attack that enslaved poorly secured connected embedded devices is yet another tangible example of the importance of security before bringing your embedded devices online. A new strain of Mirai has caused network outages to about a million Deutsche Telekom customers due to poorly secured routers. Many of these embedded devices run a variant of embedded Linux; typically, the distribution size is around 16MB today.

Unfortunately, the Linux kernel, although very widely used, is far from immune to critical security vulnerabilities as well. In fact, in a presentation at Linux Security Summit 2016, Kees Cook highlighted two examples of critical security vulnerabilities in the Linux kernel: one being present in kernel versions from 2.6.1 all the way to 3.15, the other from 3.4 to 3.14. He also showed that a myriad of high severity vulnerabilities are continuously being found and addressed—more than 30 in his data set.

Figure 1. Linux Kernel Fix Timing

Although the processes and practices of development teams clearly have a critical impact on the (in)security of software in embedded products, there is a clear correlation between the size of the software project's code base and the number of bugs and vulnerabilities as well. Steve McConnell in Code Complete states there are 1–25 bugs and vulnerabilities per 1,000 lines of code, where the variable is determined by the practices of the team. Military-certified products typically end up in the lower end due to more rigid security practices and quality testing, while consumer electronics are unfortunately in the higher end of the scale due to the greater focus on features and time to market.

Seasoned software developers always seek to reduce the size of the code base through refactoring and the reuse of functionality in libraries, but with the never-ending demand for more features and intelligence in every product, it is clear that the amount of software in embedded devices will only grow. This also necessarily implies that there will be more bugs and vulnerabilities as well.

From this, it should be clear that having a way to deploy bug and security fixes, as well as new features, remotely is an obvious requirement for embedded devices with some level of intelligence, especially if they are network-connected.

This article goes further into specific requirements and solution designs on deploying software updates to embedded devices, in particular, embedded Linux.

The State of Software Updates in the Embedded Industry

As part of the work on Mender, we are conducting interviews with many software engineers and teams in order to better understand how software updates are being done in embedded products today. In one series of these interviews, we spoke with 30 embedded software engineers and the main takeaways are described below.

In the first question, we simply asked if software updates are being deployed to their embedded products today and, if so, which tools were used (Figure 2).

Figure 2. Survey Response to Whether Software Updates Are Being Deployed to Embedded Products, and If So, Which Tools Were Used

45.5% of the respondents said that updates were never being deployed to their products. Their only way to get new software into customers' hands was to manufacture hardware with the new software and ship the hardware to the customers.

Roughly the other half, 54.5%, said that they did have a way to update their embedded products, but the method was built in-house. This also includes local updates, where a technician would need to go to a device physically and update the software from external media, such as a USB stick. Yet another category was devices enabled for remote updates, but where you could update only one at the time, essentially precluding any mass updates. Finally, some had the capability to deploy mass updates to all devices in a highly automated fashion.

One of the key findings here was that virtually nobody reused a third-party software updater—they all re-invented the wheel!

Second, we asked what the preferred method of deploying software updates is (Figure 3). You can broadly classify embedded updaters into image- or package-based. Image-based updaters will work on the block level and can replace an entire partition or storage device. Package-based updaters work on the file level, and some of them, like RPM/YUM, are well known in Linux desktop and server environments as well.

Figure 3. Survey Results on Software Update Methods

______________________

Eystein Stenberg has more than seven years of experience in security and systems management software and has spoken at various conferences. You can reach him at eystein@mender.io.