Opinion: Embedded Linux Tool Market Won't Hold WaterOpinion: Embedded Linux Tool Market Won't Hold Water

A number of suppliers of embedded software development tools have recently announced that they are "supporting" or even "embracing" embedded Linux. Having been there and done that, I have some news for these embedded Linux wannabes: there is no sustainable embedded Linux tools market. In the end, embedded software developers who value commercial tools enough to pay for them will eschew Linux.

This conclusion may seem counter-intuitive. Many of the key features and benefits that drive the traditional embedded software development tools market would appear to have even more value to developers using embedded Linux. This seeming synergy is particularly acute when it comes to optimization. Because Linux is designed as a general-purpose operating system, it requires more memory and has lower performance than one developed specifically for embedded use. The optimization abilities of commercial tools could mediate the liabilities.

Support would appear to be another value of commercial tools. Embedded Linux users or prospects that had previously depended upon homegrown solutions are generally confident that they have the in-house expertise to support an operating system. However, they do not have knowledge or experience when it comes to developing or supporting tools.

Those Linux users that are replacing commercial solutions are already accustomed to receiving support for both the tools and operating system. While they can expect to get operating system support from either a Linux distributor or the Linux community at large, they still need a solution for the tools.

The fact that Linux relies by default on the GNU compiler and debugger likely bolsters any tools vendor's confidence in the potential market. Nearly everyone who has purchased premium tools has done so in lieu of using a free GNU alternative.

The embedded Linux market is that it is highly fragmented. While the use of Linux as a common brand name suggests a high degree of homogeneity, this is not the case.

Embedded system developers cannot use de facto standard Linux distributions, such as Red Hat Linux. For embedded use, Linux must be ported to appropriate processors and modified to work in diskless, resource constrained, and custom hardware environments. Real-time performance capabilities are also often required.

Most manufacturers attempting to use Linux in embedded applications start with freely available source code from a general-purpose Linux distribution and then port, extend, and optimize it for use in their devices. Alternatively, a number of companies have developed commercial, already adapted Linux distributions.

In order to work out-of-the-box, tools must target a commercial distribution. A tools supplier could never predict the changes a customer would introduce while creating its own Linux derivative.

Because most embedded Linux users roll-their-own, that leaves less than half of the market to the six-or-so major commercial distributions. Since none of them is dominant, no specific integration is able to address more than 10 percent of the market.

To address a meaningful portion of the embedded Linux market, a tools vendor has to work with multiple commercial distributions. This makes Linux more costly to support than a traditional embedded operating system. A tools supplier will only be able to make a profit if Linux users adopt premium tools at a higher rate than users of other operating systems.

For commercial Linux tools to be broadly adopted, developers must both see and value the benefits that they provide. Size and performance improvements are key metrics used to evaluate the value of tools. This should be particularly true with embedded Linux, since these reflect the greatest reservations of many Linux users and prospects.

In our experience, embedded Linux users and prospects break down into two camps: those who care about size or performance and those who do not. Those who do not care would not look to a commercial tools vendor for improvement. Those who do care will turn away from embedded Linux and therefore have no need for Linux tools.

Those manufacturers who care about size assume that Linux can be configured to fit in a small memory footprint. Reality is quite different. A minimal Linux configuration, which is usually closer to two megabytes than to one, actually omits most of the networking features and standard interfaces that comprised its appeal. With these capabilities included, embedded Linux often requires more than five megabytes.

The memory required by Linux is ten to twenty times greater than that required by an equivalently configured traditional embedded operating system. Even with a 35 percent improvement from commercial tools, Linux is still six times larger than the alternatives.

For any manufacturer sensitive to size and thus willing to buy a tool to reduce it, this incremental cost of Linux is prohibitive. Rather than buy a Linux tool for a 35% improvement, they would switch to another operating system for a 90 percent improvement.

Most embedded applications have real-time performance requirements. That is, the software must be able to respond to one or more external events within a fixed amount of time. If it does not, the application will misbehave or the quality of the user experience will be diminished.

Performance-sensitive Linux prospects assume that it can provide some measure of real-time response. Linux vendors' use of the term realtime to describe features of their implementations reinforces this belief.

The reality is that most--if not all--embedded Linux implementations cannot guarantee any real-time performance. (When pressed, most Linux vendors will clarify that their products are soft real-time and not hard real-time. This apparently means that real-time applications may sometimes work correctly, but not always.) As a result, Linux is unusable in any applications that require high reliability or predictability. No development tool can remedy this limitation of the underlying implementation.

For those applications that can tolerate missed deadlines or that do not have hard deadlines, most Linux implementations have typical response times of about one millisecond. This is between 100 and 1000 times slower than most other embedded operating systems. A 20 percent performance improvement is inconsequential if the application attempting to use Linux is indeed performance sensitive.

Ultimately, manufacturers that have real-time requirements or care about performance will move off Linux and not buy tools for it. A 20 percent improvement, or any improvement for that matter, would not interest the rest.

One of the fundamental assumptions that most embedded Linux prospects make is that they can rely on support from the overall Linux community. This is a well-established benefit of Linux in the general-purpose computing market.

However, because of the unique nature of embedded systems, the expected support does not materialize.

In the general-purpose market, there are many more Linux developers than in the embedded market. In addition, general-purpose Linux developers coalesce around common, highly standardized hardware and software. Nearly all general-purpose Linux users are on Intel Architecture PCs and use one of a handful of very similar enterprise Linux distributions.

These community enablers do not carry over to the embedded computing market. In the embedded market there is--by definition--no standard hardware platform. Virtually every embedded computer-based product uses a unique design. Nor is there a dominant processor architecture; nearly a dozen are widely used, with hundreds of variants.

The embedded market also does not have a closely aligned core of Linux implementations. The commercial embedded distributions that share a minority of the market can differ significantly. Most manufacturers create their own version anyway. The odds of finding two embedded Linux users with identical binary images of the kernel are almost nil. There are nearly as many different embedded Linux implementations as there are manufacturers using embedded Linux.

Because nearly every embedded Linux project is using a unique hardware-software environment, the expected support advantage from the broad Linux community does not exist. No matter how benevolent they are, engineers cannot support software they do not know running on hardware that they do not have.

The obvious refuge for embedded Linux users is to seek support from a commercial supplier. Unfortunately, most embedded Linux users are not willing to consider commercial vendors.

For customers who began with their own Linux ports, switching to a commercial Linux distribution is a time consuming and expensive process. It is particularly difficult if device drivers or real-time code are involved, since nearly every vendor has implemented these interfaces differently.

Cost is another factor. An annual "subscription" to a commercial Linux distribution typically costs more than a permanent license for a traditional embedded operating system. This is a difficult expense for developers to justify after selecting Linux because it promised better support at a lower cost than their prior homegrown or commercial solutions.

The last resort available to developers is to support it themselves. This requires that they learn the inner workings of Linux in addition to the investment they have to make in their own integration and optimization. As a result, we have seen early adopters of Linux vastly increase their support costs over what had been required for their homegrown system. The potential impact is even more dramatic for a manufacturer that had previously relied on a commercial solution and does not have an internal operating system group.

Considering all of the possible support avenues, Linux support ends up being lower quality and more costly than the alternatives of using a homegrown operating system or purchasing a proprietary one. This will dissuade manufacturers who care about support from using embedded Linux. Those who remain with Linux will do so because support is not important to them, meaning that they will not turn to a commercial tools supplier for support either.

As manufacturers recognize the real impact of embedded Linux, the tools market will dissipate. Those inclined to buy tools will abandon Linux. Those who stick with embedded Linux will have no interest in tools.

Dan O'Dowd is President and chief executive officer of Green Hills Software,Inc. (Santa Barbara, Calif.)