Embedded Systems, Linux, and the Futureby Karim Yaghmour , author of Building Embedded Linux Systems
Predicting the future is a tricky endeavor, and an often-failed one, at that. So I won't try to predict the future in this article. Instead, I will present how the traditional players of the embedded systems field are currently positioned, what impact Linux is having, and what forces will decide where the field is going. I will also cover the various initiatives, moves, and trends trying to influence Linux's move into the embedded systems field, and what role the open source and free software community (along with embedded system developers) should play to ensure that Linux is the best choice for an embedded OS.
The Traditional Embedded OS Market
Before anything useful is said about this market, one has to keep in mind that for a long time, 50 percent of embedded systems were running custom-made, in-house operating systems. That's an important figure, as many of the engineers deploying a "roll your own" OS are increasingly attracted to Linux. So beyond grabbing market share from established embedded OS vendors, Linux is also penetrating the "last frontier" of the embedded OS world.
I don't intend to discuss the history of the embedded OS market in detail. Rather, I'm more interested in analyzing how this market has reacted to the surge of enthusiasm generated by Linux's increased use in embedded system development.
Contrary to the workstation and server market, there has been no traditionally dominant embedded OS vendor. Though WindRiver Systems (WRS) is considered the most important embedded OS provider (and despite Microsoft's attempts to enter the market with Windows CE), the embedded OS market remains fragmented, with no clear leading company. Nevertheless, a few companies, including WRS, QNX, and Lynx, have occupied an almost equal share of this market.
Generally, the traditional embedded vendors and publications have been dismissive of Linux. Jack Ganssle, a widely respected figure in the embedded systems arena, recently wrote in his monthly column that "Despite all the hoopla about Linux, I think the future is far too hazy to predict its dominance in the embedded world." The future may be hazy, but established embedded OS vendors haven't been taking any chances either.
The first to move was Lynx Real-Time Systems, which announced in November 1999 that it would be releasing a new distribution, called "BlueCat Linux," aimed at embedded applications. Later, in April 2000, in what was a stunning move, the company rechristened itself into "LynuxWorks" to reflect its newfound allegiance. In January 2001 at LinuxWorld New York, the company demonstrated a capability it added to its proprietary Real-Time Operating System (RTOS), LynxOS, to run unmodified Linux binaries. LynuxWorks was thereby promoting a two-fold solution, where customers wanting hard-real-time performance should use LynxOS, and those wanting a non-real-time, royalty-free embedded OS should use BlueCat. The company's strategy is unchanged to this day, and LynuxWorks continues to provide both LynxOS and BlueCat.
QNX, too, has seen the skies change, and made some moves of its own. In April 2000, around the same time LynuxWorks found its new name, QNX announced that it would make its OS available free for download for noncommercial use. True to its word, the software was made available to the public in September 2000 and continues to be available here. Contrary to LynuxWorks, however, QNX decided not to touch Linux at all. Instead, it has been preaching that Linux is not adapted to real-time applications. In a whitepaper entitled "Real Time or Real Linux?," for example, Paul Leroux from QNX explained how QNX was better for Linux adopters than Linux itself.
Needless to say, WRS wasn't going to be left behind. In February 2001, an article appeared in Integrated Communications Design where WRS's vice president of corporate marketing expressed doubts about the legality of using GPL code in embedded systems. To counter the claims, LinuxDevices.com soon published an informational piece in which one of the subtitles was "FUD? or a legitimate concern?". (I was personally guessing the former.) The company's next move was to clarify its intent a little better. In April 2001, WRS acquired BSDi's assets. It appeared that WRS was going to counter the Linux menace using BSD. The future was to be different, however. In October of that same year, WRS laid off its FreeBSD developers, and its moonwalk with BSD seems to be over.
WRS still saw Linux as a threat, however. In October 2002, Jerry Fiddler, a WRS co-founder, published a whitepaper entitled "Linux in Embedded Systems: Where Are the Benefits?," where he essentially stated that Linux is a server OS, and that it isn't fit for embedded applications. I won't cover each point he makes, but suffice it to say that the copy I printed is heavily marked around the various claims. There is, nevertheless, a hidden gem in this paper for the patient reader. Around page 5, Fiddler makes the following confession: "... we did full Linux development of a Tornado toolset ... We then decided not to release it because it would push our customers into the gray area." Though he picks up the "the GPL is dangerous" line, he clearly states that his company has been playing at matchmaking between its tools and Linux. Despite his other claims in the whitepaper, one is left with little doubt about the company's continued need to look over its shoulder.
Though all of these companies have attempted to cope with Linux's emergence in one way or another, it is interesting to note that none of them have made any noticeable open source or free software contribution; nor have they attempted to engage the open source and free software community in one way or another. Their greatest mistake, in my opinion, lies in their attempt to please embedded developers by providing substitutes or spreading FUD. The open source and free software community is here to stay, and embedded developers' interest in the software produced by this community is only likely to increase.
The "Embedded Linux" Market
While the traditional OS vendors were worrying about their place in the market, a whole new bunch of vendors appeared: some of them almost overnight, each prepared to take over the entire market.
During the initial race to conquer the embedded OS market, there were four main contenders: Lineo, LynuxWorks, MontaVista, and Red Hat. I've already covered LynuxWorks in the previous section, and I will therefore skip it in this one.
In contrast to the vendors in the previous section, quite a few "embedded Linux" vendors have collaborated in one way or another with the open source and free software community, much like their mainstream Linux distribution counterparts. For the community, this is an upside of these vendors' involvement with Linux. (I've already discussed some of the downsides in my last article.)
I don't intend to provide a history of these vendors. Instead, I'm more interested in these vendors' prospects and practices in the market.
As with all other vendors relying on the sale of software under open source and free software licenses, the most important issue to figure out for "embedded Linux" vendors is how to make money on software that is freely available. Even the largest of the mainstream Linux distributions have been struggling with this issue for quite a few years now. While Red Hat seems to be on the track of profitability, it is unclear that other vendors will be able to do the same. And the prospect for "embedded Linux" vendors is even worse than mainstream vendors, since they sell software to a very technically talented crowd. While Red Hat can count on selling support contracts to large corporations with thousands of users, there are only so many embedded developers out there.
Not surprisingly, the established embedded systems press has been having serious doubts about the viability of an "embedded Linux" market.
In November 2001, Jack Ganssle wrote a column entitled "Is Embedded Linux a Bust?" where he discusses his doubts about Linux's ability to gather any interest in the embedded world. The main problem with this article (and that of many others, I should add), is that it makes no difference between Linux's viability as an embedded OS and the viability of the "embedded Linux" market. And sure enough, some of Jack's readers took notice. Shane Nay wrote in reply: "There are very few people that ... truly understand Linux embedded development. If you are looking to MontaVista, Bluecat, or Lineo, you've already shown that you don't understand Linux embedded development." Elsewhere, Neil Horman wrote: "... Linux is still new and strange to the men and women making the big decisions. And the marketing groups from the companies that charge 50K for a yearlong OS license aren't clearing anything up for them," a feeling echoed in my previous piece.
In January 2002, Michael Barr, editor-in-chief of Embedded Systems Programming, wrote a piece entitled "Open Sores," where he said: "... the market isn't growing as rapidly as most analysts predicted it would. And I don't think it ever will." He was probably right, but, as I said earlier, the failure of the "embedded Linux" market doesn't mean the failure of Linux as an embedded OS. Tom Williams, editor-in-chief of RTC Magazine, followed suit by issuing a caveat emptor to "embedded Linux" vendors in May 2002 when he wrote in an editorial for LinuxDevices.com that "the siren song of Linux has lured many brave hearts to sea with a promise of treasure that is truly there. But many brave hearts will sleep in the deep and only a few will bear home the prize."
EETimes was not missing out on the action, either. In quite a few articles, it quoted analysts and experts calling into question the viability of the market. In May 2002, Jerry Krasner, a VP of market intelligence for Embedded Market Forecasters, was reported saying: "Because Linux is a giveaway product, vendors have to sell services or tools along with it ... and no one has yet proven that embedded developers will want to pay the cost for those services." In July 2002, Krasner was again reported saying: "Services are limited because they tend to be people-intensive. It's a model, but it's not a billion-dollar model. No company is going to get really big using a services approach." In that same article, Paul Zorfass, a senior analyst for International Data Corporation, was stating what all open source and free software developers already knew: "The success of these companies is not necessary for the survival of Linux. Linux could still be successful without them." Again in February 2003, EETimes was repeating the diagnosis: "[analysts] are not sure a service model, under which suppliers pay to help to integrate Linux into customer products, will prove a strong foundation for large-scale growth."
Such conclusions are not only substantiated by argument, but surveys such as the one recently published by LinuxDevices.com clearly show that most embedded developers prefer not to rely on prepackaged software providers.
For all the noise "embedded Linux" has made in the press, however, one is forced to notice that open source and free software developers who take part in developing the technology have been completely ignored. The pundits may rave and rant, but if they have no contact with those driving the technology (and for sure, these aren't the vendors), then how can they be expected to come up with a complete picture?
There is a lesson to be learned here by the established press covering the embedded systems field: unlike traditional embedded OSes, Linux is driven by a very active community made up of very different people with very different interests. These are the folks making the real news about Linux's use in embedded systems.
Back to the vendors' adventures in "embedded Linux" land, pieces have fallen through, as expected. After bleeding employees and divisions for quite a few months, Lineo, one of the initial contenders, was finally purchased by Metrowerks in December 2002. Red Hat, for its part, has apparently scaled down its embedded efforts in order to concentrate on its specialty, servers.
Those that remain in the market have tried adopting various mechanisms of survival. Some "embedded Linux" vendors, for example, require that you sign NDAs or other restrictive licenses before they provide you with any of their software. The compatibility of such contracts with the GPL is, of course, an open question, given that the GPL explicitly states that no other restrictions can be placed onto GPL software in addition to the GPL itself. And sure enough, one such restricted license, in this case MontaVista's, was called into question in a thread on the Linux Kernel Mailing List (LKML). As with other threads on the LKML, there were those supporting the company and others worried or angered by it. While I won't cover the entire thread, there is, I think, one comment of particular interest, made by Oliver Xymoron: "Let me point out that I never saw the NDA in question but said coworker was sufficiently intimidated by it that he was unwilling to give me a copy of the kernel and gcc sources because of it." One could reasonably argue that this is the desired effect of such licenses, even though it may be legally demonstrated that these licenses are somehow GPL-compatible.
Some vendors have also been trying to restrict the number of people to which they have to provide copies of GPL software by invoking section 3, and 3-a in particular, of the GPL, which requires that if you distribute a binary that you must fulfill one of three requirements, the first being that you provide every recipient of the binary with the full source code. Basically, they claim that they are not required to provide the source code to anyone else than the recipient of the binaries. While this applies readily for unmodified code, most proponents of this argument forget to consider section 2, and 2-b in particular, which states that there are three requirements which must all be fulfilled in case of modified GPL software, the second of which requiring that any published modification must also be licensed to all third parties under the terms of the GPL. Though I don't intend to entertain a debate about the GPL's meaning because I'm not a lawyer, I doubt the interpretation being made by these vendors is within the spirit the FSF has tried putting in the GPL.
Vendors are, however, not alone in trying to restrict access to GPL code. Some of those manufacturers putting Linux in embedded systems seem to have overlooked reading the GPL. In January 2003, EETimes was presenting a new initiative by Japanese electronics companies to standardize Linux for consumer devices. While discussing the matter of "securing" the device, the article states: "The issue is how 'to not expose certain functions' to the open source community to protect a CE device from hackers." One is left wondering why a device needs to be secured from those developers who made the software that it runs. In another instance, Kevin Dankwardt reports in an article for LinuxDevices.com that when he requested a copy of the Linux software running on a device he had recently purchased, he "was politely informed that [the] product uses 'a proprietary version of Linux.'" Examples like this are likely to increase as Linux is more often used in embedded systems. They do, however, point to a serious misunderstanding by those using the OS, which could come back to haunt them, and cost them.
As one reader of my previous article succinctly put it: "If the 'Embedded Linux vendor' is unable to contribute the code changes back to the originating open source project, that should be a warning signal; either they are too incompetent (not necessarily technically, but communication-wise) or [the] client's best interests are not in their best interests."
In light of the above, there is reason to continue asking, as the mainstream press has been doing from very early on, whether an "embedded Linux" market is even possible. What's more, there is also reason to ask whether any of the traditional embedded OS providers will survive Linux's entry in the embedded world. Since I can't predict the future, I will not venture into answering these questions. One thing is clear, however: whether any vendor is left standing at the end of the day, Linux will continue to be increasingly adopted by embedded system developers.
There is little doubt about Linux's interest to embedded system developers; however, it remains that Linux itself still has some way to go to be ideally suited to all embedded applications. Don't get me wrong, Linux is already a good fit for quite a few applications. It has, nevertheless, a laundry list of things to be worked on.
First and foremost, Linux is not a Real-Time Operating System. The Linux kernel was never designed for, nor is it able to provide, deterministic response times. Some developers are able to shortcut this limitation by controlling the exact set of applications and drivers running on a system, but it remains that the API provided by the kernel is non-deterministic.
The RTLinux project, which was seen by many as a potential solution to this problem, has only hindered the finding of a solution, given its originator's patenting of the method implemented by the project and the lack of legal clarity regarding the patent license's applicability. At the time of this writing, the RTLinux project is practically dead and none of the those working on the commercial offerings of FSMLabs, the company founded by RTLinux's creator to market RTLinux, is actively contributing new open source RTLinux code, or helping users on the RTLinux mailing list.
Instead, the leading project for providing real-time response times in Linux is RTAI. It has an active development community, and a very active user base. The project also includes some of the most innovative ideas about real-time and Linux out there, including the ability for normal memory-protected Linux processes to become hard-real-time tasks scheduled by the real-time scheduler. RTAI was, however, hindered by two issues: its use of the patented method for running a mainstream OS as the least-priority task of an RTOS, and the hard-to-browse hierarchical organization of its source tree. The first problem was solved with the introduction of the Adeos nanokernel, proposed by your author and implemented by Philippe Gerum, which allows different OSes to run side by side on the same hardware. The second problem is currently being solved with the opening of an unstable tree where RTAI developers are reorganizing the components into a more practical layout.
In addition to being useful to RTAI, it's interesting to note that Adeos could be used to run other OSes side by side with Linux, including traditional RTOSes such as LynxOS, QNX, and VxWorks. This, however, would require the vendors of those OSes to modify their products to use the nanokernel's services.
Moving beyond real-time, there's the issue of standardization. Embedded developers, as do their workstation and server counterparts, want to be able to develop applications and have them run on as many systems as possible. This is where a common set of APIs, facilities, and capabilities is worth entertaining. Before I discuss this issue, however, keep in mind that standardization in the open source and free software world works in very different ways than traditional efforts, where there is a governing body, and rules for submissions, licensing issues, etc. All standards that have lived long enough to be recognized as such by the community have had one main characteristic: they were open for all to contribute to.
In this regard, the Embedded Linux Consortium's attempt to provide a standard was doomed to fail from inception because the Platform Specification (ELCPS) it developed was entirely written behind closed doors, with anyone wanting to participate having to sign legal documents. It comes as no surprise, therefore, that many open source and free software contributors I have talked to, some of whom's input would be critical to any standardization effort having to do with Linux's use in embedded systems, do not recognize the ELCPS, nor are interested by it. Because of this lack of community grounding, there is no guarantee Linux will move in the way the ELC wants. In fact, it may move in ways that make any standard put forth by the ELC impossible to implement.
None of this, however, precludes the ability to provide embedded system developers with a standard on which to build. If nothing else, embedded system developers can rely on existing standards, such as the LSB or the FHS. Yes, both have been tailored for workstation and server systems, but because they are open projects, one can easily imagine having special embedded system addendums or some similar provision for allowing the specification of embedded Linux systems.
I have tried to initiate a move towards some form of open standardization in my book by providing a worksheet that allows developers to fully describe their system for others. The success of such efforts, however, relies heavily on a common decision by practitioners to adopt a unique set of rules. Because the open source and free software community is structured as a meritocracy, it more or less guarantees that the best proposal will be the one most widely adopted.
There are also other issues, though smaller in scope, which merit some attention. In particular, as I note in my book, Linux's support for industrial networking protocols is in its infancy at the moment. Similarly, Linux's support for being used as a USB device is embryonic. There is also the matter of embedded GUIs. As on the desktop, there is no standard GUI for embedded Linux systems. Time will show which one becomes dominant.
Linux has always been, and will continue to be, a work in progress. Its successful use in any field of application has always depended on one main thing: the active involvement of a few practitioners of that field of application to pull on Linux to fit their desired purpose. Embedded systems are no different, and Linux's successful use in embedded systems will be mainly dictated by embedded system developers' active and direct involvement in its development.
In the end, the onus is on embedded system developers. They are the ones who are increasingly deciding not to purchase traditional embedded OS vendor products, and they are the ones who are unconvinced by the notion of paying for a freely available OS. As I said in my book's preface, there is a clear opportunity for these developers, many of whom are very talented, to be active and direct participants in Linux's development. Such an involvement would bring about a best-of-breed embedded OS that is understood and used by all as the standard OS for all 32- and 64-bit embedded systems.
Return to the Linux DevCenter
O'Reilly & Associates recently released (in April 2003) Building Embedded Linux System.
Sample Chapter 5, "Kernel Considerations," is available free online.
For more information, or to order the book, click here.