"Remember pronouncements of the past that embedded Linux would take over as the operating system of the embedded market? So why have reports been surfacing that embedded Linux vendors are facing a tough road ahead? The embedded systems software market is estimated at $1 billion, but only a small portion of that is embedded Linux software. In 2001, according to Venture Development Corporation (VDC), embedded Linux solution revenues increased close to $60 million. These revenue numbers, however, represent a relatively small percentage (less than 5%) of overall worldwide shipments of embedded software development solutions. Embedded Linux is just not as easy a sell as ‘server’ Linux, say analysts...."

Hank Cohen, Business Development Manager,
Wind River Networks

The recent article in Open was very interesting, but painted too rosy a picture. While Linux may have a bright future in experimental designs such as the wrist browser from IBM, one is struck by the paucity of revenue-generating designs that employ Linux. If the embedded software market were only a $1 billion business as you suggest then the 6% claimed by Linux would be down in the noise. In fact the market for embedded software is much larger than $1 billion and the part played by Linux is still infinitesimal.

Why should this be? The answer lies in your last paragraph. Mr. Balacco claims that, "... companies with the technical talent, the strong desire to control everything, and reduce costs are creating and supporting their own distribution of Linux for their devices."

Companies that truly want to reduce costs concentrate their development budgets on those aspects of their design that can deliver market differentiation. Their competitive advantages lie in their core competencies, those things that set them apart from everybody else building a similar widget. It is almost a tautology that one cannot gain competitive differentiation from software that you are bound to redistribute for free.

I do not see this question as a confrontation between proprietary software and Open Source software. Both have their appropriate roles and places in the universe of embedded systems. Companies need to evaluate all of the alternatives and understand all aspects of the costs and benefits associated with each.

Evaluating costs and benefits is an extraordinarily complex undertaking. Many costs are hidden and many benefits are intangible. Furthermore, the calculus is different for each organization. Nonetheless, a company's ability to weigh these many factors correctly can make the difference between success and failure, profit and loss or even solvency and bankruptcy. Clearly, this is a matter of prime importance to the enterprise.

It does not really matter whether the market for embedded software development tools is a $1 billion business or a $100 billion business. What matters is how to empower the developers of embedded applications to produce the best applications in the shortest time at the lowest cost.

We have entered the much heralded era of pervasive computing. Consumer electronics, medical electronics, industrial measurement and control, aerospace, transportation, telecommunications and entertainment all benefit from the application of embedded microprocessor intelligence. In all of this vast array of applications, it is not the embedded platform, the operating system, development tools or middleware, that differentiates a product from its competitors. Those elements do not confer excellence on a design. They can at best make us equal to our competitors, but they can make us worse.

It is the application that defines the embedded system and justifies the expense of development. The application dictates the hardware and software requirements. And it is ultimately the quality and timeliness of the application that will determine the product's success or failure in the market.

The source of competitive advantage will be different for every product and for every enterprise. One organization may excel at analog electronic designs, another at networking software. Core competencies may vary from engineering, to production, to distribution, to sales or service. Whatever the core competency of an organization may be, that investing in that area will differentiate that company from its competitors and lead to further excellence. Investment in other areas may be necessary to provide the infrastructure needed to support the core competency, but those investments will at best yield parity with the competition.

That’s why a company like Oracle, which develops one of the most important distributed applications, does not write networking protocol stacks. This understanding of core competencies explains why the auto industry has shifted from the huge integrated factories of Henry Ford to a system of distributed subcontractors as pioneered by Toyota.

We are at the same turning point in the embedded systems business. In the financially troubled times we face, who has the luxury to invest precious resources into efforts that do not lead directly to product differentiation or competitive advantage? Today every system component needs to pass a rigorous analysis of costs and benefits to determine whether it should be built in-house or out-sourced.

Operating systems, development tools, and middleware are not usually the core competence of embedded application developers. For this reason, they should outsource those components. Unfortunately, they are not always the core competence of many Linux developers either.

Independent Linux developers may be writing for pride of authorship and commercial developers are often funded by semiconductor manufacturers. Only a company whose business is totally focused on excellence in embedded software can be relied on to provide the best possible components.

When we evaluate commercial vs. Open Source software products, we need to keep in mind that the goal is to empower the development of market leading embedded applications in the shortest possible time and at the lowest possible cost. Furthermore, when outsourcing critical system components one cannot underestimate the importance of reliability, quality, and trust between the producer and user of the component.

As for the evaluation of costs, what matters is the total cost over the lifetime of the product, i.e. the Total Cost of Ownership. This cost is comprised of many factors and again the exact calculus will differ from enterprise to enterprise and from project to project.

Commercial software suppliers often have to defend their licensing costs for both development tools and for production licenses. These costs are not negligible. They are required to support the enterprise that provides the software product and the pay for continuing engineering, new product development, and long-term maintenance of existing products. For this money, the customer buys a committed relationship with the vendor, which in effect extends their engineering workforce. The often-vilified production license actually serves to bind the success of the software vendor with the success of the embedded application. The software vendor will only profit from production licenses if the embedded product is a success.

On the other hand there is no free lunch and "free" Open Source products often have long-term costs that can far outweigh the initial expense of commercial products. Open Source proponents like to emphasize the cost of acquisition (free or low if from a commercial supplier) and production (free). They don't like to focus on the cost of porting to a new board or CPU architecture (high). They don't say anything about the cost of testing or integration (high) or the difficulty of maintaining a stable version in the midst of a plethora of patches, distributions, different tool chains and releases and dozens of authors each with their own coding style (high).

Projects that use Linux must dedicate a certain percentage of their engineering effort to operating system maintenance. Someone needs to follow the stream of updates and patches; someone needs to test the snippets and bits of code downloaded from the Net. Someone needs to track the development tools and keep up with bug fixes. What’s more, any programmer dedicated to maintaining the development environment clearly cannot be adding to the application specific features that will lead to a successful product.

Companies that have used Linux and then abandoned it discovered that using Linux actually increases their costs, delays their schedules, and diverts resources away from core activities into OS hacking. A big part of the service provided by a commercial software vendor is continuing engineering, integration, testing and product support that allows application developer to focus on their own game.

Finally we should look at the question of license rights and obligations. There has been a lot of FUD spread around about the viral nature of the GNU Public License; however, it is certainly possible to preserve intellectual property in a Linux environment. The trick is to keep all of your proprietary information in the user process space and out of the kernel.

That may be fine for IT applications like databases or banking applications but it can create problems for embedded applications. Embedded applications often need to modify kernel structures. They add device drivers and board support packages. Special functions may be added to the kernel to achieve better performance or to extend kernel services. But even if you stay out of the kernel, you should have safeguards to ensure that no GPL code gets included in the proprietary application. All of this adds cost.

When one evaluates the total cost of ownership, the license costs for both development and production using commercial software, it is usually dwarfed by the evaluation, integration, testing, and maintenance costs of Open Source.

If you are very cash-constrained and the initial development license cost is a barrier to entry, then by all means use Linux. But if you expect the product to be a commercial success and profitability to be determined by control of lifecycle costs, then you are almost certainly better off to use a commercial product.

Realities Not Myths of Linux

Bill Weinberg, Director Strategy/Evangelism
MontaVista Software

"While Linux may have a bright future in experimental designs such as the wrist browser from IBM, one is struck by the paucity of revenue-generating designs that employ Linux."—HC

Embedded Linux is garnering approximately 750-1,000 commercial design wins /year, and a probably equal number of in-house custom embedded designs. Indeed, industry analysts all agree that by 2003 at least half of all new embedded 32/64-bit designs will employ Linux.

The embedded Linux vendors servicing these designs include MontaVista Software, Red Hat, LynuxWorks, Lineo/Embeddix, FSMlabs, TimeSys, and about one dozen other smaller, regional players. In the last year, these vendors have confronted a variety of financial and business model challenges. Nonetheless, the number of embedded Linux designs continues to grow.

It is important to note that the embedded “powers that be” (Wind River, OSE, QNX, Accelerated Technologies, and yes Microsoft) are also experiencing comparable, if not greater, woes in the marketplace, both due to the general economic conditions and also because Open Source is eroding their proprietary value propositions.

As for “visibility,” the embedded systems market and its sub-segments are by definition not highly visible. Embedded technology hides inside a myriad of devices from the everyday to the obscure. Moreover, the companies that build those devices are not self-identifying embedded developers: They identify themselves as builders of networking equipment, transportation systems, consumer electronics devices, or other widgets and not as developers of embedded applications. This makes it difficult even for insiders like Mr. Cohen and myself to uncover every nook and cranny of our own marketplace.

"The embedded software is much larger than $1 billion and the part played by Linux is still infinitesimal."—HC

Even more than most market analysis exercises, measuring the embedded software market is a black art. Many of the companies involved are privately held and each has its own method of reporting/segmenting revenue. On top of that, each analyst has his or her own methods of slicing and dicing and representing the data.

Embedded software studies usually segment the market into the following areas:

• Development tools
• Operating systems
• Add-on and deployment components (networking, graphics, drivers, etc.)
• Source code
• Associated services

While these represent good, common sense categories, they don’t capture the subtleties of bundling services with tools or an OS with components. They don’t allow for different business models. They ignore differences arising from accounting for charges on deployed components vs. charges for the act of deployment vs. charges for tools sold on a per-engineer basis.

Bring embedded Linux into this mix and the situation gets more complicated. Since Open Source is so rich in tools and utilities, most embedded Linux revenues are built on development seats and not on charging for the tools themselves.

What's more, much proprietary embedded OS revenue is run-time royalty based. On the other hand, most Linux deployment packages are royalty-free. Also, a larger portion of Linux-based revenue is in the services area. So, it is clear to see that measuring embedded software revenues is difficult, and Linux only further complicates the task.

That being said, the combined revenues of embedded Linux companies and Linux-based revenues for multi-OS tools and add-on vendors easily top $150 million. Certainly, $150 million is greater than “infinitesimal” in a market whose total size is between $1 billion and $1.5 billion.

To put this number in context, let’s look at the rest of the market. The two largest players outside of the “Linux Bunch” are Wind River and Microsoft.  Wind River has forecast an estimated 2002 revenue of around $250 million, which is sharply down from the $400+ million of recent years. As for Microsoft, their embedded revenues have been estimated to hover around $150 million based on summing XP/NT Embedded, WindowsCE and family, legacy DOS royalties, and dual-use of desktop and server Windows products. The rest of the embedded market players all run in the $20 million to $35 million range.

So, while any embedded Linux player today may appear to be “noise” to Mr. Cohen, the nascent embedded Linux industry is a threat on a par with Microsoft to all of the established players in the embedded software industry.

"Companies that truly want to reduce costs concentrate their development budgets on those aspects of their design that can deliver market differentiation. Their competitive advantages lie in their core competencies, those things that set them apart from everybody else building a similar widget. It is almost a tautology that one cannot gain competitive differentiation from software that you are bound to redistribute for free."—HC

The above statement perpetuates two fallacies commonly propagated by proprietary software vendors: (1) that proprietary platforms are less of a commodity than their Open Source counterparts, and (2) that all software, developed in house or acquired externally, and deployed in an Open Source context, must itself be Open Source.

The first is easily debunked. In general, Open Source software, like embedded Linux, is richer, more robust, and of exceptionally high technology value. Developers who choose to deploy embedded Linux do not hope to gain some illusory competitive advantage from Linux itself. Rather they seek to differentiate themselves by building more reliable, flexible, and feature-rich products on top of embedded Linux.

The  advantage derived from building on Linux is demonstrably greater than that of building on any comparable legacy Real Time OS (RTOS). By liberating developers from having to reinvent the wheel in each and every embedded project, Linux lets developers concentrate on their core competencies more so than a traditional embedded RTOS. With Linux, developers don’t have to dedicate expensive engineering resources to building system code.

The second issue has been addressed repeatedly by this author and a host of others. Merely deploying your application, embedded or otherwise, with or on top of a GPL operating system does not force you to distribute your own value-added software under the same licensing terms. Were that the case,  IBM, Oracle, Borland, and others would not be shipping Linux products and basing their corporate strategies around Linux.

The same logic applies to embedded systems. Embedded developers can and do develop and deploy both system and application-level (user space) software for and with Linux. They license that software in a variety of ways, from full proprietary all the way to completely GPL-based Open Source, and do so without violating their own legal strictures nor those that accompany the GNU General Public License.

"Companies that have used Linux and then abandoned it discovered that using Linux actually increases their costs, delays their schedules, and diverts resources away from core activities into OS hacking. A big part of the service provided by a commercial software vendor is continuing engineering, integration, testing and product support that allows application developers to focus on their own game. "—HC

Perhaps those companies did not recognize their own core value-add. With many technologies, not just Linux, it is easy and common for developers to become enamored of hacking instead of implementing. Because Linux is frequently (mis)positioned as “free” (as in Free Beer), middle management assumes that a Linux-based platform can be built by merely vacuuming the Net.

No one should be surprised that it's not easy to build, deploy, and maintain the 2+ million lines of code in the Linux kernel and the 8+ million lines of code in compilers, debuggers and other tools. On top of that, there are the many millions of lines of code for graphics, file systems, and networking. Standard engineering methodologies project a need for as many as a dozen full-time engineers to maintain a piece of software like the Linux kernel, to say nothing of the rest of the system and middleware code.

Those companies that fail with Linux only do so when they don’t recognize the real costs of rolling their own Linux-based platforms. When they do “get it,” they don’t just start over. More and more commonly, they turn to companies like the ones listed above to outsource  their embedded Linux platform and tools just like they had been doing for a legacy RTOS.

Whether or not embedded developers have a rough trial period when they switch to Linux, they still are making the switch in droves. The benefits they receive are greater freedom (multi-vendor, multi-technology, Open Source code), greater reliability, broad hardware choices with over two dozen CPU architectures supported today, and faster time to market without sacrificing control of their IP.

Embedded Linux developers also reduce their costs by not paying for source code, by lowering or eliminating run-time fees, by leveraging readily-available device drivers and other components instead of paying RTOS vendors $30,000 apiece. They can also take advantage of the wealth of experience in UNIX systems and APIs that embedded developers share with their enterprise counterparts. An experience that is not usually applicable to RTOS programming.