By Bill Graham
In the previous posts in this series, Part 1: Automation is the Key and Part 2: Security Improvement and the Software Development Lifecycle, I talked about the connection between the typical embedded device development process and the 5+1 improvement framework for embedded security. Figure 1 is an illustration of this connection (and discussed in more detail in a previous post).
Figure 1: Mapping the security improvement framework and the device software lifecycle
By Jeff Fortin
A recent study published by GAO caused quite a bit of uproar over medical device security. The GAO declared in its report [read more] that the FDA has not put enough oversight in the premarket approval (PMA) of certain medical devices that are susceptible to threats. In the report the GEO referred to an experiment described in the 2008 IEEE Symposium on Security and Privacy. The experiment was conducted by a number of experts in security, medicine and engineering. They demonstrated the potential for a hacker to expose patient information as well as interfere with the correct operation of a medical device. The FDA has agreed to the GAOs insights – however pointed out that there have been no real world threats that have been reported to date. According to the FDA, it did consider unintentional threats for information security, but did not consider information security risks from intentional threats until recently. In summary, the HHS – governing body for the FDA, concurred with the GAO recommendations and have initiated activity in this direction. Medical device manufacturers must take the threat of security seriously.
Of all medical devices, devices that are implantable, portable or used in a home health scenario represent a growing concern for security and information privacy. I refer to these classes of devices as embedded medical devices. Developers and manufacturers of embedded medical devices are increasingly exposed to security risks as these devices become more complex and also connected to open networks. The good news is that in responding to this threat they need not operate in a vacuum nor do they need to re-invent the wheel. Best practices exist in other industries that can be leveraged by medical device manufactures. Of them the most straightforward is to design your products with security in mind. In other word, design for security. Here are a few things to think about when you design your medical devices.
By Ido Sarig
Today we announced the latest version of Wind River Test Management, which takes software test optimization to the next level.
By Bill Graham
In many of my previous posts, I’ve discussed Wind River's 5+1 step improvement framework for embedded device security. By design, it’s meant to complement the software development lifecycle (SDLC) that our customers are using – the stages or phases how they define them and the processes they follow. It’s important for a discussion on tools to put them in the context of the SDLC to see where they apply and how they are useful beyond just the coding stage.
Figure 1 shows the relationship between the SDLC and our security improvement framework. The phase names and number of phases may change from customer to customer but the general idea remains the same. It’s important to note that the improvement framework is equally compatible with Microsoft’s Secure Design Lifecycle that is gaining traction amongst embedded developers too.
By Ido Sarig
Last year, market research firm voke, Inc. named Wind River a transformational vendor in its annual "Market Mover Array Report on Application Lifecycle Management (ALM)." This year, we are pleased to once again receive this same accolade for Wind River Test Management, our testing platform, in Voke’s 2012 Market Mover Array™ Report for Testing Platforms. It is an honor to once again be ranked in the highest category, reserved for vendors that are changing the tone and direction of the market!
I don’t want to steal too much of the thunder from the report (you can download and read the report for yourself here), but a few select quotes like “taking the quality of device software to a new level” and “Wind River is changing the conversation in the market about what test professionals of the future will look like” are music to my ears.
By Bill Graham
Security Vulnerabilities are Expensive
Shipping security vulnerabilities in a finished product and having them discovered or worse, exploited, is a very expensive proposition for embedded device vendors. As I’ve discussed in an earlier post, security defects are much more expensive to patch and fix the later they are discovered. If you’re lucky to catch a vulnerability during development, you can likely save your company and project thousands if not hundreds of thousands of dollars. So, security vulnerabilities are expensive and costly not only in dollars but in reputation and customer satisfaction.
On average, a vendor loses around 0.6 percent value in stock price when a vulnerability is reported. This is equivalent to a loss in market capitalization values of $0.86 billion per vulnerability announcement. --Telang & Wattal 2007
By Ido Sarig
The Economist recently published a very interesting article on the merits of open source in medical device development, which raises some questions and sparks an important discussion.
In general, it is more common to see open source adoption in non-regulated industries. However, we are seeing it more and more in regulated industries primarily because it encourages rapid innovation at far lower costs. That said, it is important to caution those in the regulated industries (aerospace & defense, medical, industrial automation, energy, automotive) towards embracing open source concepts without investing in lifecycle management tools to support the building of safe and effective devices. [Proprietary software on the other hand, is a better fit for companies that design devices in environments where changes are few and far in between. Proprietary software also enables better control of the ecosystem, the lower the stack you go.]
As the Economist puts it, “exposing a design results in safe products.” Maybe…Designing a device or ecosystem with safety in mind, results in safer products, hence the need for “design for safety.” The FDA mandates a safe and effective design for any devices that gets approved in the US market -- this requires companies to have better control of their ecosystem. Done proactively, this is an efficient method for product development; however, done reactively, companies incur a lot of costs and overheads trying to make open source fit.
By Ido Sarig
The security world is abuzz with news about a “backdoor” - undocumented access to its programmatic interface - found in a popular FPGA manufactured in China and used in US military applications.
Whether you are concerned that this is a deliberate Chinese plot to attack Western militaries, or relieved to hear that this is just a "common" backdoor, put in for debugging purposes, you should take note of the following:
"Backdoors are a common problem in software. About 20% of home routers have a backdoor in them, and 50% of industrial control computers have a backdoor. The cause of these backdoors isn't malicious, but a byproduct of software complexity. Systems need to be debugged before being shipped to customers. Therefore, the software contains debuggers. Often, programmers forget to disable the debugger backdoors before shipping.”
By Bill Graham
In the previous posts I’ve discussed various steps that need to be taken in order to improve security, but these are all preventative measures that require validation before a device is ready for market. Enabling the security features of your embedded OS is the first step, but it’s important to test the system continuously throughout development. The importance of security-focused testing can’t be over-emphasized. Testing for security is a different activity than functional testing since the aim is to find and exploit vulnerabilities, often outside the realm of normal operation. Test automation is essential in order to do efficient and thorough security testing. Take a look at a post on this from my colleague Ido Sarig.
By Pete MacKay
A few years back the FDA launched an initiative aimed at reducing radiation exposure in patients undergoing CT scans. Clearly there are concerns with exposing the human body to ionizing radiation, and due to the prevalence and popularity of these imaging techniques the FDA feels they are in a position to ensure proactive steps are taken to minimize exposure. They cite studies done in 2009 showing this total exposure to have nearly doubled in two decades in the US population.
Several factors come into play with cause and effect here, and the FDA is exploring several different perspectives. Preventative measures start with educating patients, clinicians, and technicians, and altering approaches to therapy planning and the decision process around imaging studies. The emergence of highly connected health enterprises (and society in general) enables a proposal to track cumulative dosage using tools like a national dosage registry and a patient medical imaging record – who knows, maybe just like Fido runs around with a chip in his neck we might all someday be implanted with a dosimeter along with our medical record? Kidding aside, the obvious place to start with dosage reduction is with the device manufacturers themselves.
By Bill Graham
In my previous post I discussed the potential benefits in quality and costs that static analysis brings to software development. In addition to common coding errors, many of the bugs found by static analysis are potential security defects as well. Buffer overflow, OS command injection, unrestricted string format and integer overflows are among the top 25 most dangerous security coding defects (according to the Common Weakness Enumeration (CWS) from the MITRE organization). These types of defects are common in C and C++ and are dangerous to correct operation in general but also pose significant security threats – often because the right exploit can lead to arbitrary code execution on the target. Once an attacker can execute code on your device, they can gain complete control, which might include reflashing the firmware, installing malware or rewriting the command and control software.
By Bill Graham
Fixing bugs is expensive. Fixing bugs is more expensive the later you leave them, in fact, its been shown to cost a magnitude higher with each major phase of development. The famous defect cost chart from Capers Jones shows the cost of a bug going from $25 at the coding phase to $16,000 in development. Not only that, but most of the bugs are introduced into the system at this stage yet very little of them are found. Logic says we should fix them earlier to save money.
By Ido Sarig
Last month, I had the opportunity to take part in the Amphion Medical Forum in Minneapolis, where the theme was security challenges facing Medical devices. Amphion is a forum that brings together thought leaders from academia, business, government and technology, which was founded to provide a medium for these visionaries to define solutions to some of the issues, such as safety and security, which affect the new interconnected economy, the “internet of things”.
I participated in a panel that discussed the challenges and risks posted by the latest generation of wireless medical devices, and how proper Fuzz Testing, such as the capability offered by Wind River Test Management in conjunction with our partner Codenomicon, can help alleviate some of that risk.
By Chip Downing
MILCOM 2011 last week in Baltimore, MD was not only educational, but it was a watershed event. Last year, it was a “coming out” party for custom Android devices; this year however, was the year where many companies are now using COTS hardware and software to their fullest extent, along the entire supply chain, from core to edge and user devices. I believe this COTS adoption is driven by two powerful forces – the exploding capabilities of commercial tablet devices, and global military budget pressures.
The vastly increasing power and capability of commercially available mobile communications devices is simply amazing. Just a few years ago, almost everything in this military communications space was purpose built, and most commercial products were simply inadequate to address the rugged and complex system demands of military communications systems. As seen at MILCOM 2011, this has officially changed -- COTS end-to-end communications domains rule.
So if all of these MILCOM vendors are not making customized hardware and software, what are they doing? Wind River had over 500 people visit its booth at MILCOM this year, and most of the discussions I had with both suppliers and end users at the show were about increasing device security and leveraging high volume COTS components in demanding deployment scenarios.
By Bill Graham
Selection of secure components for an embedded system is key to a secure system. Leveraging a secure RTOS, middleware, virtualization and tools significantly reduces the effort and development costs. Moreover, there are additional benefits from using commercial-off-the-shelf (COTS) software components over Roll Your Own (RYO) code or self-ported and maintained open source code. Some of the COTS components worth considering include:
By Ido Sarig
Big day today, we just announced the release of our latest version of Wind River Test Management, WRTM 4.0, the culmination of several years of hard work - kudos to our engineering team!
In the coming weeks, I will be covering many of the new features of WRTM 4.0 in detail in a series of separate posts, but today I wanted to give you an overview of some of the new things you can look forward to.
I’ll start by sharing a quote from Wind River customer GE Healthcare on how Test Management is helping their business -- a real testament to the value that Test Management delivers.
"Wind River Test Management has tremendously helped us improve the productivity of our firmware testing team, significantly reducing the overall development cycle time. And although we already had more than 20 tools as part of our test and development process, test plans and test results were still mostly paper based. Wind River Test Management easily integrates with our existing solutions, such as HP Quality Center, to provide a complete automated regression testing suite running tests remotely on the target and collecting valuable metrics for tracking test progress and software quality. GE Healthcare operates on a ‘zero defects’ policy and Wind River helps ensure that this policy is met much faster than before."
By Ido Sarig
Last summer was a watershed event for security-consciousness in the embedded systems world: Stuxnet, a highly sophisticated worm exploited no fewer than 4 zero day vulnerabilities in Windows in order to attack a specific Siemens PLC and its associated SCADA system. The target was reportedly the Iranian nuclear facilities at Natanz, where uranium-enrichment centrifuges were taken out of commission by the worm’s malicious payload. It was perhaps not the first, but certainly the most well-publicized successful attack on critical infrastructure systems.
The software security industry has been discussing such an attack for years, mostly as a theoretical possibility – but now it appears this is no longer the stuff that Hollywood scripts are made of - it is all too real.
By Ido Sarig
When I was in high school, there was a popular TV show called “Moral Dilemma” that aired in my native country. It featured a panel of professionals such asdoctors and lawyers who discussed “life or death” issues related to their daily work: Would you represent and defend a client charged with murder after he confided in you that he indeed killed his victim? A kidney has become available for transplant – do you operate on the rich, 85-year-old patient who promised to bequeath a million dollars toward the creation of an organ bank at your hospital, or provide the kidney to the teenager who was just accepted into the country’s most prestigious college?
The show portrayed these professions as dealing with the most important moral issues imaginable, and by implication, relegated other professions to a lower rung on the ladder. As someone growing up from a family of engineers, I knew engineers can have significant roles to play and I definitely had respect for other professions not represented on this TV show. However, it did get me to thinking: Were there also life or death issues in software?
Fast forward 30 years, and I know that there are indeed life or death scenarios in the world of engineering and software. Embedded software is a vital element in the majority of today's sophisticated medical devices—and when it fails, disastrous results follow.
By Jakob Engblom
The US National Highway Traffic Safety Administration (NHTSA) recently released a deep report into last year's issue with "unintended acceleration" on certain Toyota cars. They actually employed a team from NASA who analyzed the throttle control software using a wide range of cutting-edge tools. Reading their report gives a good idea for how embedded control software is developed, and the challenges inherent in validating it.
The conclusion is that the software is not a likely cause of the problems. The real meat of the report (and its appendixes) is just how this conclusions is reached, and what it says about the tools that could be applied to develop embedded software with greater confidence about its correctness.
By Jakob Engblom
The other day, I spent some time getting a new operating system up on one of our Simics virtual platforms. The platform is stable, the hardware is shipping and it is being used with the very software I was setting up. However, as the operating system was booting, I got quite a lot of warnings from Simics about incorrect hardware settings. The operating system still worked, however, so why did we see all these warnings? Actually, when is it prudent to issue warnings to a user about suspicious uses of hardware? When building Simics models, we want them to be helpful and offer warnings as much as possible - especially for cases where some uses might be considered technically "correct" but are actually somewhat suspicious.
A virtual platform like Simics can print warnings to the simulation console when software performs operations that are not legal according to the hardware specification. Typical examples are writing to offsets in device register banks where there are no registers, changing the value of bits which are marked as "reserved", or writing illegal values to configuration registers.
By Ido Sarig
Half way into the 2010-2011 NBA basketball season, the Pacific division leading LA Lakers had an embarrassing loss to the bottom-dwelling Memphis Grizzlies. The LA Times described the performance of several of the players at the Sunday, Jan.2, 2011 game as “sleep walking” – which turned out to be not far from the truth!
Three Laker players had reported oversleeping the night before the game, because the alarm clocks on their iPhones malfunctioned that morning, part of a well publicized failure.
If this sounds familiar to iPhone users, it’s because it was a repeat of a nearly-identical episode, reported just two months earlier, when the iPhone alarm application failed to operate properly after the switch back from Daylight Saving Time to standard time.
What does this have to do with software engineering? It illustrates a well-known phenomenon in large-scale software applications, researched as early as the 1970s, when IBM performed a “distribution analysis of customer reported defects against their main commercial software applications” and were “surprised to find that defects were not randomly distributed through all of the modules of large applications” – or in plain English - some parts of the code are more buggy than others.
By Ido Sarig
Many years ago, circa 1990, as I was starting out my professional career as a software developer with Mercury Interactive, I went on a customer call to a Silicon Valley-based software company, to present our new product for automated software testing. Apparently, our message did not get through, as the response we got at the end of our pitch from the company’s VP of engineering (who shall remain nameless) is the title of this post.
I don’t want to go as far as saying that this was the prevalent attitude in those days, but I also can’t honestly say it was atypical. Back then, many in the industry, especially in engineering roles, thought it was a reasonable process to develop an application, debug it as best you can, ship it, and then fix the remainder of the issues as they surface at customer sites. This, despite numerous studies showing it is much more costly to repair bugs found in the field than it is to find and fix them before shipping.
By Chris Buerger
There is no doubt that there will always be a set of consumers that only want their mobile phone to be, well, a phone. Voice calling is likely to remain the ‘killer app’ for the majority of consumers for a long time to come, and there will remain a sizeable section of the mobile phone market that will cater to this demand.
However, the market change to smartphones is undeniable, even in the last bastions of feature phone-centered markets such as Japan and South Korea, Android-based smartphones are rapidly winning market share at the high end of the device spectrum. Now, a second wave of the Android smartphone revolution is happening in many countries, and I am excited to say that Wind River mobile software solutions for Android are right in the thick of things, helping to drive this new wave of devices.
By Jakob Engblom
When you explain what a virtual platform or full-system simulation solution like Wind River Simics is, a key part is that it runs the same software as a physical board or physical system. This does capture the core of the idea - but it does not mean that the virtual platform is only "just like the real thing".
Comparing virtual and physical systems is like comparing apples and apples, not apples and oranges: while apples are mostly interchangeable, they is certainly variation between them. Some apples are best for eating, some are better for making sauce, some are pie material, and some are best for fermenting cider. The type you select depends on what you want to cook. The difference between physical and virtual hardware is similar: they can be used as replacements for each other to some extent, but the connoisseur can make much better use of both by looking at the differences.
By Paul Henderson
I'm seeing increasing interest from many companies in using simulation environments with test automation systems to accelerate the testing process. Specifically, putting Wind River Test Management together with Wind River Simics is getting creative juices flowing in industry thought leaders.
Why? Well, development teams have started to realize the benefits of simulation systems for speeding and validating system and software design, and for accelerating software development and debug in advance of hardware availabilty. And even when hardware is available, systems like Simics provide tremendous access and control to speed analysis and diagnsotics.
By Jakob Engblom
I have a paper about "Transporting Bugs with Checkpoints" to be presented at the S4D (System, Software, SoC and Silicon Debug) conference in Southampton, UK, on September 15 and 16, 2010. The core concept presented is to leverage Wind River Simics checkpointing to capture and move a bug from the bug reporter to the responsible developer. It is a fairly simple idea, but getting it to work efficiently does require that some things are done right.
By Paul Henderson
As I've mentioned before, we've been working with IBM Rational for some time around quality management automation. Both companies see the skyrocketing software content and architectural complexity in the embedded device market as creating a tipping point where companies will not be able to continue with business as usual.
Product development teams will need to take a more managed and automated approach to quality that spans across the lifecycle and access into the devices under test. This is particularly true in markets that require strict adherance to standards and compliance regulations.
We put together a joint whitepaper on this subject downloadable from here. And we are also having a joint web seminar next week on Tuesday Aug 31 at 2pm EDT. You can register for this event here.
By Paul Henderson
RTC Magazine recently published an article that I wrote called "Time to Rethink Software Testing for Embedded Devices". In it I describe some of the new techniques that are possible, and I believe necessary, to delivery high quality device software for embedded devices.
By Nikhil Chauhan
A recent report describes potential security vulnerabilities in devices running VxWorks. Researcher HD Moore claimed during a recent talk ( slides) that a quarter million devices accessible directly from the Internet were found to be vulnerable.
VxWorks has a very strong track record of offering secure products. However, we also realize that vulnerabilities can affect VxWorks, even if very infrequently. In those cases, Wind River will act quickly to address any issues. Regarding recent vulnerabilities, Wind River responded rapidly with patches and remediation steps in conjunction with a public announcement by the CERT Coordination Center on August 2, 2010. Once CERT notified Wind River, Wind River immediately assessed the alert and was instructed by CERT to release a synchronous public response. We're confident that our customers know that Wind River is committed to supporting its products with the highest quality and security standards.
By Paul Henderson
In my last posting I mentioned I'd be running a webinar with James Grenning on Agile testing. James is a recognized expert and frequent speaker on the topic of software development and one of the original authors of the Manifesto for Agile Software Development.
We talked about the case for agility where today's embedded software projects are inevitably faced with changing requirements and market conditions that cause unplanned, mid-course corrections. The result is what went in is often not what was expected to come out. Testing folks are the tail trying to wag the dog as they try to test in quality at the end of the project.
Recent Comments