Paul Henderson

Test Automation Meets Simulation

I'm seeing increasing interest from many companies in using simulation environments with test automation systems to accelerate the testing process. Specifically, putting Wind River Test Management together with Wind River Simics is getting creative juices flowing in industry thought leaders.

Why? Well, development teams have started to realize the benefits of simulation systems for speeding and validating system and software design, and for accelerating software development and debug in advance of hardware availabilty. And even when hardware is available, systems like Simics provide tremendous access and control to speed analysis and diagnsotics.

Testing teams are also now starting to see the benefits of simulation if they can bring it into an automated environment like Wind River Test Management. One obvious benefit is the ability to start development and validation of test cases much earlier in the development cycle. Perhaps more important though is the ability to use simulators to do regression testing within Agile and continuous integration methods.

During these iterative development processes, developers need an automated environment to re-run their unit tests and module tests within a regression test process. This is dramatically easier by capturing these tests within a test management system that can run this over and over again as features are added or software is refactored. Meanwhile testers can support system integration and start running their tests on the simulation environment rather than waiting till the target device is available, getting valueable feedback earlier. Further, test groups find they can be even more flexible even late in the cycle by using both simulators and physical systems to distribute their heavy testing loads.

Wind River Test Management and its integrated Virtual Lab Managers allows testers to easily capture tests and manage the cycle, while transparently utilizing simulators along side physical boards in the system test lab. Stay tuned for more on this subject in coming weeks.

Meanwhile, if you live in the Dallas Texas area, we are holding a live seminar on this subject on Sept 14, 2010. You can register for this seminar by clicking here.

At this complimentary half-day seminar we will show how utilizing an agile and iterative development methodology across both hardware and software allows system integration and system testing to become an integral part of the product development cycle,  instead of something rushed through at the end. This can significantly reduce your risks and costs, decrease your overall time-to-market, and increase your product quality. 

We will showcase Wind River Simics and Wind River Test Management products and cover the following subject with presentations and demos:

• Perform code coverage on production software without need of source code instrumentation
• Inject system and hardware faults to test software robustness
•  Identify which set of tests to rerun based on target software changes
• Find bugs quickly by executing the system in reverse and saving/reloading the exact system state that caused the bug
•  Automate the creation and management of many different target hardware configurations
• Streamline your system integration and testing process and significantly improve your product development life cycle

Please drop me an email at paul.henderson@windriver.com if you have immediate interest in learning more about this subject.

Wind River and IBM Attack Software Quality

As I've mentioned before, we've been working with IBM Rational for some time around quality management automation. Both companies see the skyrocketing software content and architectural complexity in the embedded device market as creating a tipping point where companies will not be able to continue with business as usual.

Product development teams will need to take a more managed and automated approach to quality that spans across the lifecycle and access into the devices under test. This is particularly true in markets that require strict adherance to standards and compliance regulations.

We put together a joint whitepaper on this subject downloadable from here. And we are also having a joint web seminar next week on Tuesday Aug 31 at 2pm EDT. You can register for this event here.

In this webcast you will hear how IBM Rational and Wind River are teaming to deliver a lifecycle approach to quality management by linking our respective products, IBM Rational Quality Manager and Wind River Test Management. We will discuss seven key strategies for optimizing development – and how you can eliminate the friction in your delivery which can lead to missed cost, delivery and quality targets.

Key Highlights:

• Best practices to help facilitate effective collaboration around quality
• How a requirements-driven approach can help you deliver the right solution
• Ways to support automated and integrated testing via the ecosystem
• Processes to accelerate defect detection and resolution for efficient delivery

I will be joined in the Webcast by Moshe Cohen, Sr. Director of Products, IBM Rational, and Karla Ducharme, Market Manager for Aerospace & Defense, IBM Rational. Please join us if you are available or watch the replay when it is available at http://www.windriver.com/seminars/web-seminars/ after the event.

It’s Time for Testers to Step Up

RTC Magazine recently published an article that I wrote called "Time to Rethink Software Testing for Embedded Devices". In it I describe some of the new techniques that are possible, and I believe necessary, to delivery high quality device software for embedded devices.

• When staying 'positive' doesn't pay
• Getting negative with white box testing
• Focusing on the 'deltas' with change-based test automation

I also express my belief that test professionals have to step up to the new challenge. More sophisticated devices require more sophisticated approaches to testing them. I've seen too many test teams that are comfortable in their old ways while letting serious defects get to market. I've literally had teams tell me that "I tested to the requirements. It's not my fault that the product is defective." What kind of a quality assurance organization is that??

QA teams need to take a more active role in understanding how these devices work internally. They need to work much more closely with development teams, particularly when using new iterative and agile methodologies.

They also need to provide an independent view to device operation and deployment so that they can project where additional failures could occur from operator error, component failure, or complex deployment configurations. They must anticipate how devices may fail in ways that were NOT anticipated by designers.

With this new more educated an proacitve approach, better products will be delivered -- and testing professionals will redefine their role into a more strategic and valued place on the team.

Test Driven Development Meets Continuous Integration

In my last posting I mentioned I'd be running a webinar with James Grenning on Agile testing. James is a recognized expert and frequent speaker on the topic of software development and one of the original authors of the Manifesto for Agile Software Development. 

We talked about the case for agility where today's embedded software projects are inevitably faced with changing requirements and market conditions that cause unplanned, mid-course corrections. The result is what went in is often not what was expected to come out. Testing folks are the tail trying to wag the dog as they try to test in quality at the end of the project.

James made the case for iterative and incremental test-driven-develoment (TDD), continuous testing and continuous integration. By breaking projects down into smaller chunks that can be designed and tested, teams are more likely to hit their functional requirements with timely, quality software. In fact with TDD developers and testers work together to define the tests that will verify each chunk of functionality up front. This helps clarify what the feature will do (and what failure modes need to be verified) and gets the testing done early when the development work is fresh in the minds of the team. This avoids late cycle surprises and snowballing design errors.

We also reviewed some of the subtleties of applying TDD to embedded development. Testing needs to be done not just at the developers desktop but also cross-developed to simulators, reference boards and finally the device itself. Only when the software is verified on the actual device can you be sure you've got it working right.

To make all this work you need to be employing continuous integration methods and using automation technology. Rather than waiting till late in the cycle to integrate work from multiple developers or teams (software first then hardware and software together), you need to have an environment where you can do integration on a frequent basis -- bi-weekly, weekly or even daily. Again, the goal here is early verification before problems get out of control and early feedback to keep the program on track.

The challenge with all this is how to make it work in the real world. How do you capture tests as they are developed and then reuse them on multiple targets for regression tests? How do you efficiently manage a range of targets from hosts to reference boards to real devices? How do you know how effectively you are actually testing the device? Given the frequent changes, how do you focus on that's new or changed rather than testing the same things over again? And lastly, with testers and developers working closer together, how do you enable this collaboration, particularly across globally distributed teams?

This is where automation comes in. We discuss how Wind River Test Management can help address each of these challenges by providing an embedded-centric, collaborative testing environment with a built in target device manager plus a dynamic analytics engine that can let you 'see inside' a device under test and provide timely accurate feedback and control.

The replay for this event is now on line. Click here if you want to see the actual webinar. Enjoy!

Agile Testing for Embedded Devices

I am running a Web seminar tomorrow on agile testing with James Grenning. James is a recognized expert and frequent speaker on the topic of software development. Founder of Renaissance Software Consulting, he provides training, coaching, and consulting to corporate and government clients worldwide. James is one of the original authors of the Manifesto for Agile Software Development and is presently leading efforts to introduce Agile development practices to the challenging world of embedded systems.

 

Iterative software development methods like Agile are being adopted by many software and information technology (IT) organizations across the industry. Quick design-develop-test cycles let teams respond better to changing requirements while providing timely feedback on features and quality. Agile methods help companies produce higher-quality software faster and at lower costs.

 

But across the embedded industry, adoption of Agile is lagging. The modular designs, complex technology, cross development needs, and on-device testing techniques used in device software development can get in the way of iterative approaches.

 

In this 60-minute web seminar, we will explore Agile and Test-Driven-Development (TDD) concepts. They will review the benefits and challenges of iterative methods and provide advice on new tools and techniques that can help you win with Agile the embedded world

 

Tuesday, July 20, 2010 2:00 pm
Eastern Daylight Time (New York, GMT-04:00)

 

Click here to Register. If you miss the Webinar, register anyway and we’ll send you a link to the replay.

Industry Investing in Better Device Runtime Visibility During Testing

Here’s the final installment in my series about our embedded device software industry testing survey conducted in April-May 2010 with almost 900 respondents (see previous blog postings).  If you’d like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.comand I will send it to you.

In this section of the survey we asked participants about what test tools they use today and where they are investing in test automation. Given the high cost of product failure, accelerating complexity and reduced schedules the industry is turning to more test automation in 2010 to help address these problems. The top investment are moving to new tools that can help test teams and their management better understand how well they are testing, better focus their efforts on the areas needing testing, and reduce cycle time through more automation.

Despite Existing Tools, More Automation Required

Participants were asked various questions about their present software quality testing environment and their plans and priorities for new investments to improve their testing operations in the coming year. Test automation remains top of mind for respondents with 68% of those who knew saying they plan to automate more of their test processes this year.

As for the tools currently in use in their testing processes today, respondents not surprisingly reported that they use a wide variety of systems and techniques.  Results did show that manual testing, conducted by 81.7% of respondents, remains a substantial part of the device test processes at most organizations today.  Other well-represented categories included defect tracking systems, static or dynamic code analysis, and performance and timing measurement. There was a strong showing by simulators, which were reported as being used by over 50% of survey participants. This category includes both device simulation environments as well as traffic, load, and other external simulators that drive device behavior during the test process.

The Top 10 List for 2010

Regarding their plans for new investments, there is a clear need for more automated testing among respondents. The top 10 rank order of tools for planned investment reported was;

1)            Automated test equipment

2)            Code and test coverage analysis

3)            Simulators

4)            Runtime diagnostics tools

5)            Static or dynamic code analysis

6)            Performance and timing measurement

7)            Test execution engines

8)            Script authoring tools

9)            Test script capture/replay tools

10)          Fault injection & runtime behavior monitoring

Runtime Visibility of Device Under Test Needed

Notably, in the top 10, at least 5 of these tools are focused on helping testers understand more effectively what is happening in their code at runtime. Respondents indicated that purchasing tools for these runtime analytics is a higher priority that obtaining more traditional tools for manual testing, defect tracking or lifecycle quality management.

The survey also asked an open-ended question about other trends or investments that should be considered. A significant number of respondents expressed the need for better host-based testing and device simulation so to streamline testing early in the process before hardware available and to reduce the cost of testing assets.

“The biggest problem we have is usually not able to test for a new ASIC before the ASIC is taped out…. It would be nice if there are tools that could be used to develop production code alongside the simulator that we can use for testing before an ASIC arrive, and run in the high level system.”

“There is a strong need for host based "device dependent code" testing suite or methodology.”

“Need better simulations of embedded devices as real hardware can be expensive.”

Conclusions

I hope this series was interesting for you. We found it ratified some of our anecdotal evidence while also surfacing additional areas where we will need to focus our investments and services. It does show that industry trends and market forces have pushed the requirements for software quality testing well beyond the capabilities of the tools and processes currently in use.  The problem is getting worse with more code crammed into devices, architectural complexity off the charts, and squeezed development and test cycles. This is the new normal.

This disparity is causing serious operational problems for embedded products companies, negatively impacting not only their brands but also their bottom-line results. Ignoring the problem is no longer an option. Leaders are seeking new test automation tools that can help them with the unique problems of embedded device testing while helping them deliver on-time, on-budget, and on-quality.

One respondent summarized it best:

“We spend too little time testing and too much time hoping.”  

It’s time to take action.

The High Cost of Poor Quality – Brand, Market, Budget

I’m continuing my series on our embedded device industry software testing survey conducted in April-May 2010 with almost 900 respondents (see previous blog posting).  If you’d like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.comand I will send it to you.

In this section of the survey we asked participants about how they measure the high cost of poor quality. Respondents told us that the true cost of poor quality is much higher than program budget. The majority of respondents showed that the true cost of poor quality is measure by damage to company brand and lost revenue due to missed market windows.

Despite these risks, participants are suffering from repeated schedule slips due to late-cycle quality surprises, though a significant number make the decision to ship defective product anyway an fix it later. When they do ship defects, companies are finding the cause is not due to standard features that they tested for, but instead from unanticipated uses of the product, which naturally were not tested.

Substantial Budget Devoted to Testing, Yet Late Programs

Participants were asked about how much of their total project total resources are spent on testing. More than 77% of the respondents reported that testing activities typically consume over 20% of their total project budgets.  Nearly 21% reported testing costs totaling over 40% or their total project budgets.

Participants were asked what percentage of their overall test budget is allocated for testing maintenance releases and software upgrades.  A large majority (74.8%) said that they dedicate more than 10% to this lower profile but still evidently important testing activity.

While 20-40% of project budget for testing is significant, the responses subsequent questions point out that this hard cost of the testing department resources is only a small part of the cost of quality – or rather the cost of poor quality.

Participants were asked about how poor quality was effecting their development cycle and how they responded to and measured this effect. Seventy two percent (72%) of respondents reported that over the last year their projects had been disrupted due to late-cycle discovery of critical software quality problems. The frequency of these disruptions was surprisingly high, with most respondents that had problems reporting that this happened more than once per project in the last year.

As to the question of how they handled these issues, 57.6% of those that experienced disruptions said that they delayed their products’ ship dates in order to fix the problems and retest the products.

Unanticipated Usage the Most Common Cause of Defects

When they did ship products with software defects that were discovered later by customers, it was most often the unusual and unanticipated use that exposed the problems:

1)            Products used in unanticipated ways (58.1%)

2)            Untested edge conditions (48.9%)  

When asked how their companies measured the cost to the company when products were not delivered at planned quality, the top three measurements were significant business drivers:

1)            Higher than expected program costs

2)            Damage to brand due to shipped defects

3)            Missed market window for the product

Next time I will review respondents’ feedback on what test tools they are using today and where they are investing for new test automation.

Inadequate Management Visibility into Quality is Eroding Confidence

Here’s the next installment in my series on our embedded device software industry testing survey conducted in April-May 2010 with almost 900 respondents (see previous blog postings).  If you’d like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.comand I will send it to you.

In this section of the survey we asked participants about how they measure software quality today, the metrics most often cited by survey respondents were reactive in nature such as tracking customer-reported failures and open defects rather than metrics that can help them prevent defects.

Survey participants clearly indicated that they have a strong desire to be more proactive, but their responses also reveal a significant disparity between their intentions and their actual testing. The survey uncovered a significant disparity between testing goals and reality (as measure by test coverage analysis), where a minority of respondents actually have access to the information they need to assess quality. Respondents further report that often the software quality information they do receive is later proved to be wrong, leaving them to make product readiness decisions on information that turned out to be inaccurate.

Respondents seem to know that you can’t fix what you can’t measure.  They also know that the answer lies in more thorough testing done more often. But the common metrics in use, especially the customer-reported problems, appear to count how many “horses have already left the barn.”  The disparity between testing goals and actual results, the management blind spots regarding software quality levels, and the decision making based on bad information are all contributing to a crisis in confidence in software quality in the embedded industry. 

Metrics Not Providing Management Confidence

When asked if their exiting tools give management sufficient software quality information to make release readiness decisions with confidence, 75% of respondents said that they were at best only “somewhat confident.”  With 19.1% of respondents, their uncomfortable answer was flatly “No, not confident.” 

It seems that a high percentage of participants have learned to be skeptical about the software quality information upon which they must make decisions.  Of those that responded that they knew one way or the other, 57% said that they have made release readiness decisions based on inaccurate or insufficient software quality information. 

One telling group of data points is the responses to a question about which metrics respondents use to measure the software quality in their products.  By far, the two most frequent answers were:

- The tracking of the number of open defects

- The numbers of defects reported by customers

Notably, both of these measures are reactive; they essentially gauge the number of defects already introduced into their products.  The other, less-used metrics include test coverage metrics, numbers of tests executed, etc. were more proactive measures that can be controlled and managed by the companies. 

Inadequate Test Coverage Information

Their lack of confidence and reactive management approach can also be linked to the low use of test coverage and code coverage tools. These tools provide insight into whether software was actually tested (or not) and how thorough test suites are. They should be used in both unit testing at the developers desktop as well as in the functional system testing in the QA lab.  

Overall only a minority of participants much use of code coverage and test coverage tools. This is evidenced in respondents’ answers to the question as to whether they track which parts of their code is actually exercised by their tests.  Over 60% said they either don’t track test coverage at all or only do so during development unit testing. Only about 32% said they measure and track what code is exercised in system tests of the fully integrated device. 

Perhaps the most telling response in this section of the survey was in response to questions about test coverage goals and actual test coverage results with their current projects.  Nearly 65% of participants reported having goals for test coverage of their code bases of over 70%.  But only 35% of respondents believe they are at greater than 70% code coverage with the project they are currently working on. 

Next time I will cover respondents’ feedback on the high cost of poor quality and how they measure this.

Compressed Schedules Driving Shorter Testing & Defect Resolution Requirements

Today I’m continuing my series on our embedded device software industry testing survey conducted in April-May 2010 with almost 900 respondents (see previous blog posting).  If you’d like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.comand I will send it to you.

In part 2 of the survey we asked about schedule compression and what affect that was having on the device testing cycle. A majority of survey participants reported that market conditions have forced them to shorten their development schedules by as much as 18 months.

Many companies are turning to iterative methods to improve feedback within their product cycles. Some are using Agile techniques. However, most respondents are still heavily using manual techniques for system testing and especially for more complex testing of error conditions and performance validation. Defect isolation and repair cycles are typically measured in days per bug. Together these bottlenecks are increasing the need for new approaches to test automation.

Shorter Release and Build Cycles

A majority (64%) of respondents indicated that market conditions have forced them to shorten their development schedules. In terms of the length of their development cycles, a majority of respondents (57%) reported current cycle times of between six and eighteen months.  Of those, 44% reported having to cut their standard development cycles by six to twelve months. 

The survey also asked about the adoption of new Agile development methodologies. Of the respondents who knew, the majority of the participants (64%) reported that they have not adopted Agile methods.  However, 57% reported that they are running bi-weekly build cycles or less with 16% of these actually doing nightly builds.

So despite most teams not considering themselves formally Agile, most teams are using iterative processes.  Surprisingly, 18% indicated that they don’t operate on a formal, fixed schedule for their build cycles. 

Difficult to Evaluate Device Behavior at Runtime

A high majority of participants (over 81%) reporting that their teams test each new software build on the embedded device itself.  This shows that respondents recognize the importance of testing software on the integrated device, rather than just conducting unit tests. Surprisingly, 81% of respondents cited manual testing tools as part of their testing process today, so presumably, manual testing within this iterative environment is becoming a challeng.

Regarding the time it takes their teams to isolate and repair defects found during system integration, more than 62% reported that it takes them a full day or longer.  Over 25% said it takes them more than 2 days. This is clearly a bottleneck to the process and an expense that needs to be minimized.

The survey also asked how users were testing failure modes, error conditions and performance. Fifty seven percent (57%) of respondents said they were using manual methods to verify failure modes and 8.3% said they did not verify these conditions at all. Forty one percent (41%) of those who knew said their tests were not able to measure performance of their devices at runtime.

Numerous participants also commented on the difficulty of understand precisely what is happening within their device under test at runtime.

 “Multi-core products are creating performance challenges for the architects and more corner cases which are difficult to test.”

“Software must be tested thoroughly  under all possible operating conditions.”

“(It’s) difficult to "see inside" linked programs.”

In a somewhat surprising finding about their test environments, more than half of all participants (56.9%) reported that their software development and testing processes are not governed by a safety, security or other certification standards. Of the participants who responded in the affirmative, adherence was fragmented across many different standards.

Next time I’ll review respondents’ feedback about quality measurement and how lack of quality visibility is eroding management confidence.

 

A Crisis of Complexity – Industry Report on Growing Challenges in Embedded Testing

I’ve been talking a lot with embedded device companies around the world over the last few years and I am hearing growing concerns about software testing. I’ve mentioned several of these concerns in previous blogs. I wanted to get more quantifiable data and get some feedback that could help us shape our products and services to help. So I decided to run a survey to gather important data from our community.

The focus of this survey was to gain a detailed snapshot of how executives, development managers, QA and test team leaders and other involved staff currently view the embedded device software quality test landscape.  Recent changes, new challenges and strategies for managing them were of particular interest.  So I fielded a four-part survey to individuals who work for embedded products companies. In total, nearly 35,000 individuals in North America were invited to participate in the survey via emails. Participation was also promoted on the Wind River Website and on executive blog.  The survey was conducted between April 2nd and May 13th, 2010 using an online survey tool. 

Huge Response
Amazingly, I received nearly 900 responses (889 total) from folks in the industry. While 47.8% of respondents identified themselves as software developers, another 46.4% of participants have roles in management and test (including executives, directors, managers, project leads, quality officers, etc).  The sizes of companies from which responses were received were diverse and reflective of the industry in general. Respondents split with about 50% coming from companies with 1500 or less employees and 50% coming from companies with larger than 1500 people. About 41% work at companies with more than 5000 employees. A total of 54.1% of participants identified themselves as Wind River customers; over 40% of respondents said they were not customers

In this blog and 4 subsequent postings I will review some of the highlights of the summary report I created from the aggregate results. If you’d like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.comand I will send it to you.

Key Takeaway

Overall the survey shows that embedded device market dynamics, including skyrocketing software content and new architectural complexity, are challenging development and test teams in every industry segment. Quality Assurance groups are faced with increasingly short delivery schedules, despite having significantly more work to do. Management does not have the quality visibility they need to make confident release readiness decisions. The penalty for late-cycle quality surprises and shipped defects is extremely high. Companies across all segments are investing in new test tools with an emphasis on more automation, better measurement and new runtime analytics technologies that can help them cycle faster, and improve the thoroughness of testing for their complex embedded software.

Product Complexity is Driving Significant New Testing Challenges

Software not only makes up a huge portion of the device features, most participants reported huge growth rates of software content. More than 54% of participants indicated that the software they are incorporating in their devices increases between 25% and 100% annually. 

Ninety six percent (96%) of respondents also validated that their devices are no longer ‘fixed function’ devices but instead are platforms for which they are required to periodically upgrade the software with new features and updates. Forty seven percent (47%) report that they upgrade the software in fielded devices on an “as needed” basis while 48% update on a regular basis ranging from less than 6 months to more than 18 month cycles.

Participants identified several key issues as roadblocks to on-time delivery of software. The top 5 operational factors ranked by respondents as the biggest challenges to their ability to deliver software on time were:

1. Changing requirements
2. Becoming familiar with new technology (processors, operating systems, etc)
3. Integrating and debugging devices
4. Complex system design
5. Time allotted for testing

 

The survey also asked about top challenges affecting their software testing processes. Participants identified a range of issues with the 4 most frequently cited as:

1. The amount and complexity of code they are required to test
2. Complexity of system architecture
3. Insufficient test automation
4. Moving to new operating systems

 

Insight into why these trends are challenging can be gained from participants’ responses to a question about which architectural trends are contributing to testing complexity.   Not surprisingly, changing operating systems was the top-rated architectural challenge followed closely by transitions to multi-core processors. 

Other often-cited challenges included the use of advanced user interfaces, advanced networking functions and the switch from single to multiple processors.  Although it was not ranked in the top five, the use of virtualization technologies, cited by 23.9% of respondents, is clearly a growing challenge in the software quality testing world.  

Over 80% of the respondents felt that testing on the embedded device itself is critical and they stated that they actually test each new build on the device.  But many respondents provide comments about how challenging embedded device testing is.

 In my next blogs I will cover additional findings from the survey relating to ever-tighter release cycles, lack of visiblity and confidence in quality, and new directions in tools and test automation.