Safety First

Introduction

From time to time folks see fit to publish lists of what the general priorities for developing a software project should be. For example, here’s one from Jim McKeeth. The funny thing about these lists — and I don’t want to pick on Jim; this seems to be generally true — is that almost all of them omit what I consider to be the number one priority:

The first priority of software development must be to protect the safety of the user and anyone else affected by it.

Who, Me?

This is perfectly obvious to folks writing control software for medical devices, but does it really apply to software development in general? Those who doubt this even for a second should, I think, spend a little time reading the RISKS digest. I’ve followed this list for close to 15 years now, and I find it to be an excellent, periodic reminder of the law of unintended consequences. You may think that your application is not doing something which is critical to the user’s safety. But you have no way of knowing what else the user is running on their computer. Here are just a few ways that seemingly innocuous apps could harm user safety:

• An application could contain a bug which corrupts a file which happens to be the user’s financial database
• Applications which leak confidential data can cause problems either by the disclosure itself or by causing the user to violate laws such as HIPAA.
• Software can contain security flaws which allows a spyware application to take over the computer.
• Software can interfere with the user’s Outlook reminders, or email.

How common are these problems? Certainly not as common as most of the other potential pitfalls on a list of software development priorities. It is not the frequency of safety-related malfunctions which pushes this concern to the top of the list, but the consequences.

What to Do About It

The first step is awareness. It is important to be knowledgable about the ways in which software can harm human safety. I’ve already recommended the RISKS digest. Why not point your favorite RSS aggregator to the RISKS feed? It may not be quite as entertaining as the Daily WTF, but it’s an excellent way to learn from the mistakes of others. On a more positive note, check out the System Safety and Software Safety Research page at MIT, especially the Papers section.

Safety should be part of your requirements documentation and test plan. The requirements should specify both direct and indirect means of protecting the users’ safety. By "direct" I mean requirements for the behavior of the application itself, such as making sure that safety-related functions work, that failures happen quickly and in a controlled manner, and that confidential data is protected. By "indirect" I principally refer to interactions with other applications. For example, our software tests dependent, shared DLLs at startup and fails immediately (i.e., asks the user to re-run the client install) if the wrong version is detected. This helps prevent the software from silently failing (and potentially corrupting data or displaying erroneous results) due to installation of another product.

For especially safety-sensitive projects such as medical devices, airbag control sytems, and security-related software, more sophisticated measures are required. Follow the cleanroom model of writing specifications and then having different teams implement the specifications and compare the implementation to the specifications. Have a security professional audit your design and code, and get a tiger team to attack it.

Above all, make safety your highest priority.

Share This | Email this page to a friend

Server Response from: dnrh1.codegear.com