iPhone Vulnerability: Return of the Lock Screen Bypass

Reports yesterday of a lock screen bypass in the iPhone 5 noted that a "similar" bug was found in iOS 4.1 and fixed in 4.2. In both cases, the lock screen, which is only supposed to let you make emergency calls or enter the lock code, allows the user to perform other functions, like make other phone calls. How do these errors resurface after being fixed? In Apple's case, the problem could be a weakness in their test plans or procedures.

When an error that was fixed shows up again later it is called a regression error. Regression errors generally are when some change to the program, a new version or software patch, breaks some feature of the program. Security fixes are one type of feature that could be broken.

Controlling regression errors is a matter of proper documentation and testing. Good code documentation should at least give future developers the chance to recognize that changes will affect the feature. But it's testing that is the key to preventing regressions.

Any well-designed software project has a formal test plan as part of it. As new features and bug fixes are added, test should also be added to the test plan to make sure that new fixes don't break old features or fixes. In the case of security patches, a test needs to be added to the plan to check for each vulnerability that is fixed.

The real key to making regression testing practical is to automate it. Back around 2007 and 2008, Mozilla had a very bad problem with security patches causing regressions of other security patches. They finally got it under control and attributed their success, in part, to increased automated testing.

Almost any test can be automated, even by simulating user interface actions by hardware through the USB connection to the device. But the lock screen on iOS is a problem for test automation. The lock screen is designed not to allow external hardware to break out of it, lest someone else take your phone and gain control of it. There's no automated way to test it, so you have to test it manually.

In all likelihood, Apple has some manual tests to perform as well, but it's easy to see how they would get shrugged off in a hurry or given to some intern who didn't execute them properly. Expect an angry memo to go around at Apple about this, but deadlines are deadlines and one day the manual testing will again seem like a corner worth cutting.

Thanks to super-resaerchers Charlie Miller and Dan Kaminsky for discussing this with me.