Friday, September 26, 2014

8.0.1-gate: A Postmortem

So what the heck happened?

Apple isn't saying, at least not yet, so we can only speculate.  The facts seem to be these:


  • Folks who updated their iPhone 6's and iPhone 6+'s to 8.0.1 lost the ability to connect to the cell networks and the use of Touch ID.
  • This appears to have affected everyone who updated an iPhone 6 or 6+.
  • This appears not to have affected anyone who updated earlier models. (There are some accounts to the contrary, but I'm guessing with every update there are a handful of folks who have issues like these, so let's assume these are outliers).
  • This appears not to have affected anyone who updated via iTunes (as opposed to "over the air").
So how did this happen?  Well, let's guess.

When you update via iTunes, iTunes downloads a ".ipsw" file from Apple's servers which contains the disk image that will be written onto your device.  The .ipsw file contains multiple "RAM disks," which are essentially disk images that correspond to different situations.

For example, every .ipsw has a "restore" image which is what is used when you restore the iOS version onto your device. First your device has its memory wiped, then this restore image is copied into flash.  

Most updates also have an "update" image that is used when you choose "update" from iTunes (instead of "restore").  Some developer versions of .ipsw's have not included the "update" image, but I'm not sure if any "release" versions of iOS have failed to include it.

I'm guessing the build-and-test process for iOS looks something like this:



The operating system is "built" (i.e. compiled and assembled together) into an image.  Apple tests the image. If it fails, back to the drawing board. If not, the image is automatically packaged into various .ipsw's.  For iTunes, there is an .ipsw for each device or device family.  For example, iPhone 6 gets a special .ipsw, and the CDMA version of iPhone 5s gets its own, etc.  Within this .ipsw are separate restore and update images.

One would hope that Apple then tests each restore and update .ipsw on the appropriate device, before uploading them onto its distribution network so that you can download them.  No way to know if they do, but the reason to do it is to catch any errors introduced by the flow that turns the test iOS into the .ipsw.  

Apple must also produce a .ipsw (I assume) for over-the-air distribution.  Again, presumably this is one .ipsw for each device or device family (and presumably one for each device for each prior iOS version, since these usually are differential updates).   This may be created from the iTunes .ipsw, or, as I've shown here, may derive from the tested OS image directly; we don't know.  

It appears, though, that somewhere in this part of the process, in creating the OTA .ipsw for iPhone 6 and 6+, something went wrong.  I don't think the output was tested, either.  If the failure was in transmission, as opposed to creation, then whatever borked the files in transmission would have caused it to be rejected by your iPhone.  This is because iPhone will only install updates that are properly signed by Apple.  If even one bit of the OS is wrong, it should cause a failure in verifying the signature.  That is, one would hope the signature is properly entangled with a hash of the image contents.  

Because I believe our phones received properly signed iOS images, I think something went wrong in creating the image, and the image was never tested.  Creating the image isn't too complicated, but it does involve calculating the differences between the prior image and the new image, and it could have gotten messed up.  For example, parts of the baseband software may simply have been missing or signed wrong (meaning that, for example, that the secure enclave and baseband portions of the chip didn't get installed properly, and knocked out cell connectivity and Touch ID.)

If this is the case, it's indicative of sloppiness on Apple's part and is pretty inexcusable.  Apple should be testing each update and restore image in its final, signed, form on real devices prior to uploading to its content delivery network.  

No comments:

Post a Comment