The secret second operating system that could make every mobile phone insecure
When we talk about computers, PCs, smartphones, cars, we generally assume that there's just one operating system: A single, monolithic piece of software that manages each individual piece of hardware, from the CPU to the USB controller to the wireless connectivity. If your system crashes, we nearly always blame the operating system, or perhaps a software driver that links into the OS. In just about every computer science textbook, a computer consists of just three blocks: the hardware, the operating system, and then all the (user-land) software that runs on the OS. In actual fact, unbeknownst to the user, almost every computer has multiple operating systems running at the same time, managing various different parts of the computer, and worryingly, these OSes are usually proprietary, closed-source, bug-ridden, and have extensive, low-level access to your data.
Take your smartphone, for instance. There's definitely a primary OS, Android, iOS, Windows Phone, but there's also at least two other operating systems: The baseband OS and the SIM card OS, both of which run on their own processor that's separate from the SoC. Whenever your phone is powered up, these operating systems are running, managing their respective domains. When you send a text or receive a call (or do anything wireless), it's actually passed off to the baseband OS, which handles all of the messy details of dealing with GSM, UMTS, HSPDA, LTE, etc. When you need some secure data from your phone's SIM, Java Card, which has exclusive access to encrypted data on the SIM, takes over.
A reverse-engineered SIM card. Believe it or not, there's a small processor in there that runs a small operating system.
At first blush, this seems like quite a sensible way to do things, after all, there are so many different hardware devices in a modern computer that it would be very hard for someone like Google or Apple to create optimal solutions for all of the permutations. It also makes sense from an efficiency perspective, to have multiple computers that are working in parallel, it's good that the baseband OS and processor can manage the sending and receiving of cellular data, leaving the SoC free to work on more computationally expensive tasks, such as rendering small sprites of candy and playing obnoxiously loud music on the subway. In theory, this kind of encapsulation can also be more secure, there's no reason that users should have direct access to the baseband or SIM card, and such encapsulation should prevent hackers/malware from gaining access to those secure regions as well.
In practice, though, these operating systems are often implemented in a very messy and insecure fashion. For a start, it's important to note that almost all of these secondary OSes are proprietary closed-source software, developed in-house by the hardware maker (Qualcomm, Broadcom, Realtek, etc.) Because developers don't really have a legitimate reason to directly access these devices, there is usually very little public documentation on how these OSes work. Furthermore, according to Thom Holwerda at OS News, these OSes are often outdated and full of insecure, legacy functionality. All of these factors combine to make these secondary OSes very, very insecure.
We'd like to give you an example of a secondary operating system, but the fact that there's almost no public documentation makes it very hard. The best example we can come up with is Qualcomm's REX OS, which powered the company's baseband processors since 1999. There are indications that Qualcomm finally stopped using REX OS in 2012, but it's hard to come by any definitive data. REX OS is a real-time operating system (RTOS) that runs on the Qualcomm baseband processor (in this case an ARMv5 core). The baseband processor (and thus REX OS) has direct access to the phone's hardware (speakers, microphones), and also seemingly the ability to write to the same memory as the SoC (or application processor). REX OS is based on a very old and large code base from the '90s, and implements many standards from the '80s (including the Hayes dial-up modem command set). We only know this information, incidentally, because a hacker reverse engineered REX OS.
A Sprint cell site in San Francisco: Backhaul (right), backup power (mid), base station hardware (left). Your phone treats everything that's sent by the base station as law. God help us if someone works out how to hack a base station, or sets up their own.
As you can probably guess, a large, closed-source codebase that implements old standards is a really bad idea. Holwerda says that REX OS automatically executes any commands that it receives over-the-air from the carrier's base station, and yes, in case you were wondering, there are commands that perform a variety of heinous acts, such as turning on auto-answer, executing arbitrary code, or simply bricking your device. It goes without saying that, in theory, you could set up your own base station (say, with software-defined radio) and cause a lot of havoc. When you hear that an iPhone has been jailbroken due to a baseband exploit, it means that hackers have found a bug in the baseband OS or processor that gives them elevated access to the rest of the device.
This is just one example of a secondary OS. As I previously mentioned, your SIM also has a small processor that runs a tiny kernel that can execute Java software. (The SIM card and its OS was recently hacked, incidentally.) If your computer has some kind of secure storage area, such as ARM's TrustZone, there's probably another separate OS and processor in there, too. The minuscule size of wimpy ARM cores and lack of documentation means that it's very hard to tell just how many discrete OSes are running on your computer concurrently. In classic pre-internet, security-through-obscurity style, we won't know how secure these OSes are until they're (publicly) hacked. If the NSA wanted to deploy a wide-scale hack that gives it access to everyone's phone calls, the baseband would be the place to do it.
The only real solution to this problem is to move away from closed-source hardware and software. There is some indication that some baseband processors are moving to open solutions, such as OKL4, but again it's very hard to come by any definitive data, because commercial companies like Qualcomm aren't in the habit of broadcasting the internal workings of their chips. For the time being, then, just be aware that most of your devices are running multiple operating systems, some of which are probably very insecure, and there isn't anything you can do about it.