RetailOS
The stock operating system running on non-iOS iPods. It runs everything from device drivers to the clickwheel user interface.
Naming
There doesn't seem to be a well-known public name for the iPod operating system. The earlier stages of the boot chain call it osos, but that might just be os repeated twice to fill the 4 characters required for payload names.
The iPodLinux folks call it RetailOS.
Architecture
OSOS is a small, embedded, single-user, single-binary, real time operating system. With time it acquire more and more complex functionality, like PowerVR drivers and being able to load external applications ('eApps') which are used for games.
The core of the system is based on RTXC 3.2, with the end-user interface based on intellectual property from a company called Pixo. [1]
Security
As evidenced by the success of the Notes vulnerability, at least up to Nano 4G there was no kind of security hardening, and in fact all processes, including games, seem to be running in ARM system mode. This should make exploitation of newer OSOS bugs trivial.
Boot chain
OSOS is loaded by the second-stage bootloader (stored on NOR/NAND depending on the device generation), from NAND into DRAM.
While other stages of the boot chain (eg. the bootloader, WTF mode, the diagnostics tool) are based around EFI firmware volumes and an EFI runtime, OSOS is a single binary blob without any built-in modularity.
eApp Signing
Not yet documented fully. Each game seems to ship with a Manifest.plist.p7p which is a PKCS#7 signature for the main Manifest.plist.
Options
We have found some 'secret' options that can be set by creating specially named files. See Options.