Chinavasion Pico Projector
As I extracted and tried to run various ARM-compiled Debian binaries on the projector, a common issue popped up—either my glibc was too new or too old. The glibc libraries are one of the core libraries for a Linux system that basically every C program needs to run, and the version this projector had seemed to fall somewhere between Debian Etch and Debian Lenny, with binaries from the former not executing and binaries from the latter complaining about a glibc that was a bit too old. It seemed like Lenny was the distribution closest to this one, but I honestly am not sure on what distribution or version this Linux install is based. The libc-2.3.3.so possibly seems to be from some ARM-compiled Fedora distribution. In any case, without glibc support, there was basically no way I could get most of my binaries to run, and I didn't want to risk bricking the device by overwriting the existing libraries, so I had to find a different approach.
My next plan was to track down a small Debian Lenny-based ARM distribution so I could copy the full root filesystem to the 400MB /mnt/mtd partition. Then, I could just chroot into that and run the commands I wanted. That way, all the commands would use that glibc, and I could add extra ARM-compiled Lenny packages. The problem I ran into fairly quickly was that, well, chroot segfaulted. Both the chroot binary that was included on the projector and the version in my Lenny install failed to work.
At this point, I started refreshing my memory on LD_LIBRARY_PATH, LD_PRELOAD and other variables I could use to tell a binary to use the version of the libraries under /mnt/mtd that I had installed. It turned out that got me a lot further. I could launch ls and a number of other console applications, including useful programs like strace and lsof; however, fbgetty was abandoned before Debian Lenny, so I had to try other framebuffer terminal applications in Lenny, such as jfbterm. The application would start, but it never seemed to be able to attach to the tty it wanted, and ultimately, it would error out. After trying a few things, including changing the permissions on /dev/console to be more permissive, I gave up for the moment and turned off the projector.
The next time I turned on the projector, the splash screen started with its scrolling progress bar, but after a minute or two, the progress bar seemed to freeze, and the system was unresponsive. Great, I bricked it. But how? After all, I had just added some files to a basically empty mountpoint and changed permissions on /dev/console. At least, that's all I could remember doing. At this point, I wasn't sure what to do. I didn't want to take apart the device (at least not yet), so I rebooted it a few times and tried to see if I could get any information. One thing I thought about was that the system might have needed all that space in /mnt/mtd that I had filled up or possibly all of my changing of VTs could somehow have been remembered (even though that seemed unlikely).
At this point, I remembered the init script that automatically upgraded the device, so I decided to make a USB key with a test script that would write a file back to the USB drive. If that worked, I could just continuously boot the device with the USB key and build up a script that could repair any damage I may have done. Unfortunately, when I tried this approach, I noticed that the device didn't seem to get very far into the boot process—the script never seemed to run.
Next, I decided to connect a USB keyboard and see if the system even loaded the kernel at all. It turned out that the keyboard did work, as the Caps-Lock key lit and Ctrl-Alt-Del at the right point did reboot the device. Unfortunately, when the system froze, so did the keyboard. At this point, I tried all sorts of keyboard combinations during boot to change the VT back and forth to attempt to Ctrl-C through some script that may have stalled, and through some sort of magic voodoo, whether it was something I hit on the keyboard, or as I've come to suspect, something on the hardware that started working again, finally the system fully booted.
After I breathed a sigh of relief, I realized I needed to make an immediate backup of the filesystem, so I had some record of what was there. I also decided to take off the cover to see if I could find a serial port on the hardware and access the boot prompt and boot messages. If you look at Figure 3, you can see the bottom of the motherboard on the device, and at the top of the picture is a small five-pin white connector labeled Program that I'm assuming is some sort of serial interface. Unfortunately, this connector is incredibly tiny, so I haven't been able to track down a compatible connector yet and test this theory. Figure 4 shows the top of the motherboard—what you would see if you simply unscrewed and lifted off the top of the projector. One thing I noticed when I looked on the board was extra soldering points for an extra USB port and VGA. More-seasoned hardware hackers might find even more interesting things on the board once they take a look.
Unfortunately, after I took it apart, I had the same trouble with a frozen progress bar at boot. After a number of reboots, I finally was somehow able to get it to boot completely, but since then, I've experienced the same issue a few other times. Eventually, the system will boot, but as of yet, I've been unable to track down the source of the problem. At this point, I'm leaning toward some sort of short on the device.
Kyle Rankin is a VP of engineering operations at Final, Inc., the author of a number of books including DevOps Troubleshooting and The Official Ubuntu Server Book, and is a columnist for Linux Journal. Follow him @kylerankin.
Practical books for the most technical people on the planet. Newly available books include:
- Agile Product Development by Ted Schmidt
- Improve Business Processes with an Enterprise Job Scheduler by Mike Diehl
- Finding Your Way: Mapping Your Network to Improve Manageability by Bill Childers
- DIY Commerce Site by Reven Lerner
Plus many more.
- diff -u: What's New in Kernel Development
- Server Hardening
- Giving Silos Their Due
- 22 Years of Linux Journal on One DVD - Now Available
- What's New in 3D Printing, Part III: the Software
- Controversy at the Linux Foundation
- Don't Burn Your Android Yet
- Firefox OS
- February 2016 Issue of Linux Journal