Compiling for the OSWALD using a qemu-user chroot

In addition to cross-compiling, it is also possible to run a ARM-based chroot under QEMU user mode. However, it is recommended to have a relatively decent machine or else the emulated machine will be significantly slower than using the actual hardware. While this can be setup using pretty much any UNIX-based OS that QEMU supports, the instructions below are focused on Gentoo Linux (the documentation that this page is based off can be found in the official Gentoo Embedded Handbook). Other distributions or systems will have to build QEMU directly from source using either the oswald-qemu repository as specified below or the upstream repository.

Building a minimal QEMU with ARM user mode support

There are a few options for building QEMU so it can be used with the chroot. On Gentoo, you can build a static qemu binary package only including support for the ARM user target.

USE="-* static" QEMU_SOFTMMU_TARGETS="" QEMU_USER_TARGETS="arm" emerge -B qemu

For non-Gentoo systems or if you want to use the latest code for ARM support you should checkout the oswald-qemu git repository and build the ARM user target from it using the following commands.

git clone git://code.oregonstate.edu/oswald-qemu
cd oswald-qemu
./configure --target-list=arm-linux-user --static
make

Enabling kernel support for MISC binaries

Next you'll need kernel support for wrapper-driven binary formats. This kernel option can be found in the Executable file formats / Emulations section and can be built-in or a module. Note that if you're building it as a kernel module your machine will still need to be rebooted before continuing.

If you built the support as a module don't forget to load the module after rebooting and also make sure the binfmt_misc handler is mounted.

[ -d /proc/sys/fs/binfmt_misc ] || modprobe binfmt_misc
[ -f /proc/sys/fs/binfmt_misc/register ] || mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc

Next we'll register the format we want to use with the kernel through the procfs. Note that you'll need to run the command as root (or use sudo). If you're interested in registering additional machine types you can find the necessary commands in the Gentoo Embedded Handbook.

echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/qemu-wrapper:' > /proc/sys/fs/binfmt_misc/register

Setup the chroot

First download the latest Gentoo armv7a stage3 release. The following command should always download the latest release from the OSL. If you prefer a different mirror you'll have to find the file manually (Gentoo mirrors list) or alter the URLs in the command.

wget http://gentoo.osuosl.org/releases/arm/autobuilds/`wget -O - http://gentoo.osuosl.org/releases/arm/autobuilds/latest-stage3.txt | grep armv7a`

Once the file has finished downloading, unpack it and enter it's root directory.

Install QEMU in the chroot

Next, QEMU must be installed in the chroot. If you built the qemu-user package, then use the following command; if instead you built the qemu-kvm package, then replace qemu-user with qemu-kvm in the following command.

ROOT=$PWD/ emerge -K qemu-user

Finally, if you built qemu from the oswald-qemu repository you should use the following command:

cp /path/to/oswald-qemu/repo/arm-linux-user/qemu-arm /path/to/chroot/usr/bin

Setup the QEMU wrapper

In addition, we need to pass additional arguments to QEMU (CPU model), so we'll create a wrapper that will call QEMU.

#include <stdio.h>
#include <unistd.h>
#include <string.h>

int main(int argc, char **argv, char **envp) {
        char *newargv[argc + 3];

        newargv[0] = argv[0];    
        newargv[1] = "-cpu";
        newargv[2] = "cortex-a8";
   
        memcpy(&newargv[3], &argv[1], sizeof(*argv) * (argc - 1));    
        newargv[argc + 2] = NULL;
        return execve("/usr/bin/qemu-arm", newargv, envp);
}

Save the above code in the file qemu-wrapper.c and compile it with the following command:

gcc -static qemu-wrapper.c -o qemu-wrapper

and copy the outputted qemu-wrapper executable into the root directory of the chroot.

Mounting the required directories

Now a few directories must be mounted from the host system into the chroot. Note that the following commands assume the current working directory is the root directory of the chroot.

mount -B /proc proc
mount -B /sys sys
mount -R /dev dev

In addition, you may want more directories bound into the chroot from your host (such as your Gentoo ebuild repositories, distfiles, etc) so these can be mounted in a similar fashion.

Entering the chroot

Finally, you can enter the chroot with the following command (again assuming the current working directory is the chroot root directory):

chroot . /bin/bash --login

Now you have a ARM-based Gentoo chroot that you can use to build packages for the OSWALD or whatever else you want to do.