Recently, I’ve been experiencing a performance problem with my SheevaPlug (Marvell Kirkwood-based Linux device, which I use as a NAS). After starting up the plug, I was able to transfer files (using AFP from a Mac) at something like 20-21 MB/s (over GBit ethernet), but after a few days of uptime the transfer speed was much lower. When looking at the transfer speed graph in OS X’s ‘Activity Monitor’, I could see the transfer speed hitting 21MB/s, but then almost completely go down. Looking at the dmesg output on the plug itself, I saw a lot of error messages, all starting with:
afpd: page allocation failure. order:2, mode:0×20
I have experienced this problem with the 2.6.30-1 kernel under Debian, but it might also occur with other distributions and kernels. The exception backtrace that follows the error message always includes a kernel function called mv643xx_eth_poll (mv643xx is the ethernet chip on the Sheeva) and rxq_refill; this means that in these routines, the kernel is trying to allocate memory to hold the incoming packets, but for some reason, it fails (and then tries to swap out memory or evict buffers). The kernel simply is unable to put all the incoming stuff into memory when using gigabit speeds. Looking at vmstat, you can see that the kernel will almost never have a lot of memory free; instead, free memory is used for various caches and is automatically freed when needed. For some reason, the kernel cannot free the cached memory pages fast enough when receiving gigabit data.
Luckily, it is possible to tell the kernel to always keep a certain amount of memory free, using the vm.min_kbytes_free sysctl. On my installation, this value was something like 2000. After changing it to 20000 (~20MB), the problem went away and I can now transfer files with a sustained speed of 21MB/s (at least). This seems to be close to the 300MBit/s that is rumored to be the actual speed limit of the ethernet on the SheevaPlug (but since I’m using it with a USB hard-disk, the transfer speed is also limited by the USB bus at 270MBit/s, excluding overhead). To see whether changing the sysctl has any effect, run the command sudo sysctl -w vm.min_kbytes_free=20248. If it solves the problem, you can make the setting persistent (i.e. set each time you boot) by adding it to /etc/sysctl.conf. You can read the current value of the setting from /proc/sys/vm/min_kbytes_free.