This would probably decrease the portability, but you could include apt-get wrapper that installs to the SDK enviroinment, so you can remove/install packages from debian armhf repos.
Building through distcc solves many of these problems, which is nice. Since for faster builds you would only...