Plus you'll probably see paging/swapping happen in programs well before hitting 100% physical memory utilization.
Only if you configure swap that is. Linux doesn't do it unless you first give it the space to do it in (either a partition or a swapfile).
I run 2GB on my netbook and 2GB on my new build server (which is a second hand desktop, so tops out at 4GB if I wanted to upgrade it). I've not built anything especially big on it yet, I guess - the biggest thing was probably the whole FLIF tools at the same time. But I also built that back in the day on the previous machine in that spot, which was at the time maxxed out with 512MB, and I don't think I turned swap on for that. It was rather slow, but it did eventually complete.
Mind you, I also had to recompile the kernel back when I first tried it on my ARM 6 system, and that at the time had 16MB of RAM. Mind you it also had no net connection or working graphics, so after rebuilding the kernel to make things work, I couldn't figure out what to do with it, so I switched back to the inbuilt microcomputer OS quite quickly. That would probably have been Linux 1.something rather than the behemoth it is these days (which thankfully, I've never needed to recompile thanks to it being a matter of plonking the odd .ko file in the right spot today.
But still, I rarely use swap and I've never managed to make linux fall over when building stuff. It's a bit like magic, but when you think about it, source code is just a type of text, and text is tiny when compared to pictures, or music, or videos. That said, I've also transcoded a DVD on 2GB and that worked without swap, but I can imagine more easily how that can be broken down into components, although to be fair it's going to have to be very clever to compress against previous frames when it can't actually store more than a couple at once. More RAM makes the CPU work more efficiently because it can cache more stuff, or enable busy programs to do more stuff before having to do output, but I've been living with 2GB or less all my life, and even when I could upgrade my build machine to 4GB for under 20 quid I've just never felt the need yet. The original 1GB on this netbook was crimping my style somewhat mind you.