No, I think adding swap can actually speed up a machine. The idea is that you can swap out areas of memory that aren't actively used in order to free up space for actively running processes.
Since people usually use very light window managers and mini-menu, this does not help that much, but in theory could.
Note: of course this would not normally increase the framerate on a game!
So, currently my pandora is using ~50MB. I start a game. I could, in theory, reclaim those 50MB by swapping them out, if those memory areas are not accessed while playing the game, for example in order to increase disk buffers, or have the game use more memory in general. Both could have the pandora work faster.
That is why people often add swap to devices that don't seem to need them.
All that is in theory.
I just tried to add a swap file to my pandora by doing
# dd if=/dev/zero of=myswap.swp bs=1k count=500000
# mkswap myswap.swp
# sudo swapon myswap.swp
I then tried to play with "spawiness" by doing
# sudo sysctl vm.swappiness=90
But I didn't manage to get my pandora to swap. Partly it is because I don't have programs that are memory hungry enough...