The FAT filesystem doesn't have the notion of permissions, so anything on a FAT partition has full permissions for everyone (read, write, execute).
Unix-like filesystems (like ext2/3/4) do have permissions. Every file and directory has an owner and a group, and you can set read/write/execute permissions for the owner, the group, and the world (everyone).
In a standard unix filesystem, all "system" files and directories are owned by root, e.g. all files in /bin, /usr, /lib, and so on. The permissions for most of these files are set such that everyone can read the files and directories, everyone can execute the programs, but only root can write stuff. The idea is that the computer may be used by many different users, and you don't want the users to destroy the system. The only place were users can write is in /tmp and in their home folder (/home/username). Depending on how you set permissions, you can allow other users to see or not see, change or not change files and directories that you own.
See
http://en.wikipedia.org/wiki/File_permission#Traditional_Unix_permissions for more information.
On a Pandora, you typically have only one user. It still makes sense to distinguish between root and the user, because that way you can be sure that you don't accidentally screw up things as long as you're logged in as the user and not as root. Of course you can use sudo or similar commands to temporarily become root if you need to.
If you have an ext2/3/4 partition that you're just using for data, it's of course not a problem to make the entire partition owned by the user so that you don't need root to write there. If you're booting from an SD card, that means that all your system files are on the SD card so you want to keep them owned by root. You can put stuff in your home directory, or if you want to have a shorter path, you can create a directory in the root and make it owned by you:
sudo mkdir /mystuff
sudo chown <your_user_name> /mystuff
If you want to store PNDs on the same partition as the one you're booting from, you can create a "pandora" directory and make it owned by yourself in the same way.