You can indeed download and install Free Pascal for ARM-Linux. Nothing special required, just install and it will work, your compiled binaries also will work right away on the GP2x. If you have any questions, visit the Pascal Game Development forums,
http://www.pascalgamedevelopment.com, many people there who have experience with coding games for the GP2X.
GNU-Pascal is not recommended, since the project shows few signs of life and more important, nobody is doing it. I.e. you are fully on your own: you have to patch your GCC with GPC patches, compile it yourself, hope this will work fine on ARM Linux, then try JEDI:SDL, hope it works with GNU-Pascal etc.
Skip using GNU-Pascal...it's intrinsically a dead project. If you're interested in the FOSS equivalent to Delphi, the command-line tool is FPC (Free Pascal Compiler) from the FPC project, the GUI builder is Lazarus from the Lazarus project that uses FPC... If memory serves, JEDI:SDL works with FPC and it's what the eternally stalled Siege of Avalon Open Soure (soaos...) project used/uses for the codebase that Digital Tome provided when the opened up the Siege of Avalon engine code.