Hi, I'm a newbie too!
You need to learn C or C++ (I'm told its much easier to learn C first, then C++ isn't much of a leap from it)
Decent guides found here:
http://www.cplusplus.com/doc/tutorial/
Also, check out the "Tutorials" forum on this site. It looks like the thread is empty, but there's an option that says "show oposts from the last 30 days" or something (A drop-down menu on the page). Change this to show all posts ("from the beginning") and you'll see some tutorials for MrMirko's SDK. Read through the thread of the first tutorial and you'll see that someone has posted some other good links for learning C.
Probably easier if you start the basics of C programming for PC rather than leaping straight into GP32. I got an IDE (effectively a posh text editor, where you type in your code) from www.bloodshed.net (or a similar address - google for Bloodshed IDE)
Bloodshed will give you an interface where you can type in your code, press a button and it will be compiled, linked and assembled for you - less hassle than using gcc (the most popular compiler) on its own.
When you want to move to programming to GP32, you'll need to choose an SDK. This is - as I understand it - basically a set of specialised commands that allow you to interact with the GP32 hardware by checking for button presses, displaying on the screen etc. Offhand I can think of 3 SDKs:
Official (produced by Gamepark)
DevkitAdvance - originally produced for Gameboy Advance, but a modded version is avaiable compatable for GP32
Mr.Mirko's - An SDK currently being written (not entirely coplete but rapidly getting there) specifically for GP32. I picked this one simply because the tutorials I found on this site were very clear, and Mr.Mirko himself is on this ste a lot and very helpful!
Compiling the code specifically for GP32 is a little more complex than coding for the console in Bloodshed IDE - You need to learn about makefiles (an excellent guide here:
http://www.eng.hawaii.edu/Tutor/Make/, also check out the guide buried somewhere in Mr.Mirko's SDK help documents, available from his homepage
http://www.mirkoroller.de/gp32).
It should become much easier in the very near future though (couple of weeks?) when Alessndro publishes the new version of his own IDE - like bloodshed, you'll be able to type code in, use some handy tools and just click a button to compile using whichever SDK you choose!
In my experience, there's always lots of help to be had from the coders on this board. And, better still, they are much more concise than me!
An epic post (bad habit of mine!), but I think that covers everything you need to know. I've assumed you're a total newbie, so apologies if you're some sort of l33t god!
Have fun,
Melville
EDIT: Windows XP is fine