GP2X Python Distutils And Devkitgp2x?

[Kitsu]

I haven't run out of stuff to try, and I don't need to compile anything ASAP, But it would be nice to get working.

distutils is a python system for automatically compiling CPython modules (among other things).
devkitGP2X is the all in one windows cross compiler (or it looks like it has everything...)

I'm going to try to mangle the CygwinCCompiler module into something that uses the tools in the devkit, but I don't know/use C. If anyone has any experience with distutils, or thinks they can get it working with the devkit, it would be very helpful

The reason I mention it is because I've written a python module in Pyrex which I would like to use on my new GP2x. It is dead-easy to compile on windows using distutils and MinGW so I'd like to use distutils for GP2X too!

 

[Kitsu]

Did a little more Python on my GP2X, and was again annoyed by the slow Python+math.

I've done a tiny bit of research, and it seems I will need to know details about how PythonGP2X was built (i.e. compiler/linker flags). I expect I will have different dependencies than whatever I've got in my windows Python 2.4.x.

So if theoddbot is around could I either get:

• Some referances to stuff you used/what you did.
  A copy what your using to build (source and settings).
  A big tutorial detailing what to do to build Python extensions on Windows that are cross-compiled for GP2X :rolleyes:

I visted your site BTW, and the next version of PythonGP2X sounds very cool. An optimized build and HW accelerated SDL might curb some of my complaints. I don't know anything about the dual procs or how they work, but I've been thinking - if you could do all the FPU emulation/math stuff on the second CPU that would let the first CPU spend all its free cycles on the interpreter. I can imagine a huge number of complications this could involve, but it still sounds nice.

 

[Kitsu]

Argh? I just re-read the Pyrex quick start and the wiki page on how to use devkitGP2X. It looks like "arm-linux-gcc vector.c -o vector.so -fPIC -I./Include -shared" compiles my module with no complaints! I tested it using usb-serial and a shellscript to run Python, it seems to work. My program which uses the module fails though, and one of my properties returns 0.0 every time I call it :rolleyes: At least I have something to work with now.

BTW: Why didn't anyone tell me how ridiculously easy it was to just compile? I compiled a lot of software when I was running Slackware, and I was always impressed/frightened by the size and complexity of the scripts used to make programs. Also I always thought compiling dlls/linked libraries/whatever was even more complicated that making an executable. Meh, the longer I program the closer I get to actually using c

Anyway, for those who are interested, here is what I've done to get my pyrex file compiled:

Cross-Compiling pyrex
Download the all in one devkit from the archive:Archive page
Follow the readme instructions to install it.
I created a src directory in the devkitGP2X dir where I could throw my source.
Download and extract the Python 2.4 source package
I copied the Python Include folder into my src folder, it should probably go into the devkit Include directory.
I also had to copy pyconfig.h.in to the Include folder and truncate the '.in' part.
For the next steps you will need a dos/cmd window open in the same directory as your source files.
I had a c file from a prior build of my extension, but you should be able to use 'pyrexc.py mymodule.pyx'* to make one.
If that succeded do "arm-linux-gcc mymodule.c -o mymodule.so -fPIC -I/path/to/python/Include -shared"
Copy the resultant mymodule.so file where you want it and import it into your program!

* Calling pyrexc.py requires you have "[python_dir]/Scripts" in your path (and pyrex installed...)

There, I think I avoided saying anything too stupid in that post!?

 

[Kitsu]

Awful quite in here...

Anyway I think I have tracked down my mysterious 0.0 return value. It seems sqrt is returning 0.0? I boiled down my code to the simplest code that recreates the error, here is the Pyrex code:

# math.pxd (pyrex declare)
cdef extern from "math.h":
    float sqrt(float) # Tried with 'double sqrt(double)' too

# length.pyx
from math cimport sqrt
...
def length(vec):
	square = squareLength(vec)
	return sqrt(square)

where vec is actually a list of (python) floats. I omited squareLength because I already verified that it is working correctly.

Here are the relevent sections of the generated c file:

#include "math.h"
...
static PyObject *__pyx_f_6length_length(PyObject *__pyx_self, PyObject *__pyx_args, PyObject *__pyx_kwds); /*proto*/
...
/* "C:\devkitGP2X\src\length.pyx":9 */
  __pyx_4 = PyFloat_AsDouble(__pyx_v_square); if (PyErr_Occurred()) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 9; goto __pyx_L1;}
  __pyx_1 = PyFloat_FromDouble(sqrt(__pyx_4)); if (!__pyx_1) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 9; goto __pyx_L1;}
  __pyx_r = __pyx_1;
  __pyx_1 = 0;
  goto __pyx_L0;
...
__pyx_L0:;
  Py_DECREF(__pyx_v_square);
  Py_DECREF(__pyx_v_l);
  return __pyx_r;

Yes it is very ugly, and yes you could write it by hand and make it shorter/faster/cleaner... But you would need to know all of c, and be very familier with the Python c api.

Here is an interpreter session over usbserial:

>>> from length import *
>>> v = [5.0,0.0,0.0]
>>> squareLength(v)
25.0
>>> length(v)
0.0
>>> WTF?

I fought with this thing last night until midnight, then again since eight this morning. I've read everything I could find, searched these forums and anywhere else for similar problems, and written dozens different testing scenarios to figure this out. The only info I've found here is that you should add -lm to make math.h link (which I hadn't had problems with but did anyway). My module works perfectly on windows, and I'd rather not make hugh changes to the way I've written it. So does sqrt(float foo) return sane values in your program? Is there something sneaky I should know/do to get this working? I am thinking about switching to fixed point, but I have the feeling that could be quite difficult to get working with my existing pyrex code. I wouldn't mind playing around with fixed point if some one wanted to point me at a good/simple cross platform fixed point lib (is one in the c std library?).

 

[Kitsu]

Thanks everyone for the brilliant and insightful comments

Had to fix a friends computer today, so all I've got done is some reading. Here are a couple nice high level tutorials about fixed point:
http://www.bookofhook.com/Article/GameDeve...oFixedPoin.html
http://www.gamedev.net/reference/articles/article203.asp
The gamedev one is a little funny, the guy says his home computer is a 386/40... Still good stuff for modern portable devices.

So I think I'll try to impliment a fixed point module in Python to see how it flys. If I can intigrate that into my current code without too much trouble I'll try wrapping the allegro fixed math stuff (unless someone can point me to a better fixed point lib to link to :unsure: )

 

[Kitsu]

Fixed point in Python about half done, multiplication and string representation were fun to get right :rolleyes: . Looks like it will be a pretty straight translation to Pyrex, just add some cdefs and compile! Should be decent fast, with the added bonus that GP2X handles integers correctly...right?

 

[Kitsu]

Finished basic functionality for the Python fixed point module two days ago, so yesterday I tried to get negative numbers working right. Didn't exactly work out of the box. It'll take some time to get that working 100%, so I'll put that off for a while. Tried to port it to Pyrex and it seems to be less trivial than I thought. Might be useful so I'll try to get it working eventually.

Wrote a wrapper module for Allegro's fixed point math, but when I import it in PythonGP2X I get undefined symbols. Not suprising, I don't have an Allegro.so for my module to use. Going to try to compile a shared Allegro lib tonight.

Reviewed the Pyrex docs on using "cdef extern from" and read most of the included header files. Had some trouble finding the definition for the sqrt function. It seems to be in "bits/mathcall.h" and takes a "_Mdouble_" as its arg. Thinking about trying the Python API's float from double function, though I thought Pyrex automatically coerced double > float...

On a whim I changed my definition of sqrt to "int sqrt(int)" and instead of 0.0 I get 0 for all sqrts :lol: My minimal test case is now a module with a sqrt function that returns the result of the c sqrt. The answer is always 0^H^H^H^H^H^H^H^H^H
I just tested "float sqrt(double)" and it says the sqrt of 9 is 4.8304995820157011e-318 :unsure:

I've also tried things with and without the "-msoft-float" flag with no effect.

 

[Kitsu]

:wacko: All I need is a simple result, but to get there I need to know how to do a huge number of compilcated things! I don't know enough to make Allegro into something I can use in a Python module. I think I need to compile my fixed module -shared, but everybody says Allegro needs to be static. I would be happy if I could just make the Allegro fixed and fmath functions part of my module... but I don't know how. So I need to have a shared Allegro library my code can link to at run time right? But I can't build Allegro for the GP2X with the tools I have, and I don't know enough to do it manually! Arrrrrggh

If I had the time and interest I could spend the next six months studying c and maybe get enough knowledge to do this stuff. But I don't care about any of that, I just want decent fast math for my GP2X programs, and a sqrt function that works!

So some time last week I did a lot of research about calculating square roots, and even found an algorithm that made sense to me. So I impimented it in Python - it worked okay! So I put it in a Pyrex file, declared it as a c function -> compile -> run -> Errors! I got some some domain errors when I used it a program, and DivisionByZero errors when I imported it and tried it. So I put it aside and persued some of the other promissing things I'd found.

Well this morning I had effectivly exhausted my other options, so I did some testing on my little sqrt function. It looks like whatever was going wrong was caused by calling it a c function. Changing my cdef to a def and removing my variable delarations made it work! Tested on GP2X, It still worked! Stuck it in my program... And the damn thing finally runs again! It's slow and has lousy accuracy, but it works

Here is the code:

def sqrt(num, precision=0.001):
	"""Newton's method for solving sqrts."""
	if num < 0:  raise ValueError
	elif num == 0: return 0

	guess = 1.0
	old_guess = 0
	while abs(guess**2 - num) > precision:
  old_guess = guess
  guess = ((num/guess)+guess)/2
  if guess == old_guess: # Fixed point number can't always get there
  	break      # So if we make the same guess twice quit!
	return guess

Roughly copied from some website, adapted for my stalled python fixed point module, then re-adapted to the above. I'd be happy to take sugestions on speed improvements, but in trying a number of code samples I've found that if I don't know how it works (explicitly) then there is no point implimenting it - Even if I get some obscure code working I can't debug it when things go wrong.

So throw all the other stuff out for now, I'm going to do some profiling and optimization and see how fast I can go with what's working now. It's too bad nobody reads this thread, I think I've actually put some useful information in here...

 

[rokdcasbah]

i've been reading it, but don't know enough about any of it to offer any useful advice, though. but it's cool that you fixed that nagging problem, that's always a good feeling.

how many people out there are using pyrex? probably more once you finish the port it seems like a better way of combining c and python code in the same program, is that right?

 

[Kitsu]

Hey, thanks! Yeah I actually figured that most of the developers here didn't have any knowledge about Python internals or the rather obscure Pyrex. I don't really know how many people use Pyrex, the mailing list is pretty slow. Then again I haven't joined the mailing list... I know of several projects either coded in Pyrex or using it to link to c code.

Pyrex is a languge like Python, except it is 'compiled' to c. It basically lets you write code without worrying about all the Python-c api stuff. It is mainly used to write/wrap modules for use in Python, but it can also be used to create stand alone executables.

The main reason I wanted to use it was speed. My pure Python vector math module ran my all my tests in ~0.24 msecs, my Pyrex module compiled with fast-math -o2 and some other optimizing flags did it in ~0.12 msecs. The code for the two was as identical as possable and my speed nearly doubled! It turned out to still be too slow for what I was doing, but I expected that.

Pyrex does a lot of silly stuff in its generated code. If I knew what I was doing I could write the internal stuff in pure c, then wrap that with Pyrex. Pure c vector math (non-optimized) should be about ten times faster than Python->Pyrex, so Python->Pyrex->c ~= five times faster? I plan to get a book an c some time this year but I've started working on an actual game so my intrest has diminished a bit.

 

[tempest80]

Hi Kitsu, I started doing some experiments with Pyrex compiling too; I followed your procedure but the compiling step fails on me.

First I process the .pyx file to produce a .c source: the first line of this generated file is

#include "python.h"

python.h contains a line

#include "pyconfig.h"

which, in turn, *tries* to

#include "io.h"

And that's where the compiling fails, because in the devkit there is no file called io.h (well, actually there is one, but it's in the include/sys directory).

Did you modify your include environment variable too to get things running? Thanks!

 

[Kitsu]

Hi Kitsu, I started doing some experiments with Pyrex compiling too; I followed your procedure but the compiling step fails on me.

Did you modify your include environment variable too to get things running? Thanks!

Firstly I've moved my source into a src directory in the devkit folder, and moved the Python includes into the devkit/include directory. Next I quickly tired of typing on the command line so I built a batch script:

pyrexc.py vector.pyx
arm-linux-gcc vector.c -o vector.so -fPIC -I C:\devkitGP2X\include\python -I C:\devkitGP2X\include -shared -Wall -lm -msoft-float 
rem -mcpu=arm920 -mtune=arm920t -O2 -fexpensive-optimizations -ffast-math

Also don't forget to follow the instructions that came with the devkit!

You will of course need to replace the names of my module with your own. Or if you feel tricky you could try to build a script for D&D compiling. I was testing the stuff after the rem, but it showed little benificial impact.

P.S. Here's a hint for making useful Python scripts on windows - Add this to the first line of your file @python -x "%~f0" %* & exit /b and change the file extension to .cmd. Now you can drop files onto the script and it will recieve the dropped file name as sys.argv[1]!