Pandora Distpnd

[Tempel]

Dear everyone,

If you're a Python developer, you're probably aware that Distutils is the standard mechanism for distributing Python software. As long as you have a well-made setup script, you can distribute the sources, install to root or home, create an RPM, or, with the right extensions, package it just about any way else.

distPND is a Distutils extension I've created to build PND and PXML files easily. It provides two commands: gen_pxml will slap together a PXML file based on parameters in the setup script and command-line options, while bdist_pnd will assemble your project into a PND file (calling gen_pxml if it needs a PXML file).

To use it, download the sources from its github page and install like any Python package (sudo python setup.py install). After that, you can use the commands with any other Distutils-based setup script by specifying distpnd in command-packages (ie: "python setup.py --command-packages distpnd [command]", or make a setup.cfg file with "command-packages=distpnd" included under the "global" heading). To make iso- or squash-based PNDs, you'll need to have mkisofs or mksquashfs (respectively) available. Even without those, though, you can make a folder that will be read by libpnd.

So far, it works pretty well, but it's kind of hackish and somewhat limited. I'm hoping my fellow Python devs will take distPND for a ride and abuse the crap out of it. Tell me where it breaks or could behave more nicely and I'll fix it up. Or, if you're feeling generous, send along some patches. Some things to note:

• Almost none of this respects the dry-run flag. It shouldn't make any system changes, but watch out for that.
• gen_pxml does not nearly expose all the functionality in a PXML. Is there an elegant way to include associations, multiple applications, and multilingual titles and descriptions? One solution might be to have PXML templates; can you concoct a good approach to doing that?
• bdist_pnd doesn't yet check that it's not overwriting existing files. Again, watch out.
• If you have an icon or info file to include, bdist_pnd expects that file will make its own way into the build_pnd folder (presumably during the "install" portion of the job). Is there a better way to handle this? I think I've solved this pretty well. If you tell bdist_pnd about your icon and info, it ensures they end up in the root of the PND and in your PXML. I've updated the Lonely Tower sources to show two different ways to do it.

As an example of how it works, I've packed up my opus; you can download the sources to see how setup.py and setup.cfg work, or you can download the resulting PND to see the results (with no Pandora, I can't actually confirm that this works, so please don't laugh at me if it fails!).

Have fun everyone! I look forward to hearing back from you.

PS: If your package includes extension modules written in a compiled language (C or Cython or whatever), they'll need to be built for the Pandora. You could just run the whole build process on the Pandora, but if you can make cross-compiling work through the build command, then it should be no additional effort to make it work with bdist_pnd. You might want to take a look at distutilscross to help with this.
PPS: You might also be interested in stdeb, a Distutils extension to build deb packages. Since debs are identical to ipks, this would allow you to build PNDs and Pandora-installable ipks from the same build script! I find that exciting.

 

[bzar]

I have some simple pygame projects I could try this with. They're not finished/polished, but are playable. I'll be sure to try it out when I get my pandora

 

[Tempel]

According to people over in the Sparks thread, it looks like distPND isn't producing valid files yet! Without a Pandora of my own to test on, I'm making one last-ditch effort to fix it; if it fails, I'll just have to wait for mine to arrive.

I noticed that the PXML archetype includes an "encoding" attribute that my generated PXMLs did not. But I've fixed that and have updated the PNDs for Sparks and The Lonely Tower. Sparks is made with an isofs, while Lonely Tower is done with a squashfs, just so I can make sure fs-type isn't the culprit.

Pandora owners, please try them out; let me know if they get detected and run. There will be many thanks if you try!

 

[Tempel]

Thanks to much testing from Gruso, and vminko's VNC-accessible Pandora, I've managed to get enough testing to work out distPND's kinks. If you're curious, the problem stemmed from the fact that Angstrom does not include a python2 symlink; since my Arch box has the python symlink pointed at python3, the Distutils shebang rewriting magic was trying to make the programs start with python2. I kindly asked Distutils to look elsewhere, and now everything works.

Also, I added in specification of subcategories. I discovered that including multiple subcategories seemed to break things (at least on one PND I tried), so watch out. But otherwise, we're pretty much production-ready.

Therefore, this is likely the last build of The Lonely Tower I'll need to put up. And, as always, you can peruse its sources to see how to do the same with your own programs.

Awesome.

 

[gruso]

So, join the dots for a newbie here. Would this help me get Jan-Nik's game Tudi happening on the Pandora?

His thread: http://www.gp32x.de/board/index.php?/topic/51274-tudi

It's a simple but extremely addictive game, I love it.

 

[Tempel]

For a newbie, it probably doesn't mean much This won't help something run on the Pandora, it simply makes it easy to take already-working code and package it into a PND. And it does it in a way that works along with Python's standard distribution utilities, so it makes it easy for Python devs to distribute in many ways including PNDs.

It could be used to package up Tudi, except that its distribution method looks like it's not quite standard. There seems to be a lot of Windows-specific stuff and weird imports, but I'll take a stab at figuring it out.

 

[Esn]

Hey, is this very useful program mentioned in the wiki somewhere?

I'd do it myself, but I know very little about devving.

 

[Tempel]

I hadn't thought of putting it there. I'm not sure where in the wiki it would belong. Perhaps a "Tools for packaging" subheading on the Development Tools page?

EDIT: I finally did it.

 

[valhalla]

Arch linux users can get it from the AUR

 

[Tempel]

That's cool, valhalla! I was intending to make a PKGBUILD once it got some more testing, but no one's testing it yet. A couple notes:

1. Technically it's under the MIT license, not BSD, even though the two are so similar.
2. The git repo is structured such that official releases are on the master branch, while development happens on the develop branch. Your PKGBUILD seems to be pulling from the master branch, so it doesn't install all the latest code (and *-git packages generally want to have the latest developments, right?)
3. Since it's on GitHub, you don't need to do a full clone to get the latest sources. It automatically provides tar.gz packages of the latest source for both the master and develop branches. This should make for a simpler PKGBUILD, while still giving the latest source.

Thanks for doing this, though. Maybe a few more people will try it now.

 

[valhalla]

That's cool, valhalla! I was intending to make a PKGBUILD once it got some more testing, but no one's testing it yet. A couple notes:

1. Technically it's under the MIT license, not BSD, even though the two are so similar.
2. The git repo is structured such that official releases are on the master branch, while development happens on the develop branch. Your PKGBUILD seems to be pulling from the master branch, so it doesn't install all the latest code (and *-git packages generally want to have the latest developments, right?)
3. Since it's on GitHub, you don't need to do a full clone to get the latest sources. It automatically provides tar.gz packages of the latest source for both the master and develop branches. This should make for a simpler PKGBUILD, while still giving the latest source.

Thanks for doing this, though. Maybe a few more people will try it now.

1. fixed, sorry
2. right, I've changed it to use the develop branch; I' ve also created a new PKGBUILD for the release
3. the PKGBUILD wouldn't be really simpler, since it has to keep track of changing md5sums and directory names for said tarballs.

I have a couple of python programs, and I plan to try distpnd as soon as I get my pandora, so I needed the PKGBUILD (I don't like to install stuff outside the package manager)

 

[Tempel]

Nicely done, thank you

Of course, if you need any help or have any trouble using distPND, let me know.

 

[quartercast]

Hi Tempel,

I'm attempting to port RedNotebook. I'm getting the following error:

petes-computer:rednotebook-1.1 2.3 pete$ python setup.py bdist_pnd
data_base_dir /System/Library/Frameworks/Python.framework/Versions/2.6
running bdist_pnd
Traceback (most recent call last):
  File "setup.py", line 273, in <module>
    setup(**parameters)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/core.py", line 152, in setup
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/dist.py", line 975, in run_commands
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/dist.py", line 995, in run_command
  File "/Library/Python/2.6/site-packages/distpnd/bdist_pnd.py", line 90, in run
    '--executable=/usr/bin/python', '--force'))
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/core.py", line 221, in run_setup
  File "<string>", line 102, in <module>
  File "<string>", line 76, in build_mo_files
NameError: global name 'os' is not defined

I have looked to check 'import os' is included on all files, and it is! Any ideas what might be the cause?

 

[Tempel]

Woah, Distutils is being weird. It's always been weird, but I thought I had it all figured out until it did this.

If you look inside RedNotebook's setup.py, you'll see that it defines a bunch of functions that get used in the build process. It looks like what's happening is that, when bdist_pnd calls the build command, it runs one (or more) of those functions, but it runs them outside of the context of the rest of the file. I have no idea how you can even do that in Python, but the result is that all names defined outside of the function can't be seen within the function.

Presumably there's something I'm doing in distPND that just isn't how it's supposed to be. But this is looking like black magic to me, so I'm clearly going to have to spend some more time digging through Distutils. As a temporary workaround to get RedNotebook going, try adding the missing names inside the function. So, in function build_mo_files, adding "import os" fixes the immediate problem. Then it complains about "i18n_dir"; copy the definition from a few lines up. Then it complains about msgfmt; copy the "from rednotebook.external import msgfmt" from a few lines above that. Keep going until the errors stop or you get bored. Or just wait (indefinitely) for me to dig deeper into this bizarre world of Distutils.

EDIT: Just noticed it also depends on PyYAML, which is not installed by default on the Pandora. You'll have to make sure it gets into the PND yourself (I'm planning on making distPND do this automatically, but haven't yet). Just download its source, then run on its setup script:

python setup.py install --root=/path/to/rednotebook/build_pnd --install-lib=/

After that, all of PyYAML's files will be in the right place for bdist_pnd to pick them up automatically.

 

[Tempel]

I found the problem and a workaround, but it's certainly not ideal. It all comes down to Python's "exec" command. It's known to behave weirdly when given both global and local dicts. Apparently that's how it's supposed to behave, it's just not really what you would expect. Then the Distutils developers fell into exec's trap and seem to have misused it within the run_setup function that I call from distPND.

So that's the problem; now for the workaround. The easiest approach is to simply fix Distutils yourself. Open up /usr/lib/python2.?/distutils/core.py, then on line 221, remove the trailing ", l". Getting rid of those three characters allows bdist_pnd to work on RedNotebook (just remember to install PyYAML first, as I mentioned before).

I'm going to try to submit this as a bug report to the Distutils developers, if I can find out how, but who knows how long it'll take for the fix to make it in. I'll also see if I can take a different approach in distPND to avoid the broken run_setup function, but I don't think there's much else I can do.

Thanks for finding interesting problems for me

 

[Tempel]

I figure it deserves its own post: I fixed distPND to no longer rely on run_setup. I found a hacky solution in this old mailing list thread; they all seem to agree that run_setup is generally poor and should be avoided. The latest commit in my develop branch has the fix, so reinstall distPND from there and RedNotebook will build.

 

[quartercast]

I figure it deserves its own post: I fixed distPND to no longer rely on run_setup. I found a hacky solution in this old mailing list thread; they all seem to agree that run_setup is generally poor and should be avoided. The latest commit in my develop branch has the fix, so reinstall distPND from there and RedNotebook will build.

Thanks so much, I will try it tonight

 

[Esn]

Ok... I downloaded this and installed it by opening a terminal in its directory (using Thunar file browser) and typing "sudo python setup.py install". Then I navigated to the directory of the Python program I'm hoping to port, opened a terminal and typed in "python setup.py --command-packages distpnd bdist_pnd"

I got the following:

Traceback (most recent call last):
  File "setup.py", line 50, in <module>
    from cx_Freeze import setup, Executable
ImportError: No module named cx_Freeze

What did I do wrong?

 

[Tempel]

You did absolutely nothing wrong. The problem lies in the setup script. Many setup scripts are very simple; all of mine contain a single call to the setup function, and nothing else. But they are full Python programs, and can be arbitrarily complex.

It looks like this setup script you have expects to find cx_Freeze, used to turn Python scripts into standalone executables. A potential solution is just to install cx_Freeze in the same manner you installed distPND: just download and unpack the cx_Freeze source, navigate to its directory, and run "python setup.py install". But if that isn't enough to get your PND working, direct me to the program you're trying to port. Then I can take a look at its setup script to see what other tricks it might be trying to pull.

Good luck!

 

[Esn]

Thanks... I'm not sure that installing cx_Freeze worked as it should have. Here's a copy+paste of the terminal:

Spoiler

esn-openpandora:/media/mmcblk0p1/sources/cx_Freeze-4/cx_Freeze-4.2.3$ python setup.py install
adding base module named StringIO
adding base module named UserDict
adding base module named _abcoll
adding base module named abc
adding base module named base64
adding base module named codecs
adding base module named copy
adding base module named copy_reg
adding base module named encodings
adding base module named encodings.aliases
adding base module named encodings.ascii
adding base module named encodings.base64_codec
adding base module named encodings.big5
adding base module named encodings.big5hkscs
adding base module named encodings.bz2_codec
adding base module named encodings.charmap
adding base module named encodings.cp037
adding base module named encodings.cp1006
adding base module named encodings.cp1026
adding base module named encodings.cp1140
adding base module named encodings.cp1250
adding base module named encodings.cp1251
adding base module named encodings.cp1252
adding base module named encodings.cp1253
adding base module named encodings.cp1254
adding base module named encodings.cp1255
adding base module named encodings.cp1256
adding base module named encodings.cp1257
adding base module named encodings.cp1258
adding base module named encodings.cp424
adding base module named encodings.cp437
adding base module named encodings.cp500
adding base module named encodings.cp737
adding base module named encodings.cp775
adding base module named encodings.cp850
adding base module named encodings.cp852
adding base module named encodings.cp855
adding base module named encodings.cp856
adding base module named encodings.cp857
adding base module named encodings.cp860
adding base module named encodings.cp861
adding base module named encodings.cp862
adding base module named encodings.cp863
adding base module named encodings.cp864
adding base module named encodings.cp865
adding base module named encodings.cp866
adding base module named encodings.cp869
adding base module named encodings.cp874
adding base module named encodings.cp875
adding base module named encodings.cp932
adding base module named encodings.cp949
adding base module named encodings.cp950
adding base module named encodings.euc_jis_2004
adding base module named encodings.euc_jisx0213
adding base module named encodings.euc_jp
adding base module named encodings.euc_kr
adding base module named encodings.gb18030
adding base module named encodings.gb2312
adding base module named encodings.gbk
adding base module named encodings.hex_codec
adding base module named encodings.hp_roman8
adding base module named encodings.hz
adding base module named encodings.idna
adding base module named encodings.iso2022_jp
adding base module named encodings.iso2022_jp_1
adding base module named encodings.iso2022_jp_2
adding base module named encodings.iso2022_jp_2004
adding base module named encodings.iso2022_jp_3
adding base module named encodings.iso2022_jp_ext
adding base module named encodings.iso2022_kr
adding base module named encodings.iso8859_1
adding base module named encodings.iso8859_10
adding base module named encodings.iso8859_11
adding base module named encodings.iso8859_13
adding base module named encodings.iso8859_14
adding base module named encodings.iso8859_15
adding base module named encodings.iso8859_16
adding base module named encodings.iso8859_2
adding base module named encodings.iso8859_3
adding base module named encodings.iso8859_4
adding base module named encodings.iso8859_5
adding base module named encodings.iso8859_6
adding base module named encodings.iso8859_7
adding base module named encodings.iso8859_8
adding base module named encodings.iso8859_9
adding base module named encodings.johab
adding base module named encodings.koi8_r
adding base module named encodings.koi8_u
adding base module named encodings.latin_1
adding base module named encodings.mac_arabic
adding base module named encodings.mac_centeuro
adding base module named encodings.mac_croatian
adding base module named encodings.mac_cyrillic
adding base module named encodings.mac_farsi
adding base module named encodings.mac_greek
adding base module named encodings.mac_iceland
adding base module named encodings.mac_latin2
adding base module named encodings.mac_roman
adding base module named encodings.mac_romanian
adding base module named encodings.mac_turkish
adding base module named encodings.mbcs
adding base module named encodings.palmos
adding base module named encodings.ptcp154
adding base module named encodings.punycode
adding base module named encodings.quopri_codec
adding base module named encodings.raw_unicode_escape
adding base module named encodings.rot_13
adding base module named encodings.shift_jis
adding base module named encodings.shift_jis_2004
adding base module named encodings.shift_jisx0213
adding base module named encodings.string_escape
adding base module named encodings.tis_620
adding base module named encodings.undefined
adding base module named encodings.unicode_escape
adding base module named encodings.unicode_internal
adding base module named encodings.utf_16
adding base module named encodings.utf_16_be
adding base module named encodings.utf_16_le
adding base module named encodings.utf_32
adding base module named encodings.utf_32_be
adding base module named encodings.utf_32_le
adding base module named encodings.utf_7
adding base module named encodings.utf_8
adding base module named encodings.utf_8_sig
adding base module named encodings.uu_codec
adding base module named encodings.zlib_codec
adding base module named genericpath
adding base module named linecache
adding base module named os
adding base module named posixpath
adding base module named quopri
adding base module named repr
adding base module named stat
adding base module named stringprep
adding base module named struct
adding base module named traceback
adding base module named types
adding base module named warnings
running install
running build
running build_py
creating build/lib.linux-armv7l-2.6
creating build/lib.linux-armv7l-2.6/cx_Freeze
copying cx_Freeze/dist.py -> build/lib.linux-armv7l-2.6/cx_Freeze
copying cx_Freeze/freezer.py -> build/lib.linux-armv7l-2.6/cx_Freeze
copying cx_Freeze/finder.py -> build/lib.linux-armv7l-2.6/cx_Freeze
copying cx_Freeze/windist.py -> build/lib.linux-armv7l-2.6/cx_Freeze
copying cx_Freeze/__init__.py -> build/lib.linux-armv7l-2.6/cx_Freeze
copying cx_Freeze/hooks.py -> build/lib.linux-armv7l-2.6/cx_Freeze
copying cx_Freeze/main.py -> build/lib.linux-armv7l-2.6/cx_Freeze
running build_ext
building 'cx_Freeze.util' extension
creating build/temp.linux-armv7l-2.6/source
arm-angstrom-linux-gnueabi-gcc -march=armv7-a -mtune=cortex-a8 -mfpu=neon -mfloat-abi=softfp -D__SOFTFP__ -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c source/util.c -o build/temp.linux-armv7l-2.6/source/util.o
unable to execute arm-angstrom-linux-gnueabi-gcc: No such file or directory
error: command 'arm-angstrom-linux-gnueabi-gcc' failed with exit status 1
esn-openpandora:/media/mmcblk0p1/sources/cx_Freeze-4/cx_Freeze-4.2.3$

There seem to be some errors at the end, concerning arm-angstrom-linux-gnueabi-gcc.

The program still acts like it's not installed (I'll say what program it is if I get really stuck; I'm just trying to motivate myself to learn how this coding stuff works first).