GP2X Python/pygame List Restrictions?

[thoro]

I started to port a simple image sequence viewer I did in JavaScript to view prerendered 3D objects in a web browser without using plugins (http://www.thoro.de/3DNP).

I'm storing all images in a zip archive and am using the python zipfile module to load all the images into a list. While this works fine on my desktop computer, there seems to be a memory limit on the GP2X. It seems to me as if GP2X python crashes if my images.zip archive is bigger than 256KB.

How can I bypass this problem?

 

[chimpoid]

Hi Thoro,

256kb shouldn't be causing a problem with the internal memory. Python-wise I'm no help though I'm afraid.
Have you tried popping the images into a folder and then loading them in sequence without the zip?
Purely to eliminate the possibility that it is zip related.

Slightly OT: I was thinking about your 3DNP and was planning implementing something similar in C but it looks like there no need now

Chimpoid

 

[Radek]

I started to port a simple image sequence viewer I did in JavaScript to view prerendered 3D objects in a web browser without using plugins (http://www.thoro.de/3DNP).

I'm storing all images in a zip archive and am using the python zipfile module to load all the images into a list. While this works fine on my desktop computer, there seems to be a memory limit on the GP2X. It seems to me as if GP2X python crashes if my images.zip archive is bigger than 256KB.

How can I bypass this problem?

​

Could you show that list (how long is it and what are its elements?).

 

[BradN]

Any info on how it crashes? Maybe redirect stdout/error output to a file and examine that after.

 

[thoro]

Thanks for all your hints.

After some tests I'm sure that my problem is not zip-related

This is my test script:

import pygame, os, sys, zipfile, math

imagepool = ['0']
suffix = '.jpg'
imagenumber = 144
imagename = 'frame'
imagepath = 'images'

def read_images():

    for m in range(1, imagenumber+1):
        fullname = imagepath+'/'+imagename+leadingZero(str(m),4)+suffix
        print 'loading image: '+fullname	# debug information
        try: 
            imagepool.append(pygame.image.load(fullname))
        except pygame.error, message:
            print 'Error loading image: '+fullname
            raise SystemExit, message
  
def leadingZero(source,n):

    filled = source
    for i in range(0, n-len(source)):
        filled = '0'+filled
    return(filled)  
  
read_images()

If I run this from a shell script on my GP2X (with python option -v) it stops after 90 to 92 images (each image is about 4KB) throwing the error message 'Killed'.

I used pygame.image.frombuffer to get the size of the image data strings - I think pygame is filling the list with the uncompressed image data, which in my example is 144 x 230400 Bytes = 32.4 MB - I guess that's the killer.

I used pygame.Surface.convert to reduce the size - this let's me load 136 images now.

Is there a way to compress the string data before putting it into the list and decompress it afterwards to show the image?

 

[chimpoid]

That's the thought that occured to me last night.
The images are just raw strings of data once they are loaded into memory so any compression used initially (jpg) is discarded.
I can't think of any way to compress and the only solution I can think of is to cache a smaller number images around the current viewed image and set a tolerance value that frees old images and loads new ones in it's place.
e.g With the 2d array, load any image within 4 "squares" of the current viewed image. if the current image becomes within 1 "square" of the outer edges of your loaded images then reload a new batch of the images.
The problem there is that there is a bigger impact on the sd-card as images are loaded repeatedly

Sorry, I can't be more helpful with that.

Chimpoid

 

[Radek]

Thanks for all your hints.

After some tests I'm sure that my problem is not zip-related

This is my test script:

import pygame, os, sys, zipfile, math

imagepool = ['0']
suffix = '.jpg'
imagenumber = 144
imagename = 'frame'
imagepath = 'images'

def read_images():

    for m in range(1, imagenumber+1):
        fullname = imagepath+'/'+imagename+leadingZero(str(m),4)+suffix
        print 'loading image: '+fullname	# debug information
        try: 
            imagepool.append(pygame.image.load(fullname))
        except pygame.error, message:
            print 'Error loading image: '+fullname
            raise SystemExit, message
  
def leadingZero(source,n):

    filled = source
    for i in range(0, n-len(source)):
        filled = '0'+filled
    return(filled)  
  
read_images()

If I run this from a shell script on my GP2X (with python option -v) it stops after 90 to 92 images (each image is about 4KB) throwing the error message 'Killed'.

I used pygame.image.frombuffer to get the size of the image data strings - I think pygame is filling the list with the uncompressed image data, which in my example is 144 x 230400 Bytes = 32.4 MB - I guess that's the killer.

I used pygame.Surface.convert to reduce the size - this let's me load 136 images now.

Is there a way to compress the string data before putting it into the list and decompress it afterwards to show the image?

​

Why do you want to use lists for storing bitmaps and load them from the zip archive? The direct use of pygame.image.load should much faster and the files like the png can be compressed too. If you really want to reduce footprint of bitmaps in a memory then you might try the "RLEACCEL" flag for surfaces.

 

[thoro]

Loading them directly from the SD card is much slower (at least from my SD card).

Is it possible to load the whole zip file into memory and unzip the needed image from there at runtime?

How do I use the RLEACCEL flag? I took a look into the pygame docs but it was only mentioned as a parameter for Surface.set_colorkey and Surface.set_alpha.

 

[BradN]

Another option is unzipping the images to memory, since this is unlikely to increase the size a lot, and then doing the jpeg/etc decompression on the fly, but you still run into the situation of having compressed images too big to fit in memory.

Perhaps a hybrid caching approach is best - for example, decompress the previous, current, and next images into framebuffer-ready images, and try to cache as many of the nearby jpeg'd images as possible, or cache their decompressed equivalent if it's smaller than the jpeg size. There's lots of possible heuristics that might give better overall performance, and it could get even more complex if you want your viewer to allow zooming - how many full size decompressed images do you keep in memory, etc.

 

[theoddbot]

Yeah this is exactly what I've been doing with PDFView. I keep previous, current and next images ready to go, and the next few pages are rendered and compressed in memory with zlib. Since most pdfs are fairly empty, the memory savings are huge. When you skip to the next page, the oldest is compressed, another is decompressed and maybe another is rendered. JPEG rendering is probably a hell of a lot faster than pdf, so you probably don't have to go to the trouble of rendering more than the prev/next.

 

[Radek]

Loading them directly from the SD card is much slower (at least from my SD card).

Is it possible to load the whole zip file into memory and unzip the needed image from there at runtime?

So I can suggest another aproach - load to string as usual but only one image at time. Then convert it immediately to a surface and just destroy the string object. Repeat for all surfaces. It should be much easier on the memory.

How do I use the RLEACCEL flag? I took a look into the pygame docs but it was only mentioned as a parameter for Surface.set_colorkey and Surface.set_alpha.

​

Just like the others flags I think - add "| RLEACCEL" to convert or use the functions you said. I must admit I was never using that flag but I told about it because perhaps it could an improvement (just experiment).

 

[thoro]

Thanks for the good tips!

I chose to load the data from the zip archive and put the compressed images into a list at startup. When a new image is needed, I let pygame.image.load decompress it and display it.

This works pretty good now.

Can I use PNGs with alpha channel to add a background image? How do I combine the background image and the loaded Surface?

 

[theoddbot]

Just load your png and blit it on after you draw the jpg. You might not be able to convert it do the display format without loosing the alpha channel.