Can somebody (I) host a repo mirror?

[ShleeDragon]

Hi ED... This might be related to the increase in the number of pandora users but I am unable to use the PNDmanager due to timeouts.

So it's broken for me/impossible to show off the environment in my video review if it never syncs to update.

I'm Australian and forum has started to become very slow for me too... I think you are hosting too much on a single box..

The PNDmanager and milkyhelper keeps dropping out and never syncs successfully. I've tested and it takes more than 3 minutes to download the masterlist.....  wget tested: 1,4MB at 6.07KB/s in 3m 22s

I think it's time to investigate growing the repo environment if it is having capacity issues as I cannot be the only person with this problem.

 

[Takedown]

I actually never had a single problem with PNDmanager, the only problems I've ever had were internet issues because my provider just sucks...

EDIT: and I didn't realize it was posted in ask ED questions, my bad... (T~T)

 

[ShleeDragon]

I cannot be the only person with this problem and we're getting more and more pandoras in peoples hands. until then, I'll continue to update manually.

traceroute is pretty obvious. hits singtel.com and connection gones to shit.

 

[EvilDragon]

It's not the server, it seems to be the connection between Hetzner (my provider) in Germany and some countries overseas.


The server has capacities left and usually is blazing fast here in Germany.


I'll ask them whether something can be done here.


I'm also happy having mirrors, but it would only work for the repo and only if milkshake would implement something like that.

 

[ShleeDragon]

Please raise this with him.

 

[Neelix]

@:  You could try changing ISP...    I don't have any such issues most of the time.  (and my traceroute doesn't go through singtel)

 

[juliano]

I had this issue on monday. I couldn´t refresh PNDManager (I´m in Brazil). But then I tried again today and it worked fine.

 

[milkshake]

having a mirror will not fix PND Manager as those downloads would always come from the repo but I could provide mirror download links on the repo webpage provided a mirror existed (though no doubt only redistributable games will be on the mirror).

I had this issue on monday. I couldn´t refresh PNDManager (I´m in Brazil). But then I tried again today and it worked fine.

the repo had issues on monday anyway.

 

[_wb_]

ED's server is fine for Europe, but mirrors for Australia and the USA would be a good idea. Ideally things would be set up in such a way that repo.openpandora.org automatically redirects you to the mirror that is closest given your ip-address.

Other things that could help, especially for people with limited (client-side) bandwidth, would be:

- Compress the masterlist (I think it is plaintext now); perhaps the web server already does this automatically but I'm not sure.

- Incremental sync: now it downloads the entire list (~1.5MB and growing) each time you sync; it would be better if you could download only the "new" part of the list based on the timestamp of your latest sync.

- Incremental PND downloads: this is discussed here: http://boards.openpandora.org/index.php/topic/13108-pnd-diffs/

- Compress the PNDs for transfer: for PNDs containing an ISO filesystem this might make a big difference, and for the squashfs ones it could be possibly still save a bit (probably not worth it though)

 

[milkshake]

ED's server is fine for Europe, but mirrors for Australia and the USA would be a good idea. Ideally things would be set up in such a way that repo.openpandora.org automatically redirects you to the mirror that is closest given your ip-address.

This is something that could be looked into assuming we have mirrors that exist

- Compress the masterlist (I think it is plaintext now); perhaps the web server already does this automatically but I'm not sure.

Its JSON format at the moment.

- Incremental sync: now it downloads the entire list (~1.5MB and growing) each time you sync; it would be better if you could download only the "new" part of the list based on the timestamp of your latest sync.

This already exists but is not in use for some reason see b-zar and or cloudef for that.

- Compress the PNDs for transfer: for PNDs containing an ISO filesystem this might make a big difference, and for the squashfs ones it could be possibly still save a bit (probably not worth it though)

99% of PND's are squashfs
How many people are actually affected by this? i.e. out of all people with pandoras in australia or elsewhere how many are affected because this could be just ISP?

 

[Mr_Loon]

I've got a slightly strange issue with the repo when downloading updates / new apps via PNDManager.

I'm using an edimax mini wifi stick and get a maximum download speed of 176KB/s for any individual download.

i.e. I can have 4 PNDs downloading at once and each max's out at 176KB/s or if just downloading 1 PND the maximum speed is still 176KB/s.

Is there a hard limit on download speed in place anywhere?

 

[TrashyMG]

^ I see those same numbers, my guess there is a cap in place.

 

[milkshake]

That's a cap on pnd manager I believe not the repo bzar could answer that.

 

[_wb_]

ED's server is fine for Europe, but mirrors for Australia and the USA would be a good idea. Ideally things would be set up in such a way that repo.openpandora.org automatically redirects you to the mirror that is closest given your ip-address.

This is something that could be looked into assuming we have mirrors that exist

 Assuming that the mirrors only offer the download-part of the repo, not the uploading and account creation/management, how hard would it be to make a mirror? I would assume that you could just rsync all data files (PNDs, metadata, account details) on a regular basis (say a cronjob that does it every hour). This means that it takes some time for new accounts, changed passwords, and new uploads to propagate to the mirrors, but that's fine I guess.

- Compress the masterlist (I think it is plaintext now); perhaps the web server already does this automatically but I'm not sure.

Its JSON format at the moment.

Right, so that's basically structured plaintext. It compresses rather well: the current 1.5MB masterlist takes only 310KB when gzip'ed and 228KB when bzip2'ed.

How many people are actually affected by this? i.e. out of all people with pandoras in australia or elsewhere how many are affected because this could be just ISP?

I guess everyone is affected by all the slower-than-necessary downloads, it's just that the problem is more extreme when you're far away from ED's server. But everybody would benefit from incremental, compressed masterlist updates and incremental PND updates.

 

[skeezix]

Whats the total size of the repo now anyway? We could do a growth projection (or just double the existing size, and say good for the next year or two)..

.. but I may be able to host a North American mirror; will have to talk to some buddies who run an ISP

A mirror would have to have all the content (otherwise defeating the purpose), but be read-only; would it be the repo frontend on EDs server, but handing out file-links to the mirrors? This way one source of truth and one UI, andno need for read only.. but downloads would be local-fast? (users could pick their preferred mirror in their repo profile, to avoid milkshake having to build automatic detection in..)

... or have read-only full copies of the repo software at the mirrors.. but this becomes maintenance pain for milkshake and mirror hosts to update the software.

jeff

 

[FaeMinx]

Is it even nescessary to state that being in South Africa, I've had problems all along? 

I also pay for every bit of data downloaded... so whatever can be done to compress downloads and generally speed up the process for us poor far-flung people would be greatly appreciated.

 

[_wb_]

The mirrors don't necessarily have to keep all the archived versions of the PNDs, just the most recent one will do. That probably saves quite some space.

 

[milkshake]

There is also the commercial apps too which wouldn't be able to be mirrored without some complex dB sync between servers and its probably securer all round to have them on eds server.

 

[lingenfr]

The mirrors don't necessarily have to keep all the archived versions of the PNDs, just the most recent one will do. That probably saves quite some space.

You might also consider setting up the "mirrors" as caches. That would be more efficient that rsync or could be used in combination with rsync, i.e. always caching, rsync changes nightly as they occur, etc. That way you wouldn't be rsyncing files that no one is downloading. The "use it until it breaks" method doesn't really support growth and sets up the potential for "victims of our own success" to occur.

 

[milkshake]

You might also consider setting up the "mirrors" as caches. That would be more efficient that rsync or could be used in combination with rsync, i.e. always caching, rsync changes nightly as they occur, etc. That way you wouldn't be rsyncing files that no one is downloading. The "use it until it breaks" method doesn't really support growth and sets up the potential for "victims of our own success" to occur.

Well there is always a place in the world where the sun is up so when you say nightly ... when do you mean