Can somebody (I) host a repo mirror?


ShleeDragon

Pro-Catflip
Joined
Aug 28, 2010
Messages
1,749
Hi ED... This might be related to the increase in the number of pandora users but I am unable to use the PNDmanager due to timeouts.

So it's broken for me/impossible to show off the environment in my video review if it never syncs to update. ;)

I'm Australian and forum has started to become very slow for me too... I think you are hosting too much on a single box..

The PNDmanager and milkyhelper keeps dropping out and never syncs successfully. I've tested and it takes more than 3 minutes to download the masterlist.....  wget tested: 1,4MB at 6.07KB/s in 3m 22s

I think it's time to investigate growing the repo environment if it is having capacity issues as I cannot be the only person with this problem.
 
Last edited by a moderator:

Takedown

Still Fresh
Joined
Oct 11, 2011
Messages
73
Location
Panama
I actually never had a single problem with PNDmanager, the only problems I've ever had were internet issues because my provider just sucks...

EDIT: and I didn't realize it was posted in ask ED questions, my bad... (T~T)
 
Last edited by a moderator:

ShleeDragon

Pro-Catflip
Joined
Aug 28, 2010
Messages
1,749
I cannot be the only person with this problem and we're getting more and more pandoras in peoples hands. until then, I'll continue to update manually.

traceroute is pretty obvious. hits singtel.com and connection gones to shit.
 
Last edited by a moderator:

EvilDragon

Administrator
Staff member
Joined
Mar 4, 2003
Messages
29,515
Age
42
Location
Ingolstadt
It's not the server, it seems to be the connection between Hetzner (my provider) in Germany and some countries overseas.


The server has capacities left and usually is blazing fast here in Germany.


I'll ask them whether something can be done here.


I'm also happy having mirrors, but it would only work for the repo and only if milkshake would implement something like that.
 

Neelix

Insecticidal Maniac
Joined
Jan 8, 2011
Messages
3,243
Location
Melbourne, Australia
@:  You could try changing ISP...    I don't have any such issues most of the time.  (and my traceroute doesn't go through singtel)
 

juliano

Member
Joined
Jan 31, 2011
Messages
243
I had this issue on monday. I couldn´t refresh PNDManager (I´m in Brazil). But then I tried again today and it worked fine.
 

milkshake

Advanced Member
Joined
May 18, 2009
Messages
3,735
Age
36
Location
Rotherham, UK
having a mirror will not fix PND Manager as those downloads would always come from the repo but I could provide mirror download links on the repo webpage provided a mirror existed (though no doubt only redistributable games will be on the mirror).

I had this issue on monday. I couldn´t refresh PNDManager (I´m in Brazil). But then I tried again today and it worked fine.
the repo had issues on monday anyway.
 

_wb_

Microbe
Staff member
Joined
Apr 5, 2012
Messages
5,387
Age
39
Location
Brussels, Belgium
ED's server is fine for Europe, but mirrors for Australia and the USA would be a good idea. Ideally things would be set up in such a way that repo.openpandora.org automatically redirects you to the mirror that is closest given your ip-address.

Other things that could help, especially for people with limited (client-side) bandwidth, would be:

- Compress the masterlist (I think it is plaintext now); perhaps the web server already does this automatically but I'm not sure.

- Incremental sync: now it downloads the entire list (~1.5MB and growing) each time you sync; it would be better if you could download only the "new" part of the list based on the timestamp of your latest sync.

- Incremental PND downloads: this is discussed here: http://boards.openpandora.org/index.php/topic/13108-pnd-diffs/

- Compress the PNDs for transfer: for PNDs containing an ISO filesystem this might make a big difference, and for the squashfs ones it could be possibly still save a bit (probably not worth it though)
 
Last edited by a moderator:

milkshake

Advanced Member
Joined
May 18, 2009
Messages
3,735
Age
36
Location
Rotherham, UK
ED's server is fine for Europe, but mirrors for Australia and the USA would be a good idea. Ideally things would be set up in such a way that repo.openpandora.org automatically redirects you to the mirror that is closest given your ip-address.
This is something that could be looked into assuming we have mirrors that exist
- Compress the masterlist (I think it is plaintext now); perhaps the web server already does this automatically but I'm not sure.
Its JSON format at the moment.
- Incremental sync: now it downloads the entire list (~1.5MB and growing) each time you sync; it would be better if you could download only the "new" part of the list based on the timestamp of your latest sync.
This already exists but is not in use for some reason see b-zar and or cloudef for that.
- Compress the PNDs for transfer: for PNDs containing an ISO filesystem this might make a big difference, and for the squashfs ones it could be possibly still save a bit (probably not worth it though)
99% of PND's are squashfs
How many people are actually affected by this? i.e. out of all people with pandoras in australia or elsewhere how many are affected because this could be just ISP?
 
Last edited by a moderator:

Mr_Loon

Can't Remember
Joined
Aug 30, 2010
Messages
2,329
I've got a slightly strange issue with the repo when downloading updates / new apps via PNDManager.

I'm using an edimax mini wifi stick and get a maximum download speed of 176KB/s for any individual download.

i.e. I can have 4 PNDs downloading at once and each max's out at 176KB/s or if just downloading 1 PND the maximum speed is still 176KB/s.

Is there a hard limit on download speed in place anywhere?
 

TrashyMG

Sarcasm Dispenser
Staff member
Joined
Jan 18, 2010
Messages
10,736
^ I see those same numbers, my guess there is a cap in place.
 

_wb_

Microbe
Staff member
Joined
Apr 5, 2012
Messages
5,387
Age
39
Location
Brussels, Belgium
ED's server is fine for Europe, but mirrors for Australia and the USA would be a good idea. Ideally things would be set up in such a way that repo.openpandora.org automatically redirects you to the mirror that is closest given your ip-address.
This is something that could be looked into assuming we have mirrors that exist
 Assuming that the mirrors only offer the download-part of the repo, not the uploading and account creation/management, how hard would it be to make a mirror? I would assume that you could just rsync all data files (PNDs, metadata, account details) on a regular basis (say a cronjob that does it every hour). This means that it takes some time for new accounts, changed passwords, and new uploads to propagate to the mirrors, but that's fine I guess.

- Compress the masterlist (I think it is plaintext now); perhaps the web server already does this automatically but I'm not sure.
Its JSON format at the moment.
Right, so that's basically structured plaintext. It compresses rather well: the current 1.5MB masterlist takes only 310KB when gzip'ed and 228KB when bzip2'ed.

How many people are actually affected by this? i.e. out of all people with pandoras in australia or elsewhere how many are affected because this could be just ISP?
I guess everyone is affected by all the slower-than-necessary downloads, it's just that the problem is more extreme when you're far away from ED's server. But everybody would benefit from incremental, compressed masterlist updates and incremental PND updates.
 

skeezix

Internal Development
Joined
Mar 11, 2003
Messages
8,062
Website
www.codejedi.com
Whats the total size of the repo now anyway? We could do a growth projection (or just double the existing size, and say good for the next year or two)..

.. but I may be able to host a North American mirror; will have to talk to some buddies who run an ISP ;)

A mirror would have to have all the content (otherwise defeating the purpose), but be read-only; would it be the repo frontend on EDs server, but handing out file-links to the mirrors? This way one source of truth and one UI, andno need for read only.. but downloads would be local-fast? (users could pick their preferred mirror in their repo profile, to avoid milkshake having to build automatic detection in..)

... or have read-only full copies of the repo software at the mirrors.. but this becomes maintenance pain for milkshake and mirror hosts to update the software.

jeff
 

FaeMinx

Rainbow Liberation Instigation
Joined
Dec 11, 2010
Messages
3,128
Age
39
Location
outside looking in side looking out
Is it even nescessary to state that being in South Africa, I've had problems all along?  :)

I also pay for every bit of data downloaded... so whatever can be done to compress downloads and generally speed up the process for us poor far-flung people would be greatly appreciated.
 

_wb_

Microbe
Staff member
Joined
Apr 5, 2012
Messages
5,387
Age
39
Location
Brussels, Belgium
The mirrors don't necessarily have to keep all the archived versions of the PNDs, just the most recent one will do. That probably saves quite some space.
 

milkshake

Advanced Member
Joined
May 18, 2009
Messages
3,735
Age
36
Location
Rotherham, UK
There is also the commercial apps too which wouldn't be able to be mirrored without some complex dB sync between servers and its probably securer all round to have them on eds server.
 

lingenfr

Member
Joined
Mar 17, 2009
Messages
418
The mirrors don't necessarily have to keep all the archived versions of the PNDs, just the most recent one will do. That probably saves quite some space.
You might also consider setting up the "mirrors" as caches. That would be more efficient that rsync or could be used in combination with rsync, i.e. always caching, rsync changes nightly as they occur, etc. That way you wouldn't be rsyncing files that no one is downloading. The "use it until it breaks" method doesn't really support growth and sets up the potential for "victims of our own success" to occur.
 

milkshake

Advanced Member
Joined
May 18, 2009
Messages
3,735
Age
36
Location
Rotherham, UK
You might also consider setting up the "mirrors" as caches. That would be more efficient that rsync or could be used in combination with rsync, i.e. always caching, rsync changes nightly as they occur, etc. That way you wouldn't be rsyncing files that no one is downloading. The "use it until it breaks" method doesn't really support growth and sets up the potential for "victims of our own success" to occur.
Well there is always a place in the world where the sun is up so when you say nightly ... when do you mean :)
 
Top