Can somebody (I) host a repo mirror?


bzar

A Commando
Joined
Sep 22, 2008
Messages
4,463
Location
Finland
Website
Visit site
That's a cap on pnd manager I believe not the repo bzar could answer that.
There's a cap resulting from CPU limiting I set for the downloads. It's a tradeoff between download speed and UI performance.


The responsible line is here. Making that sleep smaller raises the download speed while degrading UI experience. On desktop you can put that sleep to 1ms or smaller and have full DL speeds and full UI speed, but on pandora I had to make a compromise there :(
 

Binky

Death's Steed
Staff member
Joined
May 28, 2003
Messages
6,913
Location
16A (TO)
Wouldn't that work better if downloads were sequential (not parallel) ?
 

milkshake

Advanced Member
Joined
May 18, 2009
Messages
3,735
Age
36
Location
Rotherham, UK

ShleeDragon

Pro-Catflip
Joined
Aug 28, 2010
Messages
1,749
@:  You could try changing ISP...    I don't have any such issues most of the time.  (and my traceroute doesn't go through singtel)
That's the stupidest thing I've ever heard... "Screw you, I got mine". try changing ISP because it's that simple.. I might as well move country too huh?

If you're not technically able to help build support for the global members of our community - Please stay out of this. there are way too many selfish nerds with no empathy in this thread already. OOOOH It works for me so that means everything is fine I SAID GOOD DAY SIR...  ;)
 
Last edited by a moderator:

ShleeDragon

Pro-Catflip
Joined
Aug 28, 2010
Messages
1,749
Thanks to you guys for compressing the masterlist for a start.

I don't really see a problem with maintaining a replicated mirror because the repo is actually pretty small and there are only a few changes every day.
We don't need to over-engineer anything but there are easier ways to deal with this.

The idea of setting up a bunch of anycast repos is overkill when the ability to change repos in a tool would do fine.
 
Last edited by a moderator:

skeezix

Internal Development
Joined
Mar 11, 2003
Messages
8,062
Website
www.codejedi.com
Theres several goals..

i) do we want full repo clones; if the main repo goes down (meteorite!), then do you want to just change to another repo website and good to go (sans backups and commercial pnds)?

ii) do we want just a file backup, and then the main repo could if it wishes send out a link to the alternate repos url (ie: for faster download, no change to normal website performance of course)

iii) do we just want backups; ie: a rsync/wget of the repo each night, so that the files are somewhere else; list woudl be a long directory listing only, so it works for people in a pinch and trivial to set up

.. do you want PNDManager and so forth to be able to query the backup repos, and talk to them as if they were the real deal?

.. or are the mirrors just for faster download, or file availability in case Germany is swallowed by Cthulhu?

Need to define the purpose otherwise the conversation will fly all over the place perhaps..

jeff
 

Binky

Death's Steed
Staff member
Joined
May 28, 2003
Messages
6,913
Location
16A (TO)
Could someone tweak PNDmanager so that the package listing and downloading went like this:

The config file contains a list of known servers in order of user preference (probably repo.openpandora.org first)

On startup, download a package list from all known servers.

Use this download to evaluate the connection to each server (which is fastest?)

Look up the package to download to see which servers have a copy

(depending on user preference either:

--- download from server furthest up list

--- download from server with fastest connection

(obviously do not choose a sever which does not send a package list, because it is either overloaded or broken)

That allows for:

  1. mirrors for speed
  2. mirrors in case ED's server explodes
  3. devs hosting their own content
 
Last edited by a moderator:

levi

Still fresh, damnit!
Joined
Oct 6, 2008
Messages
12,485
Location
Somewhere off the coast of the EU
FWIW it's possible for servers to be configured to offer gzipped versions of any files you ask for, and it's up to your http client to negotiate to get the gzipped files, which are dynamically zipped up on the server and unzipped as part of the http client. Not sure, but I think it should be possible to offer bzip2ed files the same way. If the server were configured to do that (and your web client is intelligent enough) you shouldn't see much difference between the bzipped 2 master list and just pulling down the full master list.
 

Neelix

Insecticidal Maniac
Joined
Jan 8, 2011
Messages
3,243
Location
Melbourne, Australia
@:  You could try changing ISP...    I don't have any such issues most of the time.  (and my traceroute doesn't go through singtel)
That's the stupidest thing I've ever heard... "Screw you, I got mine". try changing ISP because it's that simple.. I might as well move country too huh?
That's not what I said,  nor what I meant.    You were directing ED to take things up with his ISP when the reality is that the issue is closer to your ISP's end of the connection.  You could take things up with your ISP,  but frankly, what speaks loudest to ISPs in this country these days is churn. 

- Neelix
 

WizardStan

Mega GP Mania
Joined
May 24, 2008
Messages
16,733
Squashfs is already compressed. Compressing on the fly for transfer will accomplish nothing but CPU waste, unfortunately.
 

_wb_

Microbe
Staff member
Joined
Apr 5, 2012
Messages
5,387
Age
39
Location
Brussels, Belgium
WizardStan: I think Levi was talking about the masterlist, not the PNDs.

Binky: it's probably better (and easier to implement) to just use a sensible default repo server based on your continent, which can be overridden manually if needed.

skeezix: I think your third option would be good. Basically a read-only copy of the entire thing, excluding the commercial PNDs and maybe the archived old versions. To leave comments, rate stuff, upload stuff, etc. you still have to go to the main server, but to just download some PND or its meta-data (screenshots etc), you can use the mirror.
 

bzar

A Commando
Joined
Sep 22, 2008
Messages
4,463
Location
Finland
Website
Visit site
As far as im aware PNDManager supports the use of multiple repo's already assuming they follow the REPO json spec
The package management backend library of the pndmanagement Panorama plugin PNDManager uses (called libpndman) has that capability. It's not implemented in pndmanagement-plugin nor PNDManager because at the time there were no other repos nor any in sight.

Binky: That would require changes in all libpndman, qtpndman (Qt wrapper for libpndman), pndmanagement-plugin and PNDManager. Anyone is welcome to do that of course, but it's quite a bit more work than it sounds ;)
 
Last edited by a moderator:

ShleeDragon

Pro-Catflip
Joined
Aug 28, 2010
Messages
1,749
@:  You could try changing ISP...    I don't have any such issues most of the time.  (and my traceroute doesn't go through singtel)
That's the stupidest thing I've ever heard... "Screw you, I got mine". try changing ISP because it's that simple.. I might as well move country too huh?
That's not what I said,  nor what I meant.    You were directing ED to take things up with his ISP when the reality is that the issue is closer to your ISP's end of the connection.  You could take things up with your ISP,  but frankly, what speaks loudest to ISPs in this country these days is churn. 

- Neelix


My day job is a sysmin at the datacentre of the most awesome amount of bandwidth (it's suck there too) and I'm very much aware that the internet is a series of tubes.. we need more dump trucks.

I think my original intention was pretty much in the topic.. Can we please move towards multiple repos because it's the right thing to do.
 
Last edited by a moderator:

infinity

Member
Joined
Mar 11, 2011
Messages
2,533
Age
26
Location
Germany
I have a Server with gigabit up/down(yes, real gigabit Internet) oft you want to set up a mirror PM me;)
 
Last edited by a moderator:
Top