Release Ready-To-Flash Firmware-Image

[notaz]

When there are new updates and we have to take care of the non-Internet'ed Pandora usrs, well .. the Repo Manager Person simply creates a list of updated packages since the last update, using "opkg list-upgradable" and then copies those .opk files into a .zip for others to download, put on an SD card, and get installed on their non-Internet-accessible Pandora. There will have to be a manually-installable list of .opk's in each major update of the repo, made available through the same mechanism as you are currently using for Hotfix releases: an announcement on the forums and a direct link to the file. Until everyone with a Pandora is on the Internet with it, this is just going to have to be the way things are done, alas: but its not difficult to administer if you're using a repo in the first place to tell you what changed! You could probably script it and integrate it directly with Twitter and so on, even ..

That sounds like all this could be packaged into .pnd and installed in a user familiar way, right?

 

[torpor]

That sounds like all this could be packaged into .pnd and installed in a user familiar way, right?

EDIT: Yes, it could be just put into a .PND which fires up the opkg manager on its own local cache of (newly updated) packages, but the advantage would be that the software install is *still* being managed by opkg, with all the advantages that comes with using opkg for package management. Right now I have no clue what the hell the Hotfix is doing - is it running custom (as we have seen: buggy) scripts to overwrite things, is it doing diffs/merges on my configuration files that I may have modified locally myself, and so on ..

Everyones Pandora will still be managed by opkg, and there will be parity there - just that some Pandoras can update directly from the Internet (repo's) and others will have a "copy this file to SD, put SD in Pandora" middle phase that they have to handle themselves. Presumably, this won't be too difficult to manage.

I don't see this to be too different from releasing HF on the beta forum, except distribution. I agree that just running opkg on pandora is much more convenient than hunting for new HF on the forums every time.

It doesn't have to be opkg at the cmd line - Angstrom comes with a GUI package manager which can be configured to pop up the README/Changelog before applying upgrades ..

100% everything can be built using OE, there are no magic binary or other drop-ins.

SGX drivers? DSP Bridge/Link? Bitbake recipe to build the Pandora kernel? Bitbake recipes for ED's control panel/user-accessories? Not trying to be painful, I just want to know where those things are coming from ..

What DjWillis probably meant was that some .bb files are not properly done due to lack of time, which means they can't be sent to OE mainline. And some OE .bbs are hacked up and produce incompatible packages to those found in http://www.angstrom-...tion.org/feeds/ and similar. But all this doesn't prevent us doing our custom repo, and if you can help with that please do.

I will go take my own close, non-generalized, very detailed look .. but last time I did that (a month ago), it was still pretty messy on the ol' GIT server .. so I will have a look at the scene with renewed vigor and see if I can't come up with a workable plan for how I can help.

It doesn't seem we have manpower for that, ED can do work during overnight zombie sessions but he doesn't even know where to start. If someone prepared simple nice instruction I'm sure things would start moving forward.

Can we have a "Base OS Developers/Contributors" mailing list, and a "Repo BETA Users" list, just to get the ball rolling on how to get these instructions formulated and activated? Or, maybe just separate forums for now?

Having sync option on is a braindead thing to do, it drops write performance several times below GP2X levels! The current devsync+write_expire solution seems work to most people with nice write speeds.

I agree that there is still much to be done with regards to performance, but I got bit by this one because I didn't know about it (my own stupidity) before I pulled out my SD card and tried to use it from one Pandora to the other .. if this was properly guarded with a repo-manager popping up a Warning note (it can be configured to do that), I probably would've paid a bit more attention.

This is unrelated to corruption, but I agree the label bug not well handled and ED rushes things a bit too much (should stay in beta longer), and last HF fix should be named 3.1 or something.

We can organize as a community to help ED here, but we have to be very careful not to get into solutions that won't scale for the thousands of users to come. I imagine ED's life is *not* going to get easier if he has to produce Hotfix packs for 4,000 Pandora users by the end of the year ..

 

[notaz]

100% everything can be built using OE, there are no magic binary or other drop-ins.

SGX drivers? DSP Bridge/Link? Bitbake recipe to build the Pandora kernel? Bitbake recipes for ED's control panel/user-accessories? Not trying to be painful, I just want to know where those things are coming from ..

Yes to all, except DSP stuff which we don't use at all right now. Do you want me to link you to the respective recipes?

I will go take my own close, non-generalized, very detailed look .. but last time I did that (a month ago), it was still pretty messy on the ol' GIT server .. so I will have a look at the scene with renewed vigor and see if I can't come up with a workable plan for how I can help.

Just don't start making pessimistic assumptions if something looks suspicious to you, everything is handled with OE tools one way or another.

Can we have a "Base OS Developers/Contributors" mailing list, and a "Repo BETA Users" list, just to get the ball rolling on how to get these instructions formulated and activated? Or, maybe just separate forums for now?

That's something for ED to decide.

 

[torpor]

BTW, all thats needed to get a working public repo *right now* is for the files mentioned here:

http://www.openpandora.org/firmware/images/Angstrom-pandora-xfce-image-glibc-ipk-2010.4-test-20100705-omap3-pandora-testlab/installed-packages.txt

.. to be included in that very same website, somewhere, under a well-defined path.

Then I can add the entry to /etc/opkg/$SOMEFILE.conf, do an "opkg update && opkg upgrade" and see for myself just how much of a headache the repo is going to be .. the advantage to this is that I can get an update to the latest firmware without having to make some special SD card to flash it all from scratch, and if I happen to have a package configuration file (such as those related to my DHCP settings) that I've modified locally, I get a chance to merge/update/backup my local conf file, automatically, without losing it. (The "Burn this image and erase your entire NAND" method doesn't give me that choice, alas .. and I'd have to remember to do this myself: already proven not a good method of not hosing ones system ..)

So, I think, setting up the repo might be easier than you think, especially if notaz and DJWillis have already gotten the build-server-producing-OPK-files-for-the-image stage all sorted out ..

EDIT:

Yes to all, except DSP stuff which we don't use at all right now. Do you want me to link you to the respective recipes?

If they're there, I'll find them myself, as I should by now!

(The DSP stuff would be good to have onboard, btw .. but I'll save that suggestion for the days before Hotfix #4 has to be released, heh heh ..)

Just don't start making pessimistic assumptions if something looks suspicious to you, everything is handled with OE tools one way or another.

Imagine my surprise when I discovered that, in fact, the repo is in fact just a few minutes away from being available .. (See above.)

 

[ivanovic]

Not everyone tends to use internet with the Pandora.

I for example don't really use internet with the Pandora. The main reason being that it would be rather braindead to download the files with something like 100KB/s instead of the 1600KB/s that I get on my desktop system. So far the Pandora is not stable enough for WiFi so that not everyone is able to use it by itself for updating.

In general the hotfixes do work as intermediate solution. Yeah, naming could have been done differently, as in "every changed version gets some letter appended" (Hotfix3, Hotfix3a, Hotfix3b, ...), but in general those hotfixes are a usable solution until ED, DJWillis and Notaz find enough time for setting up and testing a package repository.

EDIT:

Then I can add the entry to /etc/opkg/$SOMEFILE.conf, do an "opkg update && opkg upgrade" and see for myself just how much of a headache the repo is going to be .. the advantage to this is that I can get an update to the latest firmware without having to make some special SD card to flash it all from scratch, and if I happen to have a package configuration file (such as those related to my DHCP settings) that I've modified locally, I get a chance to merge/update/backup my local conf file, automatically, without losing it. (The "Burn this image and erase your entire NAND" method doesn't give me that choice, alas .. and I'd have to remember to do this myself: already proven not a good method of not hosing ones system ..)

Honestly, why do you think that a complete reflash is required for every update? A reflash is most likely needed for a major upgrade, okay. But those hotfixes are not requiring a complete reflash at all. You just have some additional packages installed (using opkg), get some new scripts copied over and are done. Yes, there is no button on your pandora to check for new updates, but eventually we will get there, too. Until then all you need is to have some working firmware, download the hotfix, install it by running the pnd, reboot (in case of kernel changes) and be done.

 

[torpor]

I for example don't really use internet with the Pandora.

Then all you will need to do is download a package of .ipk files which can be packaged as .ZIP for simple unzipping in /media/SOMESD/pandora/updates, or as a PND file that will automatically unpack it, opkg install, etc. Nothing will change for you if we move over to a repo as the primary source of updates - the Repo Manager Person will simply have to produce a 'delta list' of files that have updated and make that list available for you either in a ZIP or PND file.

until ED, DJWillis and Notaz find enough time for setting up and testing a package repository.

Actually, I think things are pretty close. A lot closer than I thought at the beginning of this thread, and for my ignorant rants I may pay dearly in the public sphere, but have nevertheless learned something important: The firmware release page has all the clues: the gang already have the hard part (custom Pandora sources -> build-server -> web site) done, they just haven't published the .ipk files on the firmware server so that we can start doing an update against them .. I will do that as soon as the .ipk files are there and I can add an entry to my /etc/opkg/*conf collection ..

EDIT: Ninja, d'oh!

Honestly, why do you think that a complete reflash is required for every update? A reflash is most likely needed for a major upgrade, okay. But those hotfixes are not requiring a complete reflash at all.

The only way (now) to get the update is to use the firmware files published there and do a full NAND-burning update. If those .ipk files get added to the site, then we have a 'proper' package-based repository that we can just individually update against. Thats the point I'm making - I know of course that we don't *have* to burn the whole NAND, just that the firmware as published is only available in a form that required a full-NAND update .. until the gang put the ipk's up, that is ..

Until then all you need is to have some working firmware, download the hotfix, install it by running the pnd, reboot (in case of kernel changes) and be done.

I suppose for brevity and the sake of a clear position from which to argue you omitted the "Find out there is a Hotfix available, go to the forums to find the link, read every thread about the Hotfix to make sure you have *THE* latest version, download the latest version, re-apply it" steps .. but yeah. No thanks. I want repo, not hotfix. The Pandora can do all the work of updating its Base OS.. the days when I should *have to* pay attention to such detail are well and truly over.

 

[notaz]

BTW, all thats needed to get a working public repo *right now* is for the files mentioned here:

http://www.openpandora.org/firmware/images/Angstrom-pandora-xfce-image-glibc-ipk-2010.4-test-20100705-omap3-pandora-testlab/installed-packages.txt

.. to be included in that very same website, somewhere, under a well-defined path.

There are far more packages lying in the build directory. Any nice way to copy only those on the list from build tmp to some external location? Preferable some rsync-like solution that doesn't overwrite identical files?

 

[ivanovic]

Honestly, why do you think that a complete reflash is required for every update? A reflash is most likely needed for a major upgrade, okay. But those hotfixes are not requiring a complete reflash at all.

The only way (now) to get the update is to use the firmware files published there and do a full NAND-burning update.

The Hotfixes *ARE* an updater. There is *NO* need to reflash doing a NAND-burning update. You have a working pandora, you run the "installer" (aka Hotfix.pnd) and are *done*.

Until then all you need is to have some working firmware, download the hotfix, install it by running the pnd, reboot (in case of kernel changes) and be done.

I suppose for brevity and the sake of a clear position from which to argue you omitted the "Find out there is a Hotfix available, go to the forums to find the link, read every thread about the Hotfix to make sure you have *THE* latest version, download the latest version, re-apply it" steps .. but yeah. No thanks. I want repo, not hotfix. The Pandora can do all the work of updating its Base OS.. the days when I should *have to* pay attention to such detail are well and truly over.

The part I left out is really simple, too. Just check the "support" site on http://www.open-pandora.org. There is no forum dwelling required.

 

[torpor]

I'm not sure I understand you notaz - what do you mean there are 'far more packages lying in the build directory'? Do you mean that you've built more packages than are included in the published firmware image? Thats okay, its to be expected - you won't harm anyone by putting *all* the files out there.

But: the *canonical* list of packages to use is the one in:

http://www.openpandora.org/firmware/images/Angstrom-pandora-xfce-image-glibc-ipk-2010.4-test-20100705-omap3-pandora-testlab/installed-packages.txt

This lists all the ipks that have gone towards making the firmware .image files .. this is *the* list of packages that have been installed into the firmware image, and since the idea is that both a NAND-flash installation or an 'opg upgrade' installation should result in *exactly* the same system, it would be wise to use this .txt file to build a script to copy the files into a public 'repo' tree on some webserver. And then we're most of the way towards being done with setting up the repo!

I'm just looking for details now on how to set up the Packages file - this used to be easy, but the Angstrom/OE docs seem to not focus on what it takes to set this file up for a repo, alas .. *BUT* maybe you've already got it built for you in the build tree, somewhere .. ?

(Our local Angstrom fork is much-modified from the stock, alas, so I can't say for sure that the method we're using to set up the distro these days for internal uses at work is going to work for PandoraOS right now..)

If you get this sorted, tell me the URL and I'll set up one of my Pandoras to "opg update && opgk upgrade" this evening, and we'll see how things progress ..

 

[torpor]

[quote name=""]
The Hotfixes *ARE* an updater. There is *NO* need to reflash doing a NAND-burning update. You have a working pandora, you run the "installer" (aka Hotfix.pnd) and are *done*.
[/quote]

My understanding is that the firmware link includes a 'beta/untested' release of a new image, which won't be whats included in Hotfix #3 .. but I could be wrong here, I'm just going by the filenames and dates/timestamps of what I'm seeing on the server.

The part I left out is really simple, too. Just check the "support" site on http://www.open-pandora.org. There is no forum dwelling required.

STILL: too much human interaction required. Pandora can do it for me, I just need to approve. I don't wanna have to check next week, if there are updates .. the Pandora can remind me. Thats what computers are for: to do work for their human masters. I will not be subjugated to the role of Adminstrator, when what I want to do on my Pandora is Play/Develop/Play!

 

[torpor]

BTW, folks: we should be saying "PandoraOS feed", not "PandoraOS repo" .. I realized this mistake when I went off to find out how to set up a 'repo' when what I really mean is how to set up a 'feed' .. Repo for sources, Feed for end-user packages.

EDIT: and, notaz, I think there is a default openembedded-feed.bb recipe that can be used to build a proper feed (With Packages file) or maybe there is already one customized for the Pandora by DJWillis?

"bitbake whateveritis-feed.bb" should produce something like $BUILD_DIR/tmp/deploy/*/ipk/[openembedded, pandora, something], which can then just be simply copied to a webserver for use as a standard PandoraOS update feed ..

 

[fusion_power]

Hm...
I'm a Windows guy so I have to provoke and ask "WTF is a REPO?" :lol:
But I know hotfixes very well and it looks like they are very easy to install onto the Pandora (don't have a pandora yet). And as more fixes as better. As long as it is easy to install, I would use it. Don't want to deal with all the "Linux stuff", command line or compiling the kernel for yourself...scary!!! h34r:

As an End-User, I simply want that the Pandora just WORKS...nice, fast, clean...nothing more, nothing less. How this is done, I don't care. If you want to summon a Demon and seal him into the Pandora for keep the stuff working, do it.
This Linux "repo" stuff sounds very risky and I guess I can't control what it actualy does. Even if I'm not a Linux guy I want at least to know what's going on when I update the System. But a nice readme that lists all fixes is good for this. ED does this for the Hotfixes and I'm fine with that.

I still don't understand the concept of a "repo" (my english is not good enough to follow this Tech-Talk here) but if it is better/more secure/faster...whatever...than the current system, OK. General software automation is a wide field and could be very important for the Pandora too some day.
Please make a repo/feed for us to test it, torpor. You want one, you should make one. Have fun. But please don't stop the Pandora Devs doing their work, they have to do better things instead of posting here.

 

[Bosbeetle]

Yeah one fun thing is that you mentioned that you'd rather have ED working on a repro/feed instead of fixing some immediate problems users were having (the sound fix, the mount fix, the python fix). This would have led to a month long whining about sound being crap and sd speed being sucky. I think they choose wisely about releasing the hotfixes.

 

[mali]

I think the show is over now and we should let the pros work out a solution. Despite torpor's furious temperament I'm sure he's a nice guy and I'm glad he wants to help setting up a feed.

 

[torpor]

Please make a repo/feed for us to test it, torpor. You want one, you should make one.

What you guys don't seem to understand is that Angstrom, the OS-building system that is used to make Pandora OS, *already has the tools needed to deliver a user-accessible feed*. So the work to get this done is simply to figure out where the feed directory is and copy it to the web server.

I was under the impression that there were *other* problems preventing DJWillis and co from producing this feed - but as we have now discovered, that is not the case. DJWillis has cleaned up the recipes for OE, and the Pandora-specific tools for OE are being used already to deliver the firmware images .. so the team just need to decide to add the feed directory to a publicly accessible webserver: and we are DONE with the feed problem.

I have been complaining about this because it never made sense to me that the team is using OpenEmbedded/Angstrom for building their Base OS images, but *not* using the feed feature. It had been explained to me that the feed wasn't being used because there were problems earlier in the production line - namely that there were components of the Pandora Base OS that were being hand-constructed outside of the OE tool set. This is not the case.

So, it doesn't make any sense *not* to have the standard feed mechanism in place - except that those with the authority (it really should be an OPT core member who decides to do this) simply haven't caught up with this aspect of things .. yet. Now, all thats needed is to take the feed that OE is *already* producing with every build of the Base OS, and add it to the OPT webserver artillery .. and we're firing away!

 

[torpor]

Yeah one fun thing is that you mentioned that you'd rather have ED working on a repro/feed instead of fixing some immediate problems users were having (the sound fix, the mount fix, the python fix).

Actually, I already asked ED to add some fixes to the Hotfix #3 for sound/midi, and so I believe I've more than demonstrated my willingness to play along with Hotfix-style releases .. just that I don't understand *why* we have to have Hotfixes when the feed is really there, already in place, with the build of the Base OS. Now that we know what it was (basically, just needing to know how to use that feed tree), we can stop arguing about it and start setting up our Pandora's to use the official OPT feed as soon as they publish it .. and I hope we can then push fixes out through the feed mechanism for future issues requiring a patch.

 

[Pleng]

Thanks for the hotfixes, Ed

 

[sebt3]

Thanks for the hotfixes, Ed

Thanks Pleng to be a smart guy

 

[porg]

BTW, all thats needed to get a working public repo *right now* is for the files mentioned here:

http://www.openpandora.org/firmware/images/Angstrom-pandora-xfce-image-glibc-ipk-2010.4-test-20100705-omap3-pandora-testlab/installed-packages.txt

.. to be included in that very same website, somewhere, under a well-defined path.

There are far more packages lying in the build directory. Any nice way to copy only those on the list from build tmp to some external location? Preferable some rsync-like solution that doesn't overwrite identical files?

cd path/to/build-dir;
rsync --files-from=path/to/installed-packages.txt --relative
--archive \
--any-other-options-you-need \
. user@feed-server:/path/to/publish-dir

While you are in the testing stage, use --dry-run ! ;-)

 

[EvilDragon]

Yeah one fun thing is that you mentioned that you'd rather have ED working on a repro/feed instead of fixing some immediate problems users were having (the sound fix, the mount fix, the python fix).

Actually, I already asked ED to add some fixes to the Hotfix #3 for sound/midi, and so I believe I've more than demonstrated my willingness to play along with Hotfix-style releases .. just that I don't understand *why* we have to have Hotfixes when the feed is really there, already in place, with the build of the Base OS.

... because the building server has only been setup a few days ago (the same day when I posted the Ready-To-Flash Firmware image information... and that was TWO DAYS AGO!)

Until then, DJWillis used is own one. notaz was nice enough to take the time to set it up on the openpandora.org server, so I just made my first steps creating the new images there.
I wanted to check out the repositories before making them public. I rushed too many things out now as people were always asking (and I was too tired to really pay attention what I was doing), the last thing I want to rush out would be a non-working repository that could theoretically break stuff.
And I surely don't want to make anything public without discussing that with DJWillis and notaz first.