Contributing to PyraOS

[ToastBucket]

I think the problem that's arisen in this thread is communication.

ToastBucket's original post makes it sound like the official PyraOS kernel is no longer coming from Letux... which seperates Pyra from Letux... which I think what probably irked hns the most. I get the need for a Pyra test kernel that doesn't require formalised submission (clean formatting, clear descriptions etc.) that's just used to get things working before an offical clean patch is sent to Letux; but it didn't really come across that this is what the new repo was for.

Yep. I realize now I made a big mistake not communicating with HNS first.

My main intention was to have this be the main PyraOS kernel. It is based on Letux, but is managed separately from it so it can stay up to date with the most recent Pyra features and fixes. In the meantime, once they’re tested and merged here, HNS can merge them into Letux to make sure they still work with his other devices. Pyra kernel benefits from general Linux updates from Letux, Letux benefits from community Pyra-specific updates. The idea here is not to drop Letux, but to create a copy of Letux where we can focus on what is necessary, then pull things back together.

As ED stated earlier, we are one community, but when it comes to the software, a blurry line appears. Currently, building the kernel requires patching it to add AUFS, patching the generic LPAE defconfig to add some required features, and applying the patched defconfig. Recently, updates to the master branch have broken SGX functionality so that is currently not working. Then, to build U-Boot, a particular fork which fixes RAM timings specific to the Pyra must be used. I have not yet been able to build a working rootfs.

When I first got involved, I saw this as a huge problem. There is no fully working defconfig for the Pyra. All of the repositories are in different places, essentially in unknown states of functionality with no Pyra-related documentation. So my fix for the problem was to take everything and put it in one centralized location where everyone can find it, where it can be focused towards Pyra development and will ALWAYS fully support the Pyra, where it can be documented, and where it can be fully controlled by ED. Once everything is in one place and is stable, a simple, scripted build system can be pulled together for building PyraOS images. The goal here is creating a focused, simple, and stable PyraOS development environment.

So this was my reasoning for doing what I did. As I said earlier, my main intention was to maintain this fork as the official PyraOS kernel, based on and tracking the LetuxOS kernel. This made sense to me from a development perspective, but what I didn’t consider was the community perspective. This was a mistake.

The community is the project, and HNS and Letux are a big part of the community. As this has clearly caused an issue within the community, something needs to change. I am not set in stone with this method, this just made a lot of sense to me at the time. Let’s find something that works best for everybody. It may be this system with some changes to create better integration with Letux workflow. Maybe not. I am open to suggestions.

 

[levi]

Looking at the changes in the pyra kernel maintained by Toast, I'm a little confused. Is the intent ever to push back changes? If so the fact you removed the entirety of the letux folder from the repo will cause extra work for hns to integrate back, and might have an effect on the tests we would like you to run before pushing back. Are there even plans to push the AUFS patches you applied back to upstream ever? I guess you could so the work to rebaseline specific patches to be pushed back up if they ever are destined to go that way, but you haven't said that. I'd be interested to hear what the plans in this direction are going forward.

I understand the benefits of having a more local and less restricted repo for local testing and patch loops, it's just not clear to me how work ever finds its way back upstream, if that's even an intention.

P.S. I'm not sure I would deign this with the label of an 'issue' just yet. We've just figured out something we're doing isn't optimal or at least clearly optimal, so we need to discuss it to work it out. That's how we almost always resolve problems here.

 

[EvilDragon]

Looking at the changes in the pyra kernel maintained by Toast, I'm a little confused. Is the intent ever to push back changes? If so the fact you removed the entirety of the letux folder from the repo will cause extra work for hns to integrate back, and might have an effect on the tests we would like you to run before pushing back.

As far as I'm aware, you can easily do that with git. You can remove stuff you don't need from your own git, but still push back specfic changes for other files back to the original ones.
They're not affected.

Are there even plans to push the AUFS patches you applied back to upstream ever? I guess you could so the work to rebaseline specific patches to be pushed back up if they ever are destined to go that way, but you haven't said that. I'd be interested to hear what the plans in this direction are going forward.

AUFS is special. The original author tried multiple times to get them mainline, but that didn't work:
https://en.wikipedia.org/wiki/Aufs

To my knowledge, overlayFS doesn't add the features we need for dbp.

So we need to add it.

However, that's another reason to have the Pyra Kernel GIT:

We can add such patches, that won't go mainline ever HERE.
They are only needed for the dbp system, so they are not needed for Letux.

So the Pyra kernel can be based on mainline / Letux OS, bringing in all the bugfixes and patches from there but it can also add patches we need that won't go mainline.
At the same time, it can be used to quickly work on and fix other bugs that can then go back to Letux (and hopefully later mainline) once they're properly done.

Often fixes start as quick hacks to try if they work and are then being properly written. The hacks can be on the Pyra git, but those shouldn't be in Letux. Once the hacks are properly coded, they can go to Letux as well.

This means:
If you compile a Letux or mainline kernel, it would basically work but they will be missing stuff like AUFS or some other quick hacks that have not been properly rewritten.
And if you compile the Pyra kernel, you basically have a stable Letux kernel with additional stuff on top that's helpful / needed for the Pyra and needs to be more properly tested.

 

[hns]

Let me cut your answers down and group a little (the forum doesn't handle quotes of quotes well)...

Features and backporting

• newer kernels may always have security fixes or performance improvements, so there is always something newer.
• the Linux community is already backporting such things to the official stable kernels, but sometimes even abandons older stable kernels. Then you are completely on your own.
• AUFS: is only missing in LetuxOS because it has a very uncommon way for installation and can't be rebased on top of mainline kernels and therefore violates the rebase+merge principle of kernel development (I tried several different approaches to fix that but all did fail). If someone helps with this, there is no big reason to not integrate it. The problem why I can't do it myself is that we have no users of AUFS in LetuxOS.
• if a newer kernel breaks something, there should be an archive of all versions (Letux even has a binary archive) so that one can go back one or two steps without hassle. So this can be solved without staying with a specific version or having separate repositories.
• fbdev is now emulated by DRM.

Access to information and repositories

• about getting write access: there is a known mailing list and it should be common sense how to ask for that.
• u-boot, kernel and user-space should be independent binaries so that a developer should not need to build complete new images every 10 minutes. Just replace the changed files (I rsync new kernels to a running device and then reboot; akemndae does even boot everything over USB on OMAP3 devices which is good for automated bisects).
• finding out what source should come from where could be solved by meta-data like Android does (repo sync).
• about a new dev not knowing where to find information: you could also point him to the right places instead of becoming the right place. A simple "howto" page with direct links and some Pyra specific comments should suffice.

Role and purpose of new repository and issue-tracker

• IRC and separate repo seems now to be described as a temporary workaround between ToastBucket and aTc until he has hardware to test completely locally. That differs quite much from the initial announcement of a maintained fork.
• IMHO the "nothing happend" was not because we urgently need different platforms or forks. Nothing happened because lack of devoted developers with hardware.
• issue tracker: yes, ToastBucked has copied the issues from the LetuxOS Issues list, but why not simply answer to the original? What is the duplication good for? IMHO, it makes it more difficult for users to know where to place issues. In the worst case we have two identical issues and nobody works on it. Or two independent developers come up with different solutions.

Communication tools IRC vs. Mail

• IRC is only fast if people sit at the computer and are willing to respond within seconds (this may be true for software developers working full-time on a single project).
• On the other side E-Mail is usually delivered within minutes and can also be quickly answered.
• Discussions on IRC may also last hours or days, especially if someone is not available. So IRC is only faster if some conditions are met.
• Having things fixed faster seems to be good. If the fixes are done right.
• That the first patches come now very fast is not because we were missing a separate playground. It is because there was no developer looking at them, until ToastBucket arrived. I personally know all the issues but can not devote the required time (unless someone would pay for it so that I can abandon other tasks).
• IRC: which IRC are you using? I myself find it very confusing to find the right one and use it (is it for example somewhere on irclog.whitequark.org which I use to follow some discussions like Neo900)... It seems to be equally hidden like you think the LetuxOS information is.

Goals of LetuxOS and PyraOS - single or two communities/parties

• you claim LetuxOS has completely different goals: well, it has the goal to best support the Pyra hardware. This seems to be that they have a big overlap. What else do you expect?
• the problem is that we are talking about two communities/parties ("we" and "you") but also say that we don't want to have them...
• And setting up two instances of collaboration tools which need "sync" doesn't help collaboration, IMHO.

Ok, still too long

Long discussion short: I can only invite to actively contribute to LetuxOS (incl. Pyra) and use the existing infrastructure (and even suggest improvements) rather than running separate repository and issue tracker and let LetuxOS completely independently decide what and when to pick and work on. As said the risk is that despite our current believes and promises things will run apart. The argument after it has happened will be that it would have to be too much work to keep things together and it was more important to be quick.
[doublepost=1564587981,1564587476][/doublepost]

We can add such patches, that won't go mainline ever HERE.
They are only needed for the dbp system, so they are not needed for Letux.

Often fixes start as quick hacks to try if they work and are then being properly written. The hacks can be on the Pyra git, but those shouldn't be in Letux. Once the hacks are properly coded, they can go to Letux as well.

You describe exactly the concept of how Letux is run

We can add such patches, that won't go mainline ever to Letux.
They are only needed for the dbp (pvr/sgx, touch precalibration, nubs, ...) system, so they are not needed for Mainline.

Often fixes start as quick hacks to try if they work and are then being properly written. The hacks can be on the Letux git, but those shouldn't be in Mainline. Once the hacks are properly coded, they can go to Mainline as well.​

​

 

[levi]

Hmm, so can AUFS live in Letux or not?

I can see the benefits of IRC and email lists for different cadences of development. You publish a kernel via the mailing list, and any testing done over the next week or so is all valid to be reported back. IRC is more like remote paired programming, where stuff happens more or less concurrently.

All our IRC channels are on freenode as far as I know. #dragonbox-pyra is the user focussed one I lurk in when I can dedicate the time, and there's another development focussed on I forget the name of. The only problem is that while I suspect some users maintain their own logs of happenings, there are no public logs. Maybe though, now we can watch git commits and issues to the pyra-only repo as well as upstream, we can see the results of discussions there at sufficient granularity

 

[Silent-Hunter]

What is missing from OverlayFS that is needed?

 

[hns]

If you compile a Letux or mainline kernel, it would basically work but they will be missing stuff like AUFS or some other quick hacks that have not been properly rewritten.
And if you compile the Pyra kernel, you basically have a stable Letux kernel with additional stuff on top that's helpful / needed for the Pyra and needs to be more properly tested.

Missing stuff like AUFS source can be integrated to LetuxOS git (but probably disabled by default) and if you have your pyra_defconfig it can be enabled there.
So it boils down to that you just need your pyra_defconfig in the Letux tree?

Yep. I realize now I made a big mistake not communicating with HNS first.

My main intention was to have this be the main PyraOS kernel. It is based on Letux, but is managed separately from it so it can stay up to date with the most recent Pyra features and fixes. In the meantime, once they’re tested and merged here, HNS can merge them into Letux to make sure they still work with his other devices. Pyra kernel benefits from general Linux updates from Letux, Letux benefits from community Pyra-specific updates. The idea here is not to drop Letux, but to create a copy of Letux where we can focus on what is necessary, then pull things back together.

As ED stated earlier, we are one community, but when it comes to the software, a blurry line appears. Currently, building the kernel requires patching it to add AUFS, patching the generic LPAE defconfig to add some required features, and applying the patched defconfig. Recently, updates to the master branch have broken SGX functionality so that is currently not working. Then, to build U-Boot, a particular fork which fixes RAM timings specific to the Pyra must be used. I have not yet been able to build a working rootfs.

When I first got involved, I saw this as a huge problem. There is no fully working defconfig for the Pyra. All of the repositories are in different places, essentially in unknown states of functionality with no Pyra-related documentation. So my fix for the problem was to take everything and put it in one centralized location where everyone can find it, where it can be focused towards Pyra development and will ALWAYS fully support the Pyra, where it can be documented, and where it can be fully controlled by ED. Once everything is in one place and is stable, a simple, scripted build system can be pulled together for building PyraOS images. The goal here is creating a focused, simple, and stable PyraOS development environment.

So this was my reasoning for doing what I did. As I said earlier, my main intention was to maintain this fork as the official PyraOS kernel, based on and tracking the LetuxOS kernel. This made sense to me from a development perspective, but what I didn’t consider was the community perspective. This was a mistake.

The community is the project, and HNS and Letux are a big part of the community. As this has clearly caused an issue within the community, something needs to change. I am not set in stone with this method, this just made a lot of sense to me at the time. Let’s find something that works best for everybody. It may be this system with some changes to create better integration with Letux workflow. Maybe not. I am open to suggestions.

Sorry, the "show new posts" sometimes skips the most important things! Therefore I missed it before answering later posts.

This is a good clarification. Thank you!

Focussing on things is very ok and I have no problem if a playground clone is used as described here. Everyone has such a thing on his local machine... It is just not common to have it too public

Yes, the issue with RAM parameters in U-Boot is also an issue I have. I am not sure if the Letux U-Boot repo has the right set of parameters or not. It seems to work with the one 4GB board I have, but I don't know if it is the best one and/or I am just lucky (there are parameter variations in the DDR3 chips).

Patching AUFS can be prepared for letux-4.19-next (I already have done it). It will just not be configured by the letux_defconfig (unless someone comes and asks to have it configured by default).

SGX is our troubled child... No idea if we ever get it properly and completely working. Unfortunately even a cross-platform initiative involving several kernel maintainers has stalled a little: https://github.com/topics/pvrsgx I tried to get it into a shape that we can dare to submit it upstream to staging which would improve collaboration even beyond Letux. But it is too early to do that. It compiles and partially runs but fails to download the firmware. If it did download firmware, I would say: this is already good enough... Optimal would be if we could demonstrate the kmscube.

Regarding different defconfig needs, there is a similar topic with the Replicant kernel builds of Letux. The letux_defconfig does not enable some Android-specific features (e.g. lowmemorykiller) because they interfere with e.g. Debian. Therefore we have a script that takes the letux_defconfig and modifes the CONFIG options as needed. Then one can do a make letux_defconfig to build a Replicant specific kernel.
This is where the "one kernel binary for everyone" approach comes to its limitations.

A similar approach could be done for the Pyra if we just want to maintain a handful of differences. Alternatively we can add a pyra_defconfig - like there is an omap2plus_defconfig coming from upstream, but it is more difficult to keep in sync with upstream changes to the list of Kconfigs.

For the centralized and Pyra focussed documentation topic, I would suggest to add pages to the Pyra wiki. IMHO there can be a description which repos to clone and which commands to run to get a Pyra variant mixed from all components. This does not require to have everything in one repo.

Thanks again for clarification.
[doublepost=1564593115,1564592936][/doublepost]

What is missing from OverlayFS that is needed?

The answer I got from an earlier discussion is that it does not support mixing arbitrary file system types. And the combination needed for DPB isn't supported.
So there are two solutions:
* merge AUFS
* ask OverlayFS maintainers to fix the missing combination (if it is technically possible at all - I don't know)

 

[ToastBucket]

Looking at the changes in the pyra kernel maintained by Toast, I'm a little confused. Is the intent ever to push back changes? If so the fact you removed the entirety of the letux folder from the repo will cause extra work for hns to integrate back, and might have an effect on the tests we would like you to run before pushing back.

The Letux directory was removed because aTc has build errors when building the kernel package if it’s present. I haven’t been able to investigate this issue yet. I have no test platform, so at the moment I just build the zImage to make sure there are no errors before pushing for aTc to test. This commit can be easily removed or ignored when pushing fixes back to Letux.

 

[EvilDragon]

Missing stuff like AUFS source can be integrated to LetuxOS git (but probably disabled by default) and if you have your pyra_defconfig it can be enabled there.
So it boils down to that you just need your pyra_defconfig in the Letux tree?

Sorry, the "show new posts" sometimes skips the most important things! Therefore I missed it before answering later posts.

Hm, it doesn't do that usually. Unless you're answering while someone else posted something, but then it shows "Show new posts" while you're typing your answer

SGX is our troubled child... No idea if we ever get it properly and completely working. Unfortunately even a cross-platform initiative involving several kernel maintainers has stalled a little: https://github.com/topics/pvrsgx I tried to get it into a shape that we can dare to submit it upstream to staging which would improve collaboration even beyond Letux. But it is too early to do that. It compiles and partially runs but fails to download the firmware. If it did download firmware, I would say: this is already good enough... Optimal would be if we could demonstrate the kmscube.

I'm a bit puzzled here. To my knowledge, the SGX was at least working fine sometime last year without TILER?

When did it break?

 

[hns]

I'm a bit puzzled here. To my knowledge, the SGX was at least working fine sometime last year without TILER?
When did it break?

I never had it working with my setup. AFAIR it mentioned here on the forum that someone having it running on the OMAP5EVM. Ca. 3 years ago?

 

[levi]

Hm, it doesn't do that usually. Unless you're answering while someone else posted something, but then it shows "Show new posts" while you're typing your answer

I guess he missed the little note saying 'x more messages' at the bottom of the page when you're reading, if the new posts have pushed over the page limit. It'll show you the last posts on the page you were reading, but you have to manually hit the next page to see the rest.

 

[hns]

Hm, it doesn't do that usually. Unless you're answering while someone else posted something, but then it shows "Show new posts" while you're typing your answer

I think I got it. If I click on a title in "Last Replies" on the forum main page it jumps there and wipes out how far I did read earlier posts in that thread. So I might miss that there was something between last reading and clicking on "Last Replies", especially if the latest post is on a new page. Well, if you would ask me how that could be done better, I have no idea...

 

[ptitSeb]

My 2 cts is that, even if LetuxOS needs to keep up with upstream and add all new kernel revision, I think for Pyra, we should concentrate on only 1 Kernel version, and focus all energy on this one only. Once everything work, back/forth porting can be done, but before that, newer kernel is just noise that potentialy disturb what need to be done for the Pyra.

 

[levi]

Oh, right. Try using this interface instead which uses your login to always take you to the post after the last read one:
https://pyra-handheld.com/boards/find-new/posts

Edit: @hns

 

[ptitSeb]

I'm a bit puzzled here. To my knowledge, the SGX was at least working fine sometime last year without TILER?

When did it break?

On my side, last time I worked on that when my preprototype worked was with some 4.19.0. SGX was working, I could have offscreen (in PBuffer) GLES rendering, but for other mode there was some conflict with TILER that prevent the creation of the Framebuffer. zmatt had some idea to fix that. Then my preprototype broke and lost the thread of what was going on. In the mean time, countless new kernel revision pops out, each with random stuff working or not, making it even more complicated to follow.

 

[EvilDragon]

I never had it working with my setup. AFAIR it mentioned here on the forum that someone having it running on the OMAP5EVM. Ca. 3 years ago?

My last known status is that it was working fine with our OS, just TILER breaks it and zmatt knows why and how to fix it.
I think Wally used it last year on the EVM?

 

[ToastBucket]

SGX is our troubled child... No idea if we ever get it properly and completely working. Unfortunately even a cross-platform initiative involving several kernel maintainers has stalled a little: https://github.com/topics/pvrsgx I tried to get it into a shape that we can dare to submit it upstream to staging which would improve collaboration even beyond Letux. But it is too early to do that. It compiles and partially runs but fails to download the firmware. If it did download firmware, I would say: this is already good enough... Optimal would be if we could demonstrate the kmscube.

SGX is one of my bigger concerns and one of the reasons I decided to fork. These drivers are barely maintained and poorly at that. They only work with very specific versions of the Linux kernel. Because of this, frequently updating the kernel is a bit of an issue.

I plan on using repo to handle management of the build system. This will allow us to configure the manifest to check out particular commits which we know work, so particular commits from the Letux kernel could be used for a specified PyraOS version. However, this doesn’t solve the issue as the SGX will need to be repaired (if even possible) each time the Letux kernel is updated from mainline before it can actually be used. As the Pyra is primarily a gaming device, SGX seems like a very important feature to me.

Another option is to allow users to choose (using extlinux or their own custom image) between an up-to-date kernel with maybe working SGX or an older kernel with definitely working SGX.

 

[EvilDragon]

On my side, last time I worked on that when my preprototype worked was with some 4.19.0. SGX was working, I could have offscreen (in PBuffer) GLES rendering, but for other mode there was some conflict with TILER that prevent the creation of the Framebuffer. zmatt had some idea to fix that. Then my preprototype broke and lost the thread of what was going on. In the mean time, countless new kernel revision pops out, each with random stuff working or not, making it even more complicated to follow.

We're still using 4.19 on the Pyra, but yeah, that confirms it that it's basically working.

 

[gpb]

@ToastBucket I think it’s a pretty good idea to have an automated build system for the firmware, which ED is shipping.
I think ultimately we all (you, hns and all of our community) have the same goal: full mainline u-Boot and Linux support for Pyra.
First step is a build System. Next step would be a workflow to stabilize a shipping Kernel while getting small patches upstream. Without Hardware it will be hard for you, to test your refactorings, but hopefully that problem will be solved soon.
I did something similar for an I.MX53 powered device [1]. Building u-Boot, Linux and Debian with multistrap in a Docker gitlab-runner. We even boot test everything in real Hardware.
Credits should go to Robert C Nelson [2] where we picked up the idea. Of course kernel-ci integration should always be the far goal.

[1]https://github.com/Beckhoff/CX9020
[2]https://www.digikey.com/eewiki/display/linuxonarm/BeagleBone+Black

 

[hns]

On my side, last time I worked on that when my preprototype worked was with some 4.19.0. SGX was working, I could have offscreen (in PBuffer) GLES rendering, but for other mode there was some conflict with TILER that prevent the creation of the Framebuffer. zmatt had some idea to fix that. Then my preprototype broke and lost the thread of what was going on. In the mean time, countless new kernel revision pops out, each with random stuff working or not, making it even more complicated to follow.

No problem. 4.19.0 only moved to 4.19.63

My 2 cts is that, even if LetuxOS needs to keep up with upstream and add all new kernel revision, I think for Pyra, we should concentrate on only 1 Kernel version, and focus all energy on this one only. Once everything work, back/forth porting can be done, but before that, newer kernel is just noise that potentialy disturb what need to be done for the Pyra.

Well, in most cases something is broken/missing in all kernel versions. So solving it for one is often good for all (older and newer) and porting back and forth can be done (and should be immediately).

SGX is one of my bigger concerns and one of the reasons I decided to fork. These drivers are barely maintained and poorly at that. They only work with very specific versions of the Linux kernel. Because of this, frequently updating the kernel is a bit of an issue.

I have spent years with the SGX code The kernel updates are not the major problem (sometimes a function gets another parameter, sometimes a wrapper etc.) and occur only on major versions. We have a handful of patches in LetuxOS that fix such kernel API differences up to 5.3 so that it compiles and partially runs (but not completely). Take a look at: http://git.goldelico.com/?p=letux-kernel.git;a=shortlog;h=refs/heads/letux/latest-pvr
The major problem are closed source user-space libraries which I am not able to get working and interacting with X11 (or even without).
My kernel work (latest-pvr) would urgently need a working reference... I have one with 3.12 for the OMAP3 with a very old SGX version... But that doesn't help much.
So if someone has a recipe to make sgx work on omap5 (without tiler is not an issue now) on any 4.19 kernel, please share all details. That would allow to do a big step forward.