Audio Driver (with bounty!)

[EvilDragon]

The Pyra has some not-so-easy things that need fixing - and I'd like to offer bounties for those!

The audio driver is part of that.

What's the issue with audio on the Pyra?

At the moment, we're using a very basic audio driver, which causes quite a few problems.

* I provides ONE audio device with four channels (two are for the headset and two for the speakers) instead of providing multiple devices with two channels each.
* These channels are even mixed up somehow so that Left / Right are mixed together in the speakers. aTc made some workaround in PulseAudio - but that also doesn't fix this 100% it seems.
* The TWL6040 runs with 96kHz default audio quality. Every audio stream that is NOT 96kHz natively is being SOFTWARE-converted at the moment (which needs a lot of CPU power)
* It also is a bit unstable whereas the stream can suddenly crash and you only hear noise. This especially happens with higher volumes.

How should it be?

The TWL6040 provides hardware acceleration for resampling, etc. It is also able to create multiple audio devices (PCM streams) so that we don't need to mess around with these 4 channels-split-up issue anymore.
That accelaration needs to have a working ABE / AESS firmware. Sadly, I'm not knowledgeable enough to give exact technical details about that.

The problem is that the TI didn't update the firmware / driver anymore since an ancient kernel version - and the firmware formats in Linux has changed since then. So it needs to be ported to a more recent kernel version.

Here's part of a reply from hns:

AFAIR the last almost working version did have a working driver, but firmware file format modern Linux kernels do accept have changed. There had been attempts by Peter Ujfalusi and Marek Belisko (who is much more experienced with Audio than me) to recompile it from source but they did not succeed completely.

AES/AESS integration is available in the latest Letux kernels. But incomplete.
It compiles, does not harm but does not have the expected functionalities (because I don't know how to set up the DAI links and widgets correctly).

And as said above, the AES firmware is not in a good shape.

If you take letux-5.10 as the basis it should work.

Otherwise the latest AESS patch set is here:

Golden Delicious Computers - letux-kernel.git/shortlog

So some work has been done - but for some reaosn, it doesn't work yet.

Here are some more links I was able to dig out:

[Kernel] OMAP5 ABE (Audio Back End) and Pyra Prototype V3

And here's the old driver:

GitHub - omap-audio/linux-audio: Linux audio drivers for OMAP processors.

Linux audio drivers for OMAP processors. Contribute to omap-audio/linux-audio development by creating an account on GitHub.

github.com

ABE / AESS Firmware:

GitHub - omap-audio/abefw: ABE AESS firmware and configuration generation tools.

ABE AESS firmware and configuration generation tools. - GitHub - omap-audio/abefw: ABE AESS firmware and configuration generation tools.

github.com

The bounty

Since this is a pretty important thing to fix in my opinion and it doesn't seem to be that easy, the bounty I'm giving out is quite high.

If someone gets the full ABE / AESS / Driver setup to work, I'm happy to pay 2000 EUR as bounty!
Of course, everyone else here is welcome to add something to the money pile if he / she likes!

Oh, and:
I DID ask at the mailing list first if someone of the persons working originally on the driver / audio setup would be offended if I offer such bounties but I didn't receive any replies.
But once again: If someone is helping out with improving the software already on the Pyra (not just audio stuff) and you would like to receive some payment for your work, let me know.
I'm sure we can set something up here. As the hardware development for the Pyra has been finished, I have some spare money for software development now.

 

[Silent-Hunter]

Is there a reason we aren't using letux-5.10 as the basis for PyraOS?

 

[TrashyMG]

Likely because it's best to stick to a known stable? Also it's not likely going to change the audio issues.

 

[EvilDragon]

Yes, each kernel updste caused various issues which need to be solved before we can switch to them.
aTc is usally trying out the kernels, so I'm not sure what the issue is with this one. Could be non-bootable, 3D not working anymore, etc.

 

[aTc]

It's mostly the sgx drivers that don't work in the newer kernels.
Apparently it's a userspace thing , ( from hns: "The pvrsgx issues are induced by my user-space (old X11 tries to access pvr by wrong ioctl)". ).
Don't know any more details.

Other than that the kernels seem to work fine.
Although I still need to redo the config for them. The config used by letux is very minimal, and doesn't have many drivers and features debian expects to be enabled.

 

[Silent-Hunter]

Would using a newer version of X11 help?

 

[hns]

I'd suggest to base work on letux-5.10 because it already contains all the work I have invested to make AESS work.
Unfortunately I am no longer in a situation where I could help out ED with this topic, even with a bounty (I have to focus my plethora of projects).
Status is:
* a full AESS driver is included in latest letux-kernels (but some setup is disabled because it has to be reworked that the code can be compiled without errors - mainly due to changes in kernel struct components or some macros)
* but firmware load fails even if installed
* main issue is that the format of the binary AES firmware has changed so that the standard firmware loader of Linux does not recognise the old firmware format
* and I have not researched how to build AES firmware from scratch
* Peter Ujfalusi from TI did have it woking some years ago and IMHO he is still available through the Kernel mailing lists and may give some support
* regarding the split of the 4 channels that is challenging since it means modifying the upstream McPDM driver to multiplex multiple DAI links. As far as I understand this, it is just a single 4-channel link because the PDM between OMAP5 and TWL6040 is.
* there may be a chance to cooperate with the Motorola Droid 4 kernel people. This is an omap4 based device which shares the audio architecture. This is quite active on the omap-linux mailing list.

Post automatically merged: May 4, 2021

It's mostly the sgx drivers that don't work in the newer kernels.
Apparently it's a userspace thing , ( from hns: "The pvrsgx issues are induced by my user-space (old X11 tries to access pvr by wrong ioctl)". ).
Don't know any more details.

Other than that the kernels seem to work fine.
Although I still need to redo the config for them. The config used by letux is very minimal, and doesn't have many drivers and features debian expects to be enabled.

I am running Debian with the Letux kernel configs on different devices (since ca. 12 years) so what does Debian expect to be enabled in addition?
And, there is a special pyra_defconfig in the Letux kernel tree which might just need an update.
Regarding SGX it did work a while ago but we need people to permanently test newer releases or it may get broken unnoticed. And if it is broken, we need people to find out why and develop a fix.
(This is one of the points I mean that I can't help ED as good as I'd like: I can't run all the tests for the Pyra every week myself and fix the bugs in short enough time - only the list of known bugs would grow).
My problem with SGX is that I was never able to have a working setup to run SGX tests (besides loading the firmware) but others did report to have a fully working system. But unfortunately they didn't tell me how or did provide enough help to fix my user space This is what I meant by the comment about a mismatch between user-space and kernel. IMHO it needs some config fixes with Weston, X11, EGL, GLES and other components I am not familiar with. Agian, the Droid 4 people may also be interested in. And maybe the N900 community.
Generally it is much easier to fix something if there is a working (even older) reference setup. I.e. some device and kernel and user-space where some recent AESS or SGX is working.

 

[Wally]

@hns there's a bunch of folks with Pyras on EvilDragon's Discord or IRC (freenode #dragonbox-pyra / #dragonbox-dev - aTc only is there) if you'd like to join us or bring some other folks over

Right now the SGX draws a blank, transparent window with a title and border, doesn't do much other than that

 

[aTc]

I am running Debian with the Letux kernel configs on different devices (since ca. 12 years) so what does Debian expect to be enabled in addition?
And, there is a special pyra_defconfig in the Letux kernel tree which might just need an update.
Regarding SGX it did work a while ago but we need people to permanently test newer releases or it may get broken unnoticed. And if it is broken, we need people to find out why and develop a fix.
(This is one of the points I mean that I can't help ED as good as I'd like: I can't run all the tests for the Pyra every week myself and fix the bugs in short enough time - only the list of known bugs would grow).
My problem with SGX is that I was never able to have a working setup to run SGX tests (besides loading the firmware) but others did report to have a fully working system. But unfortunately they didn't tell me how or did provide enough help to fix my user space This is what I meant by the comment about a mismatch between user-space and kernel. IMHO it needs some config fixes with Weston, X11, EGL, GLES and other components I am not familiar with. Agian, the Droid 4 people may also be interested in. And maybe the N900 community.
Generally it is much easier to fix something if there is a working (even older) reference setup. I.e. some device and kernel and user-space where some recent AESS or SGX is working.

Most noticable is the missing filesystem ACL support (systemd complains about this a lot) , and some net filter options not being enabled, so firewall software fails.
The rest is mostly drivers for usb devices, and probably a lot of smaller things to make it more secure.

SGX on newer kernels all fail here on pvrsrvctl:

pvrsrvctl[1958]: /opt/omap5-sgx-ddk-um-linux/bin/pvrsrvctl: SrvInit failed (already initialized?) (err=PVRSRV_ERROR_INIT_FAILURE)
pvrsrvctl[1958]: PVRError): PVRSRVBridgeCall: Failed to access device. Function ID:3223086861 (strerror returns no value.). [0, ]
pvrsrvctl[1958]: PVRError): OpenServices: PVRSRVBridgeCall failed. [0, ]
pvrsrvctl[1958]: PVRError): PVRSRVInitSrvConnect: PVRSRVConnect failed [0, ]
pvrsrvctl[1958]: PVRError): SrvInit: PVRSRVInitSrvConnect failed (4) [0, ]

 

[KidPaddle]

Did we need the newest kernel? Most for us, need a functional kernel and not a recent kernel.

Thomas

 

[EvilDragon]

Did we need the newest kernel? Most for us, need a functional kernel and not a recent kernel.

Thomas

That's why we use an older one. However, some fixes that improves things (including audio driver support) have not been implemented in the one we use, so all those improvements would need to be backported.

 

[hns]

Most noticable is the missing filesystem ACL support (systemd complains about this a lot) , and some net filter options not being enabled, so firewall software fails.

I haven't observed that myself or run into trouble, but if a CONFIG is missing, please let me know to add it to the pyra_defconfig. It is done in minutes if I know.

Did we need the newest kernel? Most for us, need a functional kernel and not a recent kernel.

Well, in an ideal world the latest Debian kernel would simply boot the Pyra fully functional.

That's why we use an older one. However, some fixes that improves things (including audio driver support) have not been implemented in the one we use, so all those improvements would need to be backported.

When starting the GTA04 project many many years ago this did lead to a perfect v2.26.32 kernel. But some time later we were forced by user-space demand to upgrade to v3.12 and made it almost perfect. This did only find user-space acceptance for a while. So we switched the development model to update and follow the latest kernels so that this device is still supported by v5.12 and we can run Debian Bullseye. This would not have been possible if we had sticked to v3.12 and fixed some minor bugs.

Well, you rarely need the newest kernel as a user, but developing and fixing bugs for the newest one and backporting to older ones is usually much easier than the other way round. It gives a more continuous support rather than the need for really big upgrade tasks every 2 or 3 years (where you have to find a team doing it).

The downside is if something can not easily be backported, it will not find someone to invest a lot of time and do it... The same the other way round: if something gets fixed for the "functional kernel" it will unlikely be forward-ported. That is how many "vendor kernels" work. They work when the device starts shipping but they are not maintained after a while. They deliver a "functional kernel" once when the product is new and then they abandon development. I see the Pyra as a long-term project and that needs a better strategy for continuously maintaining it over many years.

And note, the whole Linux kernel project works that way: there is a "newest" (mainline) and there are stable and longterm kernels (currently 4.4, 4.9, 4.14, 4.19, 5.4, 5.10, 5.11, 5.12).

Therefore for long-term maintainance it would be more work for developers not to work on the mainline kernel and provide backports as a side-product. Also, the Linux community and geeks can't help any more for older kernels. They have no spare time (like everyone) and suggest to upgrade the kernel before they can help... And in some cases they have already solved the problem in a newer one we start try to solve in an older one. So it ends in parallel development.

And to be honest, I have the same view. I can only support developers working on the newest kernels. It needs more time for me to share my years of knowledge (e.g. about SGX or ABE/AESS) w.r.t. older kernels. It is like in sports: you have to look forward to win, not backwards...

Obviously, there are better and worse kernel versions to choose as a basis.

Not using one of the longterm/stable/mainline kernels means also that there are no fixes from upstream developers - and they have much more eyes than we have. For example there can come some general OMAP improvements or fixes (e.g. power state management) which are not provided automatically for older kernels or kernels outside the stable/longterm line. So unless someone tracks and picks these changes we would never know to backport that into our older "freezed" version. A specific example: I have recently seen a proposed patch for DWC3 (wich is our USB3 driver and seems to improve VBUS detection). It will likely come in v5.14. But it may not ripple back to e.g. v5.4 because a simple cherry-pick doesn't work.

Another thing came with the v5.12 kernel: the whole omap display subsystem has been reworked (i.e. is already done!) so that we can use a more generic panel driver. E.g. if ED decides or is forced to replace the panel for a next big production batch. This may either need a big kernel upgrade for the older Pyras or maintaining two different kernels. Neither is good. Following mainline this just needs a new panel driver and a new device tree file - because the maintainers have improved the kernel architecture to make our life easier.

So my suggestion is be to base kernel improvements on v5.10 (longterm) or better v5.12 (mainline) but not on any older kernel.

Finally I have not given up the hope that the Pyra would be supported fully by mainline kernels. I.e. also become part of Debian kernels. To make this dream come true only works by developing for the newest kernels. But it is a long-term project needing more energy (i.e. people sending patches upstream and reviewing/debugging them with maintainers).

 

[hns]

SGX on newer kernels all fail here on pvrsrvctl:

pvrsrvctl[1958]: /opt/omap5-sgx-ddk-um-linux/bin/pvrsrvctl: SrvInit failed (already initialized?) (err=PVRSRV_ERROR_INIT_FAILURE)
pvrsrvctl[1958]: PVRError): PVRSRVBridgeCall: Failed to access device. Function ID:3223086861 (strerror returns no value.). [0, ]
pvrsrvctl[1958]: PVRError): OpenServices: PVRSRVBridgeCall failed. [0, ]
pvrsrvctl[1958]: PVRError): PVRSRVInitSrvConnect: PVRSRVConnect failed [0, ]
pvrsrvctl[1958]: PVRError): SrvInit: PVRSRVInitSrvConnect failed (4) [0, ]

Well, this can have thoudsands of reasons.

Which kernel and which config are you using? Which user-space setup? Or in other words: do you have an SD card image or a full description of all steps which I could use to try to reproduce this and analyse e.g. differences to my LetuxOS setup?

 

[KidPaddle]

@hns

Thanks for your explanation. You are right, working and fixing bugs on latest source is much easier than mainting different version of stable releases.

But if something breaks, like loading the firmware and can't be backported, what can we do?

But this discussion is not part of this thread.

Thomas

 

[levi]

Why can't firmware be backported? There should always be some source to the library shim that includes it in the kernel, and you can port that to enable the firmware to be usable on any kernel you like within reason.

 

[EvilDragon]

Why can't firmware be backported? There should always be some source to the library shim that includes it in the kernel, and you can port that to enable the firmware to be usable on any kernel you like within reason.

The firmware is for a really old kernel (2.6.x I think), so it needs to be updated to work with the latest kernel versions. And yes, it is available and I even linked to it in the first post.

 

[Phlyra]

Does this count?

 

[aTc]

Well, this can have thoudsands of reasons.

Which kernel and which config are you using? Which user-space setup? Or in other words: do you have an SD card image or a full description of all steps which I could use to try to reproduce this and analyse e.g. differences to my LetuxOS setup?

sd card image with both 5.6 and 5.12 kernel https://packages.pyra-handheld.com/images/extra/pyra-5.12.1.img.xz
defaults to 5.12 , 5.6 available from the boot menu on the serial port.
5.12.1 kernel is from the letux repo, with letux_lpae_defconfig
sgx userspace is in /opt , started through systemd at /lib/systemd/system/omap5-sgx-ddk-um-linux.service

Meanwhile, i'll take a look at what happens when running a letux os image, haven't tried that in a while.

 

[hns]

sd card image with both 5.6 and 5.12 kernel https://packages.pyra-handheld.com/images/extra/pyra-5.12.1.img.xz
defaults to 5.12 , 5.6 available from the boot menu on the serial port.
5.12.1 kernel is from the letux repo, with letux_lpae_defconfig
sgx userspace is in /opt , started through systemd at /lib/systemd/system/omap5-sgx-ddk-um-linux.service

Meanwhile, i'll take a look at what happens when running a letux os image, haven't tried that in a while.

Thanks!
I've now flashed an SD card and could boot up to the login: prompt.
Now I am stuck with the standard-issue of prebuilt images
What is the user/password to get into the console?

 

[aTc]

Thanks!
I've now flashed an SD card and could boot up to the login: prompt.
Now I am stuck with the standard-issue of prebuilt images
What is the user/password to get into the console?

It should boot into the first run wizard where it asks you to set up a username/password and some other things.
As part of that it disables the root login.
If you've not completed the first-run-wizard, there is only a user:root password:root account.

Also, getting makesd to produce a letux image isn't really cooperating anymore, can't seem to be able to generate a bootable image.
I had been using "sudo DEV=/dev/sdh ./makesd pyra+lc15-4gb" to generate it, which worked fine before. (haven't checked if it's the sd card, or if maybe the uboot files end up at the wrong block on the card)
What are the recommended options to get it to generate a pyra image with a more up to date kernel ?