Hiya folks,
Well, it's been a while since I heard anything about the audio driver, so I wanted to check back what the status is.
I guess AESS couldn't be made to work? We just tried playing some emulation on the Pyra - and PulseAudio needed a whopping 38% CPU power simply to play the audio.
aplay with a 32 bit 4 channel 96000Hz wav takes 0.7-1.0% cpu, so it's pretty clear that audio is TOTALLY unusable without hardware mixing.
So we NEED to get that working. Does anybody know someone who would be capable of that? Even if he needs to be hired and paid, but sound is unusable at the moment.
Am 05.12.2019 um 06:11 schrieb Michael Mrozek EvilDragon@openpandora.org:
Hiya folks,
Well, it's been a while since I heard anything about the audio driver, so I wanted to check back what the status is.
I guess AESS couldn't be made to work?
We know it can likely work, but we don't know how to do it.
If you want to build a kernel without any AESS contaminations, just replace sound/soc/omap/ and sound/soc/ti by upstream code. That should all be code tested by upstream developers and remove the extra audio devices created by our AESS attempt.
We just tried playing some emulation on the Pyra - and PulseAudio needed a whopping 38% CPU power simply to play the audio.
aplay with a 32 bit 4 channel 96000Hz wav takes 0.7-1.0% cpu, so it's pretty clear that audio is TOTALLY unusable without hardware mixing.
So we NEED to get that working. Does anybody know someone who would be capable of that? Even if he needs to be hired and paid, but sound is unusable at the moment.
Well, I could offer myself since I already have a basic understanding on what AESS and ALSA is about and the data sheets. But I am not the über-specialist for Linux Audio. Just a specialist for fixing bugs. So it needs some time to fill the knowledge gaps by studying code and more data sheets.
I stopped voluntarily looking into the audio driver (and TILER) a while ago since I have much higher priority topics. So we are carrying along code where we know it is broken.
How fast I can do something depends on the priority you can give a topic by your budget.
Am Tue, 10 Dec 2019 10:24:03 +0100 hat H. Nikolaus Schaller hns@goldelico.com geschrieben:
Hi,
I guess AESS couldn't be made to work?
We know it can likely work, but we don't know how to do it. If you want to build a kernel without any AESS contaminations, just replace sound/soc/omap/ and sound/soc/ti by upstream code. That should all be code tested by upstream developers and remove the extra audio devices created by our AESS attempt.
Ah, so those extra audio devices that are segfaulting the kernel are actually the ones that try to make use of AESS?
So, if I understand it correctly:
* AESS is supported by TI in older kernel versions. * Newer kernel versions don't support it anymore and work on the driver has been made to fix this. * Right now, the driver loads the firmware and creates these extra audio devices, but accessing the these devices leads to a kernel segfault.
Is that correct so far?
So what's left is trying to find out why the segfault happens when we try to use the AESS firmware, correct?
Has anyone ever tried to compared how an old kernel with AESS works compared to the latest one with crashing AESS?
We just tried playing some emulation on the Pyra - and PulseAudio needed a whopping 38% CPU power simply to play the audio. aplay with a 32 bit 4 channel 96000Hz wav takes 0.7-1.0% cpu, so it's pretty clear that audio is TOTALLY unusable without hardware mixing. So we NEED to get that working. Does anybody know someone who would be capable of that? Even if he needs to be hired and paid, but sound is unusable at the moment.
Well, I could offer myself since I already have a basic understanding on what AESS and ALSA is about and the data sheets. But I am not the über-specialist for Linux Audio. Just a specialist for fixing bugs. So it needs some time to fill the knowledge gaps by studying code and more data sheets. I stopped voluntarily looking into the audio driver (and TILER) a while ago since I have much higher priority topics. So we are carrying along code where we know it is broken. How fast I can do something depends on the priority you can give a topic by your budget.
Well, I can't spend thousands of EUR if this doesn't lead to any results. But as mentioned multiple times, I have no problems paying someone to fix the remaining issues we still have open.
So if you think it's possible for reasonable time / money, let me know.
Am 10.12.2019 um 16:10 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Tue, 10 Dec 2019 10:24:03 +0100 hat H. Nikolaus Schaller hns@goldelico.com geschrieben:
Hi,
I guess AESS couldn't be made to work?
We know it can likely work, but we don't know how to do it. If you want to build a kernel without any AESS contaminations, just replace sound/soc/omap/ and sound/soc/ti by upstream code. That should all be code tested by upstream developers and remove the extra audio devices created by our AESS attempt.
Ah, so those extra audio devices that are segfaulting the kernel are actually the ones that try to make use of AESS?
Yes.
So, if I understand it correctly:
- AESS is supported by TI in older kernel versions.
Yes. At least for OMAP4.
- Newer kernel versions don't support it anymore and work on the driver
has been made to fix this.
Yes. But Kernel APIs have changed over time (functions have been renamed, structs have been reordered, macros and constants changed) and nobody from our team did look to really fix issues. I just did keep the compiler from stopping with errors. So it got more and more broken.
- Right now, the driver loads the firmware
Well it tries, but does not find one. Usually it would then fail completely. I have added a hack that it still continues.
and creates these extra audio devices,
I think they are for intermediate stages of the audio pipeline, similar to the image signal processor (ISP) for the OMAP3 camera.
but accessing the these devices leads to a kernel segfault.
Is that correct so far?
Yes.
So what's left is trying to find out why the segfault happens when we try to use the AESS firmware, correct?
There are likely multiple segfaults or other bugs hidden by it.
Has anyone ever tried to compared how an old kernel with AESS works compared to the latest one with crashing AESS?
Probably no, because there is no known kernel tree with working AESS.
We just tried playing some emulation on the Pyra - and PulseAudio needed a whopping 38% CPU power simply to play the audio. aplay with a 32 bit 4 channel 96000Hz wav takes 0.7-1.0% cpu, so it's pretty clear that audio is TOTALLY unusable without hardware mixing. So we NEED to get that working. Does anybody know someone who would be capable of that? Even if he needs to be hired and paid, but sound is unusable at the moment.
Well, I could offer myself since I already have a basic understanding on what AESS and ALSA is about and the data sheets. But I am not the über-specialist for Linux Audio. Just a specialist for fixing bugs. So it needs some time to fill the knowledge gaps by studying code and more data sheets. I stopped voluntarily looking into the audio driver (and TILER) a while ago since I have much higher priority topics. So we are carrying along code where we know it is broken. How fast I can do something depends on the priority you can give a topic by your budget.
Well, I can't spend thousands of EUR if this doesn't lead to any results. But as mentioned multiple times, I have no problems paying someone to fix the remaining issues we still have open.
So if you think it's possible for reasonable time / money, let me know.
Well, the question is what is reasonable. For each unexpected single bug it takes me usually 3 full working days to identify the reasons and work out and test and debug a fix so that it is acceptable upstream. This may include setting up user space code to be able to test something and as a first step tryng to understand what the code did intend to do with an older kernel and search for the newest APIs doing the same effect with different function calls.
I would estimate that there are ca. 5-10 of this type hidden behind the first one leading to a segfault which makes 15-30 hours full time work.
Maybe someone raises his/her finger and does fix it in less time?
BR, Nikolaus
Am 10.12.2019 um 16:48 schrieb H. Nikolaus Schaller hns@goldelico.com:
Am 10.12.2019 um 16:10 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Tue, 10 Dec 2019 10:24:03 +0100 hat H. Nikolaus Schaller hns@goldelico.com geschrieben:
Hi,
I guess AESS couldn't be made to work?
We know it can likely work, but we don't know how to do it. If you want to build a kernel without any AESS contaminations, just replace sound/soc/omap/ and sound/soc/ti by upstream code. That should all be code tested by upstream developers and remove the extra audio devices created by our AESS attempt.
Ah, so those extra audio devices that are segfaulting the kernel are actually the ones that try to make use of AESS?
Yes.
So, if I understand it correctly:
- AESS is supported by TI in older kernel versions.
Yes. At least for OMAP4.
- Newer kernel versions don't support it anymore and work on the driver
has been made to fix this.
Yes. But Kernel APIs have changed over time (functions have been renamed, structs have been reordered, macros and constants changed) and nobody from our team did look to really fix issues. I just did keep the compiler from stopping with errors. So it got more and more broken.
- Right now, the driver loads the firmware
Well it tries, but does not find one. Usually it would then fail completely. I have added a hack that it still continues.
and creates these extra audio devices,
I think they are for intermediate stages of the audio pipeline, similar to the image signal processor (ISP) for the OMAP3 camera.
but accessing the these devices leads to a kernel segfault.
Is that correct so far?
Yes.
So what's left is trying to find out why the segfault happens when we try to use the AESS firmware, correct?
There are likely multiple segfaults or other bugs hidden by it.
Has anyone ever tried to compared how an old kernel with AESS works compared to the latest one with crashing AESS?
Probably no, because there is no known kernel tree with working AESS.
We just tried playing some emulation on the Pyra - and PulseAudio needed a whopping 38% CPU power simply to play the audio. aplay with a 32 bit 4 channel 96000Hz wav takes 0.7-1.0% cpu, so it's pretty clear that audio is TOTALLY unusable without hardware mixing. So we NEED to get that working. Does anybody know someone who would be capable of that? Even if he needs to be hired and paid, but sound is unusable at the moment.
Well, I could offer myself since I already have a basic understanding on what AESS and ALSA is about and the data sheets. But I am not the über-specialist for Linux Audio. Just a specialist for fixing bugs. So it needs some time to fill the knowledge gaps by studying code and more data sheets. I stopped voluntarily looking into the audio driver (and TILER) a while ago since I have much higher priority topics. So we are carrying along code where we know it is broken. How fast I can do something depends on the priority you can give a topic by your budget.
Well, I can't spend thousands of EUR if this doesn't lead to any results. But as mentioned multiple times, I have no problems paying someone to fix the remaining issues we still have open.
So if you think it's possible for reasonable time / money, let me know.
Well, the question is what is reasonable. For each unexpected single bug it takes me usually 3 full working days to identify the reasons and work out and test and debug a fix so that it is acceptable upstream. This may include setting up user space code to be able to test something and as a first step tryng to understand what the code did intend to do with an older kernel and search for the newest APIs doing the same effect with different function calls.
I would estimate that there are ca. 5-10 of this type hidden behind the first one leading to a segfault which makes 15-30 hours full time work.
Sorry, that was a wishful miscalculation... 3 full working days with 5-10 such issues is ca. 100 - 200 hours.
Maybe someone raises his/her finger and does fix it in less time?
BR, Nikolaus
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Okay... now the only thing to explain is: How do you fit 200 hours into 3 full working days :D
On 10 December 2019 21:40:51 CET, "H. Nikolaus Schaller" hns@goldelico.com wrote:
Am 10.12.2019 um 16:48 schrieb H. Nikolaus Schaller
Am 10.12.2019 um 16:10 schrieb Michael Mrozek
Am Tue, 10 Dec 2019 10:24:03 +0100 hat H. Nikolaus Schaller hns@goldelico.com geschrieben:
Hi,
I guess AESS couldn't be made to work?
We know it can likely work, but we don't know how to do it. If you want to build a kernel without any AESS contaminations, just replace sound/soc/omap/ and sound/soc/ti by upstream code. That should all be code tested by upstream developers and remove the
extra
audio devices created by our AESS attempt.
Ah, so those extra audio devices that are segfaulting the kernel are actually the ones that try to make use of AESS?
Yes.
So, if I understand it correctly:
- AESS is supported by TI in older kernel versions.
Yes. At least for OMAP4.
- Newer kernel versions don't support it anymore and work on the
driver
has been made to fix this.
Yes. But Kernel APIs have changed over time (functions have been
renamed,
structs have been reordered, macros and constants changed) and nobody
from
our team did look to really fix issues. I just did keep the compiler
from
stopping with errors. So it got more and more broken.
- Right now, the driver loads the firmware
Well it tries, but does not find one. Usually it would then fail
completely.
I have added a hack that it still continues.
and creates these extra audio devices,
I think they are for intermediate stages of the audio pipeline,
similar to
the image signal processor (ISP) for the OMAP3 camera.
but accessing the these devices leads to a kernel segfault.
Is that correct so far?
Yes.
So what's left is trying to find out why the segfault happens when
we
try to use the AESS firmware, correct?
There are likely multiple segfaults or other bugs hidden by it.
Has anyone ever tried to compared how an old kernel with AESS works compared to the latest one with crashing AESS?
Probably no, because there is no known kernel tree with working AESS.
We just tried playing some emulation on the Pyra - and PulseAudio needed a whopping 38% CPU power simply to play the audio. aplay with a 32 bit 4 channel 96000Hz wav takes 0.7-1.0% cpu, so it's pretty clear that audio is TOTALLY unusable without hardware mixing. So we NEED to get that working. Does anybody know someone who
would
be capable of that? Even if he needs to be hired and paid, but sound is unusable at the moment.
Well, I could offer myself since I already have a basic
understanding
on what AESS and ALSA is about and the data sheets. But I am not
the
über-specialist for Linux Audio. Just a specialist for fixing bugs. So it needs some time to fill the knowledge gaps by studying code
and
more data sheets. I stopped voluntarily looking into the audio driver (and TILER) a while ago since I have much higher priority topics. So we are carrying along code where we know it is broken. How fast I can do something depends on the priority you can give a topic by your budget.
Well, I can't spend thousands of EUR if this doesn't lead to any results. But as mentioned multiple times, I have no problems paying someone
to
fix the remaining issues we still have open.
So if you think it's possible for reasonable time / money, let me
know.
Well, the question is what is reasonable. For each unexpected single
bug
it takes me usually 3 full working days to identify the reasons and
work
out and test and debug a fix so that it is acceptable upstream. This
may
include setting up user space code to be able to test something and
as a
first step tryng to understand what the code did intend to do with an older kernel and search for the newest APIs doing the same effect
with
different function calls.
I would estimate that there are ca. 5-10 of this type hidden behind
the
first one leading to a segfault which makes 15-30 hours full time
work.
Sorry, that was a wishful miscalculation... 3 full working days with 5-10 such issues is ca. 100 - 200 hours.
Maybe someone raises his/her finger and does fix it in less time?
BR, Nikolaus
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Am 10.12.2019 um 21:47 schrieb Michael Mrozek EvilDragon@openpandora.org:
Okay... now the only thing to explain is: How do you fit 200 hours into 3 full working days :D
By hiring a team of ca. 8-9 people :)
On 10 December 2019 21:40:51 CET, "H. Nikolaus Schaller" hns@goldelico.com wrote:
Am 10.12.2019 um 16:48 schrieb H. Nikolaus Schaller hns@goldelico.com:
Am 10.12.2019 um 16:10 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Tue, 10 Dec 2019 10:24:03 +0100 hat H. Nikolaus Schaller hns@goldelico.com geschrieben:
Hi,
I guess AESS couldn't be made to work? We know it can likely work, but we don't know how to do it. If you want to build a kernel without any AESS contaminations, just replace sound/soc/omap/ and sound/soc/ti by upstream code. That should all be code tested by upstream developers and remove the extra audio devices created by our AESS attempt.
Ah, so those extra audio devices that are segfaulting the kernel are actually the ones that try to make use of AESS?
Yes.
So, if I understand it correctly:
- AESS is supported by TI in older kernel versions.
Yes. At least for OMAP4.
- Newer kernel versions don't support it anymore and work on the driver
has been made to fix this.
Yes. But Kernel APIs have changed over time (functions have been renamed, structs have been reordered, macros and constants changed) and nobody from our team did look to really fix issues. I just did keep the compiler from stopping with errors. So it got more and more broken.
- Right now, the driver loads the firmware
Well it tries, but does not find one. Usually it would then fail completely. I have added a hack that it still continues.
and creates these extra audio devices,
I think they are for intermediate stages of the audio pipeline, similar to the image signal processor (ISP) for the OMAP3 camera.
but accessing the these devices leads to a kernel segfault.
Is that correct so far?
Yes.
So what's left is trying to find out why the segfault happens when we try to use the AESS firmware, correct?
There are likely multiple segfaults or other bugs hidden by it.
Has anyone ever tried to compared how an old kernel with AESS works compared to the latest one with crashing AESS?
Probably no, because there is no known kernel tree with working AESS.
We just tried playing some emulation on the Pyra - and PulseAudio needed a whopping 38% CPU power simply to play the audio. aplay with a 32 bit 4 channel 96000Hz wav takes 0.7-1.0% cpu, so it's pretty clear that audio is TOTALLY unusable without hardware mixing. So we NEED to get that working. Does anybody know someone who would be capable of that? Even if he needs to be hired and paid, but sound is unusable at the moment. Well, I could offer myself since I already have a basic understanding on what AESS and ALSA is about and the data sheets. But I am not the über-specialist for Linux Audio. Just a specialist for fixing bugs. So it needs some time to fill the knowledge gaps by studying code and more data sheets. I stopped voluntarily looking into the audio driver (and TILER) a while ago since I have much higher priority topics. So we are carrying along code where we know it is broken. How fast I can do something depends on the priority you can give a topic by your budget.
Well, I can't spend thousands of EUR if this doesn't lead to any results. But as mentioned multiple times, I have no problems paying someone to fix the remaining issues we still have open.
So if you think it's possible for reasonable time / money, let me know.
Well, the question is what is reasonable. For each unexpected single bug it takes me usually 3 full working days to identify the reasons and work out and test and debug a fix so that it is acceptable upstream. This may include setting up user space code to be able to test something and as a first step tryng to understand what the code did intend to do with an older kernel and search for the newest APIs doing the same effect with different function calls.
I would estimate that there are ca. 5-10 of this type hidden behind the first one leading to a segfault which makes 15-30 hours full time work.
Sorry, that was a wishful miscalculation... 3 full working days with 5-10 such issues is ca. 100 - 200 hours.
Maybe someone raises his/her finger and does fix it in less time?
BR, Nikolaus Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
-- Mit freundlichen Grüßen / Greetings,
Michael Mrozek OpenPandora GmbH Geschäftsführer / CEO: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland / Germany Tel.: +49 841 / 990 5548 http://www.openpandora.de/ http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt eMail: mrozek@openpandora.org
On Di, 2019-12-10 at 16:48 +0100, H. Nikolaus Schaller wrote:
I've replied to your answers below, however, the more I'm reading about the TWL6040 audio stuff, the more I'm beginning to wonder about things...
If I understand things correctly, ABE is responsible for the realtime audio mixing, etc. And also for turning our weird 4-channel setup into proper Stereo Channels.
The last kernel it seems to work is 3.14.
From what I can see here, ABE needs some parameters (even in U-Boot)
setup to properly work: https://github.com/intel/parameter-framework-samples/tree/master/omap-tutori...
And if I'm not mistaken, AESS is the firmware for the ABE? Or what is it? But why should the firmware be kernel dependent? Or WHAT part of the ABE / AESS system is kernel dependent?
Here's a free guide (it's for the OMAP4, but the OMAP5 is very similar only NDA): http://www.ti.com/litv/pdf/swpu245h
Chapter 4 has examples how to play audio with AESS/ABE. It looks lowlevel to me, so directly playing without any ALSA or similar inbetween.
Would this be a start to implement that into an ALSA driver?
Also, there seems to be a TWL6040 driver for ALSA for boards that use ABE: https://github.com/torvalds/linux/blob/master/sound/soc/ti/omap-abe-twl6040....
It's by the same one who's written the normal TWL6040 ALSA driver (without ABE).... but isn't it ABE we want to use?
This is pretty confusing for me, so if someone knows how this is all supposed to work in theory, that would be great, as it would make it easier asking for help.
Ah, so those extra audio devices that are segfaulting the kernel are actually the ones that try to make use of AESS?
Yes.
And is this based on the omap-abe-twl6040 driver? Or the omap-twl6040 (which probably doesn't have ABE support)
So, if I understand it correctly:
- AESS is supported by TI in older kernel versions.
Yes. At least for OMAP4.
According to posts on the TI webboards, it should also work on the OMAP5. Up until kernel 3.14 it seems.
- Newer kernel versions don't support it anymore and work on the
driver has been made to fix this.
Yes. But Kernel APIs have changed over time (functions have been renamed, structs have been reordered, macros and constants changed) and nobody from our team did look to really fix issues. I just did keep the compiler from stopping with errors. So it got more and more broken.
Ah, so it's not really been worked on, just modified so it can at least be compiled and someone can work on fixes from here on. Understood.
What I'm not sure about it is: We've got the TWL6040 audio working. The AESS firmware shouldn't be kernel dependent, as it is just a firmware blob.
What I don't know is how the AESS firmware and the TWL6040 audio need to be combined to provide hardware accelerated audio.
But looking at the examples I mentioned above it seems the communication between the TWL6040 and AESS firmware also isn't kernel dependent.
So what IS affected by the API change? The only thing I can think of would be the ALSA driver of the TWL6040, as the Kernel / ALSA APIs surely have changed.
So is it only an update / rewrite of the ALSA driver we need so it can communicate again with the AESS firmware? Or what is the issue?
Are these the changes we need to check for?
http://man7.org/tlpi/api_changes/#Linux-3.14
So basically, starting with the old working ABE / AESS setup, check the API changes of the kernel and see how it affects the setup and how to update it?
- Right now, the driver loads the firmware
Well it tries, but does not find one. Usually it would then fail completely. I have added a hack that it still continues.
Ah, we don't have the firmware included in our system? Does this still work to create the aess firmware? https://github.com/omap-audio/abefw
So what's left is trying to find out why the segfault happens when we try to use the AESS firmware, correct?
There are likely multiple segfaults or other bugs hidden by it.
Well, a lot of these might go away if the firmware is actually loaded and the API calls have been checked and maybe fixed.
Has anyone ever tried to compared how an old kernel with AESS works compared to the latest one with crashing AESS?
Probably no, because there is no known kernel tree with working AESS.
Hmm... back when we bought the uEVMs, TI was providing fully working images... I think based on Ubuntu 12.04 or similar.
That demo image had a working hardware accelerated X Server and probably a working ABE AESS backend as well.
It was from around the time Kernel 3.14 was current...
Does anyone still have that demo image lying around on an SD Card? It might be a good start to check how it should work.
So if you think it's possible for reasonable time / money, let me know.
Well, the question is what is reasonable. For each unexpected single bug it takes me usually 3 full working days to identify the reasons and work out and test and debug a fix so that it is acceptable upstream. This may include setting up user space code to be able to test something and as a first step tryng to understand what the code did intend to do with an older kernel and search for the newest APIs doing the same effect with different function calls. I would estimate that there are ca. 5-10 of this type hidden behind the first one leading to a segfault which makes 15-30 hours full time work.
Well, sounds like it would be cheaper replacing the TWL6040 by a different dedicated sound chip ;) Now I know why the Pandora used a simple DAC... way less hassle to make it work properly.
Maybe someone raises his/her finger and does fix it in less time?
Unfortunately, there are not many reading the mailing list here and I have no clue who is into the materia so he could work on it faster.
We don't need a fully fledged setup at the beginning. Maybe the examples above are enough to write a very simple but working ALSA driver that can use the ABE to play some audio. The TWL6040 probably has a lot of features we don't even need.
-
H. Nikolaus Schaller -
Michael Mrozek