Hi, I have worked a little on our patch set for omap-abe-twl6040 and AESS and disabled some features that make trouble when no firmware can be loaded.
Now I have a hack that does not break booting and ends up in a sound card with ca. 7 subcards. What is strangely missing, are all mixer settings.
This version has the same problems with loading firmware as Marek reported before.
Most likely it is that firmware loading is a standard "topology" feature of sound/soc/soc-topology.c which defines the header format and we know that it has changed a while ago. So prebuilt binaries may simply be incompatible.
Well, to be honest, I understand less than 10% of the code and did only analyse stack traces to find out why there were NULL pointer dereferences and other issues. And find a quick work-around (mostly by disabling offending code).
So I'd like to ask as many people as possible to dig into the code and review and send patches. Cooperation seems to be the only way to get this aess audio beast tames and working for us...
I don't know if there is an official Linux mailing list where this could also be discussed. So if you know one, please forward.
BR, Nikolaus
Here is the patch set on top of mainline:
http://git.goldelico.com/?p=letux-kernel.git;a=shortlog;h=refs/heads/work/hn...
And here is a complete letux kernel tree with all patches plus some more for debugging printk:
http://git.goldelico.com/?p=letux-kernel.git;a=shortlog;h=refs/heads/work-ae...
And I have collected a lot of questions, e.g. about legacy things, missing mcpdm-abe DAI etc.:
http://git.goldelico.com/?p=letux-kernel.git;a=blob;f=sound/soc/omap/omap-ab...
* are omap_abe_hw_params() and omap_abe_mcpdm_hw_params() the same? then keep upstream omap_abe_hw_params() only * what should we do with OMAP4_SDP and tps6130x stuff? * the extension to handle more DMICs seems not to be upstream! * support for spdif-dit unclear * what is this "legacy" stuff? Is this still needed for anything? * error path in probe() and to err_unregister isn't sane * here, we have better error messages than upstream * dmic_codec_dev is duplicated * there is a problem with "mcbsp-abe" dai_link which is defined nowhere else * no support for McBSP3? * how does this twl6040_dapm_widgets[] relate to the one defined in sound/soc/codecs/twl6040.c? If at all... * why do we define mcbsps and dai links in code; shouldn't they be by DT?
Am 27.08.2018 um 19:55 schrieb H. Nikolaus Schaller hns@goldelico.com:
Hi, I have worked a little on our patch set for omap-abe-twl6040 and AESS and disabled some features that make trouble when no firmware can be loaded.
Now I have a hack that does not break booting and ends up in a sound card with ca. 7 subcards. What is strangely missing, are all mixer settings.
This version has the same problems with loading firmware as Marek reported before.
Most likely it is that firmware loading is a standard "topology" feature of sound/soc/soc-topology.c which defines the header format and we know that it has changed a while ago. So prebuilt binaries may simply be incompatible.
Well, to be honest, I understand less than 10% of the code and did only analyse stack traces to find out why there were NULL pointer dereferences and other issues. And find a quick work-around (mostly by disabling offending code).
So I'd like to ask as many people as possible to dig into the code and review and send patches. Cooperation seems to be the only way to get this aess audio beast tames and working for us...
I don't know if there is an official Linux mailing list where this could also be discussed. So if you know one, please forward.
I should add that despite the subject of this mail, this patch set should work on all OMAP5 boards (aess setup is done in omap-board-common.dtsi).
BR, Nikolaus
Here is the patch set on top of mainline:
http://git.goldelico.com/?p=letux-kernel.git;a=shortlog;h=refs/heads/work/hn...
And here is a complete letux kernel tree with all patches plus some more for debugging printk:
http://git.goldelico.com/?p=letux-kernel.git;a=shortlog;h=refs/heads/work-ae...
And I have collected a lot of questions, e.g. about legacy things, missing mcpdm-abe DAI etc.:
http://git.goldelico.com/?p=letux-kernel.git;a=blob;f=sound/soc/omap/omap-ab...
- are omap_abe_hw_params() and omap_abe_mcpdm_hw_params() the same? then keep upstream omap_abe_hw_params() only
- what should we do with OMAP4_SDP and tps6130x stuff?
- the extension to handle more DMICs seems not to be upstream!
- support for spdif-dit unclear
- what is this "legacy" stuff? Is this still needed for anything?
- error path in probe() and to err_unregister isn't sane
- here, we have better error messages than upstream
- dmic_codec_dev is duplicated
- there is a problem with "mcbsp-abe" dai_link which is defined nowhere else
- no support for McBSP3?
- how does this twl6040_dapm_widgets[] relate to the one defined in sound/soc/codecs/twl6040.c? If at all...
- why do we define mcbsps and dai links in code; shouldn't they be by DT?
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Hi,
AESS / Audio is one of the biggest remaining issues we have on the Pyra. Everything else seems to work good enough for a start now, but missing audio on a gaming handheld is a bit annoying :)
With some kernel version, speaker output stopped completely, but Nikolaus is currently looking into what changed with each kernel version and when exactly it stopped working to get this to work again, at least.
This would make audio work again, but it would still have the following issues:
* Weird setup of sound devices: You would expect that the system would provide different audio outputs (speakers, headset) to the Linux system, but AFAIK, for some weird reason, there's a 4-channel device where the headset is channel 1 and 2 and the speakers are channel 3 and 4. This should be fixed, so that Pulseaudio shows different outputs (like on normal Linux systems) so you can move around your streams where you want them to have.
* AESS... well, not mandatory for a first release, but in the future, it should work, as software mixing just unnecessarily needs CPU power.
@Marek: I know you've been working on this, question is: Are you still doing something here? Or have you given up for now?
@All: If no one wants to work on that (or CAN work on that), I'm thinking of hiring a team that could do this in fulltime. Nikolaus pointed me to these guys: https://bootlin.com/engineering/technical-support-and-consulting/
Does anyone have experience with them or have any other reliable freelancers by hand which I could hire for that.
Geez, I didn't think the TWL6040 would be SO hard to get to work properly...
Thanks in advance :)
hi Michael, On Mon, Oct 29, 2018 at 12:44 PM Michael Mrozek EvilDragon@openpandora.org wrote:
Hi,
AESS / Audio is one of the biggest remaining issues we have on the Pyra. Everything else seems to work good enough for a start now, but missing audio on a gaming handheld is a bit annoying :)
With some kernel version, speaker output stopped completely, but Nikolaus is currently looking into what changed with each kernel version and when exactly it stopped working to get this to work again, at least.
This would make audio work again, but it would still have the following issues:
- Weird setup of sound devices: You would expect that the system would provide
different audio outputs (speakers, headset) to the Linux system, but AFAIK, for some weird reason, there's a 4-channel device where the headset is channel 1 and 2 and the speakers are channel 3 and 4. This should be fixed, so that Pulseaudio shows different outputs (like on normal Linux systems) so you can move around your streams where you want them to have.
- AESS... well, not mandatory for a first release, but in the future, it should
work, as software mixing just unnecessarily needs CPU power.
@Marek: I know you've been working on this, question is: Are you still doing something here? Or have you given up for now?
I've given up because changes would need a lot of time as sound firmware loading was changed rapidly.
@All: If no one wants to work on that (or CAN work on that), I'm thinking of hiring a team that could do this in fulltime. Nikolaus pointed me to these guys: https://bootlin.com/engineering/technical-support-and-consulting/
They are very good but I expect they will not be cheap :)
Does anyone have experience with them or have any other reliable freelancers by hand which I could hire for that.
Geez, I didn't think the TWL6040 would be SO hard to get to work properly...
Thanks in advance :)
-- Greetings,
Michael Mrozek
OpenPandora GmbH CEO: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Germany Tel.: +49 841 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
BR,
marek
Am Montag, den 29.10.2018, 12:53 +0100 schrieb Belisko Marek:
Hi,
@Marek: I know you've been working on this, question is: Are you still doing something here? Or have you given up for now?
I've given up because changes would need a lot of time as sound firmware loading was changed rapidly.
Thanks for the information!
Well, do you know when firmware loading changed? Because right now, it seems Kernel 4.16/4.17 broke the audio system even without AESS and 4.16 also broke TILER.
With 4.15, all this was still working fine - so if your patch would work with 4.15, I am thinking of using 4.15 for a release. Once it's out, we have more devs who can work on future kernel releases.
https://bootlin.com/engineering/technical-support-and-consulting/
They are very good but I expect they will not be cheap :)
On Mon, Oct 29, 2018 at 1:22 PM Michael Mrozek EvilDragon@openpandora.org wrote:
Am Montag, den 29.10.2018, 12:53 +0100 schrieb Belisko Marek:
Hi,
@Marek: I know you've been working on this, question is: Are you still doing something here? Or have you given up for now?
I've given up because changes would need a lot of time as sound firmware loading was changed rapidly.
Thanks for the information!
Well, do you know when firmware loading changed? Because right now, it seems Kernel 4.16/4.17 broke the audio system even without AESS and 4.16 also broke TILER.
With 4.15, all this was still working fine - so if your patch would work with 4.15, I am thinking of using 4.15 for a release. Once it's out, we have more devs who can work on future kernel releases.
Well I cannot remeber which exact version but TI is using 3.15 for old AESS FW.
https://bootlin.com/engineering/technical-support-and-consulting/
They are very good but I expect they will not be cheap :)
-- Greetings,
Michael Mrozek
OpenPandora GmbH CEO: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Germany Tel.: +49 841 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
BR,
marek
On 29.10.18 13:30, Belisko Marek wrote:
On Mon, Oct 29, 2018 at 1:22 PM Michael Mrozek EvilDragon@openpandora.org wrote:
Am Montag, den 29.10.2018, 12:53 +0100 schrieb Belisko Marek:
Hi,
@Marek: I know you've been working on this, question is: Are you still doing something here? Or have you given up for now?
I've given up because changes would need a lot of time as sound firmware loading was changed rapidly.
Thanks for the information!
Well, do you know when firmware loading changed? Because right now, it seems Kernel 4.16/4.17 broke the audio system even without AESS and 4.16 also broke TILER.
With 4.15, all this was still working fine - so if your patch would work with 4.15, I am thinking of using 4.15 for a release. Once it's out, we have more devs who can work on future kernel releases.
Well I cannot remeber which exact version but TI is using 3.15 for old AESS FW.
https://bootlin.com/engineering/technical-support-and-consulting/
They are very good but I expect they will not be cheap :)
I can second this.
And here are two similar companies from Germany: https://www.pengutronix.de/de/ https://linutronix.de/
Beware, if I remember correctly they are not best friends ;-)
Regards, Patrick
Hi,
Am 29.10.2018 um 13:22 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Montag, den 29.10.2018, 12:53 +0100 schrieb Belisko Marek:
Hi,
@Marek: I know you've been working on this, question is: Are you still doing something here? Or have you given up for now?
I've given up because changes would need a lot of time as sound firmware loading was changed rapidly.
Thanks for the information!
Well, do you know when firmware loading changed? Because right now, it seems Kernel 4.16/4.17 broke the audio system even without AESS and 4.16 also broke TILER.
Are you sure? For me, 4.19-rc1 broke the TILER and it is fixed since 4.19-rc6.
And I have not seen any difference between 4.15 and 4.16 for display.
With 4.15, all this was still working fine - so if your patch would work with 4.15, I am thinking of using 4.15 for a release.
Once it's out, we have more devs who can work on future kernel releases.
Just wait. Maybe, it is just one (missing?) patch which fixes audio for 4.19...
But beware: 4.15 has several bugs and missing features (e.g. charger, OTG mode, etc.) that have been fixed between 4.16 and 4.19.
So it could happen that devs focus on and waste time fixing bugs a second time in 4.15 that have already been fixed up to 4.19...
BR, Nikolaus
Am Montag, den 29.10.2018, 13:31 +0100 schrieb H. Nikolaus Schaller:
Hi,
Well, do you know when firmware loading changed? Because right now, it seems Kernel 4.16/4.17 broke the audio system even without AESS and 4.16 also broke TILER.
Are you sure? For me, 4.19-rc1 broke the TILER and it is fixed since 4.19-rc6. And I have not seen any difference between 4.15 and 4.16 for display.
At least with 4.15 it already shows and rotates the console output when booting and that didn't work anymore with 4.17 (I'm not sure we've tested 4.16).
With 4.15, all this was still working fine - so if your patch would work with 4.15, I am thinking of using 4.15 for a release. Once it's out, we have more devs who can work on future kernel releases.
Just wait. Maybe, it is just one (missing?) patch which fixes audio for 4.19...
Maybe, but right now, 4.15 seems to work best for us, at least regarding TILER and audio.
But beware: 4.15 has several bugs and missing features (e.g. charger, OTG mode, etc.) that have been fixed between 4.16 and 4.19.
Have these all fixed by you or upstream? Can these patches be applied to 4.15 as well or is that not possible?
Hi,
Am 29.10.2018 um 14:47 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Montag, den 29.10.2018, 13:31 +0100 schrieb H. Nikolaus Schaller:
Hi,
Well, do you know when firmware loading changed? Because right now, it seems Kernel 4.16/4.17 broke the audio system even without AESS and 4.16 also broke TILER.
Are you sure? For me, 4.19-rc1 broke the TILER and it is fixed since 4.19-rc6. And I have not seen any difference between 4.15 and 4.16 for display.
At least with 4.15 it already shows and rotates the console output when booting and that didn't work anymore with 4.17 (I'm not sure we've tested 4.16).
The key problem seems to be that nobody did look into that so far and try to find the reason. There was plenty of time (3 months per major release, i.e. ~9 months).
With 4.15, all this was still working fine - so if your patch would work with 4.15, I am thinking of using 4.15 for a release. Once it's out, we have more devs who can work on future kernel releases.
Just wait. Maybe, it is just one (missing?) patch which fixes audio for 4.19...
Maybe, but right now, 4.15 seems to work best for us, at least regarding TILER and audio.
So why not fix these on 4.19ff? Next Monday I expect we will see 4.20-rc1.
But beware: 4.15 has several bugs and missing features (e.g. charger, OTG mode, etc.) that have been fixed between 4.16 and 4.19.
Have these all fixed by you or upstream?
I refer to our Letux changes for all those components that are not supported by upstream kernel. Some rely on or were required by upstream changes.
Can these patches be applied to 4.15 as well or is that not possible?
Everything is possible. It is just a matter of the energy needed and how much we can and want to invest.
If someone invests additional time to backport and test the Letux specific patches, you can stay with 4.15 and get it to same feature level.
Generally, I would prefer if there were a team that invests time to fix the small number of remaining bugs of the latest kernel, rather than backporting fixes to some kernel which is no longer maintained upstream and will be abandoned at some not too far future.
So it is just double-work to speed thing up a little (if at all since I can't estimate what goes faster: backporting important things of fixing the remaining bugs).
BR, Nikolaus
Am Montag, den 29.10.2018, 15:04 +0100 schrieb H. Nikolaus Schaller:
Hi,
Well, do you know when firmware loading changed? Because right now, it seems Kernel 4.16/4.17 broke the audio system even without AESS and 4.16 also broke TILER.
Are you sure? For me, 4.19-rc1 broke the TILER and it is fixed since 4.19- rc6. And I have not seen any difference between 4.15 and 4.16 for display.
At least with 4.15 it already shows and rotates the console output when booting and that didn't work anymore with 4.17 (I'm not sure we've tested 4.16).
The key problem seems to be that nobody did look into that so far and try to find the reason. There was plenty of time (3 months per major release, i.e. ~9 months).
Well, AFAIK, only zmatt really knows what's going on here. He is aware of the issue (also the problem that TILER breaks 3D at the moment), but at the moment, he seems pretty busy :(
So unless someone else has the time / mood to work himself into it, it will take an unknown amount of time.
Maybe, but right now, 4.15 seems to work best for us, at least regarding TILER and audio.
So why not fix these on 4.19ff? Next Monday I expect we will see 4.20-rc1.
I'm perfectly fine if these issues are being fixed, but wo is going to do that? That is the bigger issue.
Can these patches be applied to 4.15 as well or is that not possible?
Everything is possible. It is just a matter of the energy needed and how much we can and want to invest.
Well, is it more complex to fix TILER of backport the changes you did to Letux? It could also be that TILER broke because of a different configuration issue in Letux (similar to the audio system you just posted about).
Generally, I would prefer if there were a team that invests time to fix the small number of remaining bugs of the latest kernel, rather than backporting fixes to some kernel which is no longer maintained upstream and will be abandoned at some not too far future.
While it's nice to have a bleeding edge kernel, it's also more time consuming keeping all your stuff working with future upgrades.
Especially if these kernel upgrades happen so often these days and literally EVERYthing we have needs to be retested / reassured to be working.
And that's where we all lack manpower at the moment. See the recent issues with TILER or Audio. And who knows what other new bugs have been introduced.
So I think it's more important to decide to use ONE kernel which we'll work on, and make it run perfect with all hardware.
This can also be Kernel 4.19, if you like. (A normal Debian system still runs Kernel 4.9.0).
This stable kernel can be used for productive systems and all new versions being worked on should be in testing / unstable until all issues have been ironed out.
At the moment it feels to me that you fix some things with each kernel version but that new kernel versions introduces multiple new bugs - so we'll never have a kernel where EVERYTHING simply works.
As you said yourself: 4.15 is missing some OTG / Charger fixes. 4.16 and later has broken audio (okay, fixed already) and TILER
There is not a single kernel we could use for delivery of the device so far. If the OTG / Charger fixes had been applied to 4.15 and also the later in- development kernels, then 4.15 would be ready for delivery.
But right now, not a simple kernel is fully usable.
Hi,
Am 29.10.2018 um 16:04 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Montag, den 29.10.2018, 15:04 +0100 schrieb H. Nikolaus Schaller:
Hi,
Well, do you know when firmware loading changed? Because right now, it seems Kernel 4.16/4.17 broke the audio system even without AESS and 4.16 also broke TILER.
Are you sure? For me, 4.19-rc1 broke the TILER and it is fixed since 4.19- rc6. And I have not seen any difference between 4.15 and 4.16 for display.
At least with 4.15 it already shows and rotates the console output when booting and that didn't work anymore with 4.17 (I'm not sure we've tested 4.16).
The key problem seems to be that nobody did look into that so far and try to find the reason. There was plenty of time (3 months per major release, i.e. ~9 months).
Well, AFAIK, only zmatt really knows what's going on here. He is aware of the issue (also the problem that TILER breaks 3D at the moment), but at the moment, he seems pretty busy :(
So unless someone else has the time / mood to work himself into it, it will take an unknown amount of time.
Maybe, but right now, 4.15 seems to work best for us, at least regarding TILER and audio.
So why not fix these on 4.19ff? Next Monday I expect we will see 4.20-rc1.
I'm perfectly fine if these issues are being fixed, but wo is going to do that?
Anyone is invited...
That is the bigger issue.
Can these patches be applied to 4.15 as well or is that not possible?
Everything is possible. It is just a matter of the energy needed and how much we can and want to invest.
Well, is it more complex to fix TILER of backport the changes you did to Letux? It could also be that TILER broke because of a different configuration issue in Letux (similar to the audio system you just posted about).
Generally, I would prefer if there were a team that invests time to fix the small number of remaining bugs of the latest kernel, rather than backporting fixes to some kernel which is no longer maintained upstream and will be abandoned at some not too far future.
While it's nice to have a bleeding edge kernel, it's also more time consuming keeping all your stuff working with future upgrades.
Especially if these kernel upgrades happen so often these days and literally EVERYthing we have needs to be retested / reassured to be working.
And that's where we all lack manpower at the moment. See the recent issues with TILER or Audio. And who knows what other new bugs have been introduced.
So I think it's more important to decide to use ONE kernel which we'll work on, and make it run perfect with all hardware.
This can also be Kernel 4.19, if you like. (A normal Debian system still runs Kernel 4.9.0).
This stable kernel can be used for productive systems and all new versions being worked on should be in testing / unstable until all issues have been ironed out.
At the moment it feels to me that you fix some things with each kernel version but that new kernel versions introduces multiple new bugs - so we'll never have a kernel where EVERYTHING simply works.
I am waiting for more developers having hardware and helping to fix missing things where we know for long time they are missing. So trying to provide them already with a kernel where EVERYTHING works seems to build a deadlock...
As you said yourself: 4.15 is missing some OTG / Charger fixes. 4.16 and later has broken audio (okay, fixed already) and TILER
There is not a single kernel we could use for delivery of the device so far. If the OTG / Charger fixes had been applied to 4.15 and also the later in- development kernels, then 4.15 would be ready for delivery.
But right now, not a simple kernel is fully usable.
Well, that is life with open-source and volunteers. There are always bugs not found for a long time and new ideas introducing new bugs. People live with that and know that they can blame nobody for that.
LetuxOS follows the mainline, stable, long-term scheme of kernel.org:
http://projects.goldelico.com/p/gta04-kernel/
This means that development is done only on the mainline (alpha) branch.
And as soon as it becomes stable upstream, a new development branch is started and upstream patches from Linux/stable are merged in every now and then. With this approach we got e.g. some spectre/meltdown patches (where I hope they are useful for OMAP5) or others fixing unintended root access.
Like with mainline Linux (where not everything is backported to stable), some (not all!) of the Letux specific patches are backported to older stable kernels when they seem to be mature. But not all of them go into every old kernel because not all are compatible from the code basis.
So if we have the Audio fix at some time (at the moment, we have only a work-around for the Pyra but it has negative side-effect on other platforms), we will backport it to older kernels.
Charger and OTG fixes have already been backported to 4.18 and partially to 4.14 because of to too many conflicts. See for example: http://download.goldelico.com/letux-kernel/letux-4.18.15/src/CHANGES (4.18.11 and 4.14.72 have got charger fixes and 4.18.7 got OTG working).
So I'd suggest to take either 4.14 or 4.18 because they will have long-term support from upstream. Maybe fixing critical security holes we all do not expect.
BTW: the lastest official stable release of LetuxOS kernel is the "master" branch, currently 4.18.
If it still has bugs (and there is always another one), we need bug reports and fixes. If it is still not fully useable it is lack of people cooperating to make it fully useable.
Therefore it IMHO makes not much sense to take 4.15 as the basis. It is not maintained anywhere. Unless you want to maintain it yourself. 4.14 would be a better choice but lacks some backports. Or 4.18, as soon as we have a backport of the Audio/Tiler fixes.
IMHO the next tasks are: * fix CONFIG_IDLE_CPU for OMAP5 (probably any kernel version) * fix TILER console rotation (recent kernels)
BR, Nikolaus
Hi Ed, I want to add a few words to support Nikolaus position: On 29.10.18 17:26, H. Nikolaus Schaller wrote:
Hi,
Am 29.10.2018 um 16:04 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Montag, den 29.10.2018, 15:04 +0100 schrieb H. Nikolaus Schaller:
Hi,
Well, do you know when firmware loading changed? Because right now, it seems Kernel 4.16/4.17 broke the audio system even without AESS and 4.16 also broke TILER.
Are you sure? For me, 4.19-rc1 broke the TILER and it is fixed since 4.19- rc6. And I have not seen any difference between 4.15 and 4.16 for display.
At least with 4.15 it already shows and rotates the console output when booting and that didn't work anymore with 4.17 (I'm not sure we've tested 4.16).
The key problem seems to be that nobody did look into that so far and try to find the reason. There was plenty of time (3 months per major release, i.e. ~9 months).
Well, AFAIK, only zmatt really knows what's going on here. He is aware of the issue (also the problem that TILER breaks 3D at the moment), but at the moment, he seems pretty busy :(
So unless someone else has the time / mood to work himself into it, it will take an unknown amount of time.
Maybe, but right now, 4.15 seems to work best for us, at least regarding TILER and audio.
So why not fix these on 4.19ff? Next Monday I expect we will see 4.20-rc1.
I'm perfectly fine if these issues are being fixed, but wo is going to do that?
Anyone is invited... Without any hardware available, it's hard to help.
That is the bigger issue.
Can these patches be applied to 4.15 as well or is that not possible?
Everything is possible. It is just a matter of the energy needed and how much we can and want to invest.
Well, is it more complex to fix TILER of backport the changes you did to Letux? It could also be that TILER broke because of a different configuration issue in Letux (similar to the audio system you just posted about).
Generally, I would prefer if there were a team that invests time to fix the small number of remaining bugs of the latest kernel, rather than backporting fixes to some kernel which is no longer maintained upstream and will be abandoned at some not too far future.
While it's nice to have a bleeding edge kernel, it's also more time consuming keeping all your stuff working with future upgrades.
Especially if these kernel upgrades happen so often these days and literally EVERYthing we have needs to be retested / reassured to be working.
And that's where we all lack manpower at the moment. See the recent issues with TILER or Audio. And who knows what other new bugs have been introduced.
So I think it's more important to decide to use ONE kernel which we'll work on, and make it run perfect with all hardware.
This can also be Kernel 4.19, if you like. (A normal Debian system still runs Kernel 4.9.0).
This stable kernel can be used for productive systems and all new versions being worked on should be in testing / unstable until all issues have been ironed out.
At the moment it feels to me that you fix some things with each kernel version but that new kernel versions introduces multiple new bugs - so we'll never have a kernel where EVERYTHING simply works.
I am waiting for more developers having hardware and helping to fix missing things where we know for long time they are missing. So trying to provide them already with a kernel where EVERYTHING works seems to build a deadlock...
That's the point. We don't need perfect HW/SW all we need to start is HW, which is able to boot. For me there is only one target kernel version and that is Linus prepatch tree. With our lack of man power we have to mainline as much and fast as possible. You already recognized that the Linux community is working pretty fast, and all that work is done on the most recent version. Besides bug fixes (for already supported boards) you won't get any help from the community. If you want someone to look into your patch it has to be against mainline. To keep this pace, we need as much automated testing as possible. This should be the long term target: https://kernelci.org/
As you said yourself: 4.15 is missing some OTG / Charger fixes. 4.16 and later has broken audio (okay, fixed already) and TILER
There is not a single kernel we could use for delivery of the device so far. If the OTG / Charger fixes had been applied to 4.15 and also the later in- development kernels, then 4.15 would be ready for delivery.
But right now, not a simple kernel is fully usable.
Well, that is life with open-source and volunteers. There are always bugs not found for a long time and new ideas introducing new bugs. People live with that and know that they can blame nobody for that.
LetuxOS follows the mainline, stable, long-term scheme of kernel.org:
http://projects.goldelico.com/p/gta04-kernel/
This means that development is done only on the mainline (alpha) branch.
And as soon as it becomes stable upstream, a new development branch is started and upstream patches from Linux/stable are merged in every now and then. With this approach we got e.g. some spectre/meltdown patches (where I hope they are useful for OMAP5) or others fixing unintended root access.
Like with mainline Linux (where not everything is backported to stable), some (not all!) of the Letux specific patches are backported to older stable kernels when they seem to be mature. But not all of them go into every old kernel because not all are compatible from the code basis.
So if we have the Audio fix at some time (at the moment, we have only a work-around for the Pyra but it has negative side-effect on other platforms), we will backport it to older kernels.
Charger and OTG fixes have already been backported to 4.18 and partially to 4.14 because of to too many conflicts. See for example: http://download.goldelico.com/letux-kernel/letux-4.18.15/src/CHANGES (4.18.11 and 4.14.72 have got charger fixes and 4.18.7 got OTG working).
So I'd suggest to take either 4.14 or 4.18 because they will have long-term support from upstream. Maybe fixing critical security holes we all do not expect.
BTW: the lastest official stable release of LetuxOS kernel is the "master" branch, currently 4.18.
If it still has bugs (and there is always another one), we need bug reports and fixes. If it is still not fully useable it is lack of people cooperating to make it fully useable.
Therefore it IMHO makes not much sense to take 4.15 as the basis. It is not maintained anywhere. Unless you want to maintain it yourself. 4.14 would be a better choice but lacks some backports. Or 4.18, as soon as we have a backport of the Audio/Tiler fixes.
IMHO the next tasks are:
- fix CONFIG_IDLE_CPU for OMAP5 (probably any kernel version)
- fix TILER console rotation (recent kernels)
BR, Nikolaus
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Am Mon, 29 Oct 2018 20:44:23 +0100 hat patrick patrick@wylight.de geschrieben:
Hi,
Thanks for your support and addition.
Hi Ed, I want to add a few words to support Nikolaus position:
I am waiting for more developers having hardware and helping to fix missing things where we know for long time they are missing. So trying to provide them already with a kernel where EVERYTHING works seems to build a deadlock...
That's the point. We don't need perfect HW/SW all we need to start is HW, which is able to boot. For me there is only one target kernel version and that is Linus prepatch tree. With our lack of man power we have to mainline as much and fast as possible.
Well, yes, but that is exactly the problem here. Almost ALL of the prototypes we've sent out had some kind of hardware issue. The one who wanted to work on the volume wheel daemon got a unit with a broken volume wheel. A lot of units had other random issues as well. I know this is normal for a prototype production run (it was the same with the Pandora), however, it was a very frustrating experience for most developers who wanted to work on the unit and didn't even know if these issue were broken hardware or software. Some even spent a lot of their free time trying to fix something that wasn't broken - they simply were unlucky enough to receive a unit with hardware issues.
So I don't want to send any more units out which haven't run through basic hardware tests. With basic I mean: Just a simple test of every known component to show that the unit itself shouldn't have an issue.
Optimizing and fixing drivers, etc. can be done by the devs, but we first need to be sure the hardware is working.
Nikolaus did test all the hardware components during development (he had to make sure his layout isn't broken), so these tests all existed and were working at some times.
This was nothing that made the hardware itself usable (sound still needs a lot of work, even though the tester can output some sound).
I simply don't want to send a unit out to someone, for example, who wants to work on audio and after he spent hours and hours, we find out that the hardware chip on the board is broken.
Bluetooth, GPS, Audio, etc. had all been tested by Nikolaus before - but it wasn't working anymore (not even the simple tests) with the latest kernel versions.
I'm fearing we lose a lot of good devs simply because I ship out broken units. So all I need are simple hardware tests now, not fully fledged bug-free software solutions.
Am 29.10.2018 um 12:44 schrieb Michael Mrozek EvilDragon@openpandora.org:
Hi,
AESS / Audio is one of the biggest remaining issues we have on the Pyra. Everything else seems to work good enough for a start now, but missing audio on a gaming handheld is a bit annoying :)
With some kernel version, speaker output stopped completely, but Nikolaus is currently looking into what changed with each kernel version and when exactly it stopped working to get this to work again, at least.
Ok, here is the result:
letux-4.16-rc3 is the last version where audio works fine letux-4.16-rc4 makes the play command stop after some random time (ca. 1 .. 8 seconds) and audio sounds fluttering like a damaged capstan
In letux-4.17 and later there is no sound from the handsfree speakers any more, but headsets work. At least sometimes.
BR, Nikolaus
Hi all,
Am 29.10.2018 um 13:50 schrieb H. Nikolaus Schaller hns@goldelico.com:
Am 29.10.2018 um 12:44 schrieb Michael Mrozek EvilDragon@openpandora.org:
Hi,
AESS / Audio is one of the biggest remaining issues we have on the Pyra. Everything else seems to work good enough for a start now, but missing audio on a gaming handheld is a bit annoying :)
With some kernel version, speaker output stopped completely, but Nikolaus is currently looking into what changed with each kernel version and when exactly it stopped working to get this to work again, at least.
Ok, here is the result:
letux-4.16-rc3 is the last version where audio works fine letux-4.16-rc4 makes the play command stop after some random time (ca. 1 .. 8 seconds) and audio sounds fluttering like a damaged capstan
In letux-4.17 and later there is no sound from the handsfree speakers any more, but headsets work. At least sometimes.
I have tried to find diffs for twl6040 or generally sound/ or mcpdm driver or device tree changes. But there aren't any.
What I did ffind is a diff in the letux_defconfig:
-# CONFIG_CPU_IDLE is not set +CONFIG_CPU_IDLE=y +CONFIG_CPU_IDLE_MULTIPLE_DRIVERS=y +# CONFIG_CPU_IDLE_GOV_LADDER is not set +CONFIG_CPU_IDLE_GOV_MENU=y +CONFIG_DT_IDLE_STATES=y
+# +# ARM CPU Idle Drivers +# +CONFIG_ARM_CPUIDLE=y
+CONFIG_VIDEOBUF2_V4L2=m
Could CPU_IDLE break the twl6040 audio subsystem? No idea. Therefore, I tried to remove it from 4.19.
And ... it makes audio work!
So we have a simple fix for 4.19. Although no explanation. And no idea how to make it work with CONFIG_CPU_IDLE=y.
If I now look at the commit message where it was introduced into letux-4.16-rc4:
http://git.goldelico.com/?p=letux-kernel.git;a=commit;h=77a30a6c6d8d100022db...
it says:
CPU idle support was not enabled, so lets enable it. That reduces the idle current seen by around 10mA.
(where the 10mA refers to the GTA04 board.)
BTW: omap2plus_defconfig also sets this config (but because the omap5evm has no handsfree speakers I assume nobody can notice such audio problems).
So we face this situation: * CONFIG_CPU_IDLE=y reduces idle current of OMAP3 by 10mA (important ower saving!) * CONFIG_CPU_IDLE=y breaks audio on OMAP5 + twl6040 (but not on OMAP4 + twl6040)
Which config can we use that works for both, GTA04 and Pyra?
BR, Nikolaus
On Mon, 29 Oct 2018 15:42:49 +0100 "H. Nikolaus Schaller" hns@goldelico.com wrote:
Hi all,
Am 29.10.2018 um 13:50 schrieb H. Nikolaus Schaller hns@goldelico.com:
Am 29.10.2018 um 12:44 schrieb Michael Mrozek EvilDragon@openpandora.org:
Hi,
AESS / Audio is one of the biggest remaining issues we have on the Pyra. Everything else seems to work good enough for a start now, but missing audio on a gaming handheld is a bit annoying :)
With some kernel version, speaker output stopped completely, but Nikolaus is currently looking into what changed with each kernel version and when exactly it stopped working to get this to work again, at least.
Ok, here is the result:
letux-4.16-rc3 is the last version where audio works fine letux-4.16-rc4 makes the play command stop after some random time (ca. 1 .. 8 seconds) and audio sounds fluttering like a damaged capstan
In letux-4.17 and later there is no sound from the handsfree speakers any more, but headsets work. At least sometimes.
I have tried to find diffs for twl6040 or generally sound/ or mcpdm driver or device tree changes. But there aren't any.
What I did ffind is a diff in the letux_defconfig:
-# CONFIG_CPU_IDLE is not set +CONFIG_CPU_IDLE=y +CONFIG_CPU_IDLE_MULTIPLE_DRIVERS=y +# CONFIG_CPU_IDLE_GOV_LADDER is not set +CONFIG_CPU_IDLE_GOV_MENU=y +CONFIG_DT_IDLE_STATES=y
+# +# ARM CPU Idle Drivers +# +CONFIG_ARM_CPUIDLE=y
+CONFIG_VIDEOBUF2_V4L2=m
Could CPU_IDLE break the twl6040 audio subsystem? No idea. Therefore, I tried to remove it from 4.19.
And ... it makes audio work!
So we have a simple fix for 4.19. Although no explanation. And no idea how to make it work with CONFIG_CPU_IDLE=y.
What about simply adding pm_qos stuff like in omap-mcbsp.c to omap-abe-twl6030.c? If used harsh enough it should prevent any cpu idling. This as a first step. As a second step, RTFM again and check what can be idled and what not.
Well, no hardware here to test it myself.
Regards, Andreas
Hi all, because it may be lost in a more general discussion on the pyra mailing list, I have reworded and resent this mail with a more specific subject.
Am 29.10.2018 um 13:50 schrieb H. Nikolaus Schaller hns@goldelico.com:
Am 29.10.2018 um 12:44 schrieb Michael Mrozek EvilDragon@openpandora.org:
Hi,
With some kernel version, speaker output stopped completely, but Nikolaus is currently looking into what changed with each kernel version and when exactly it stopped working to get this to work again, at least.
Ok, here is the result:
letux-4.16-rc3 is the last version where audio works fine letux-4.16-rc4 makes the play command hang after some random time (ca. 1 .. 8 seconds) and audio sounds fluttering like a damaged capstan
In letux-4.17 and later there is no sound from the handsfree speakers any more, but headsets work. At least sometimes.
I have tried to find diffs for twl6040 or generally sound/ or mcpdm driver or device tree changes. But there aren't any.
What I did find is a diff in the letux_defconfig:
-# CONFIG_CPU_IDLE is not set +CONFIG_CPU_IDLE=y +CONFIG_CPU_IDLE_MULTIPLE_DRIVERS=y +# CONFIG_CPU_IDLE_GOV_LADDER is not set +CONFIG_CPU_IDLE_GOV_MENU=y +CONFIG_DT_IDLE_STATES=y
+# +# ARM CPU Idle Drivers +# +CONFIG_ARM_CPUIDLE=y
+CONFIG_VIDEOBUF2_V4L2=m
Reverting this in 4.19 in letux_defconfig makes audio work again!
If I now look at the commit message where it was introduced into letux-4.16-rc4:
http://git.goldelico.com/?p=letux-kernel.git;a=commit;h=77a30a6c6d8d100022db...
it says:
CPU idle support was not enabled, so lets enable it. That reduces the idle current seen by around 10mA.
(where the 10mA refers to the GTA04 board.)
BTW: omap2plus_defconfig also sets this config (but because the omap5evm has no handsfree speaker connection I assume nobody else than Pyra users can notice such audio problems).
So we face this situation: * CONFIG_CPU_IDLE=y reduces idle current of OMAP3 by 10mA (important for power saving! also for Pyra) * CONFIG_CPU_IDLE=y breaks audio on OMAP5 + twl6040 (but not on Pandaboard ES OMAP4 + twl6040)
What fix is needed to make mainline (does not include AESS) OMAP5+TWL6040 work in combination with CONFIG_CPU_IDLE=y?
BR and thanks, Nikolaus
* H. Nikolaus Schaller hns@goldelico.com [181029 18:12]:
So we face this situation:
- CONFIG_CPU_IDLE=y reduces idle current of OMAP3 by 10mA (important for power saving! also for Pyra)
- CONFIG_CPU_IDLE=y breaks audio on OMAP5 + twl6040 (but not on Pandaboard ES OMAP4 + twl6040)
What fix is needed to make mainline (does not include AESS) OMAP5+TWL6040 work in combination with CONFIG_CPU_IDLE=y?
The audio components do not block deeper idle states in the hardware. So the driver needs to use PM QoS to prevent glitches, see for example what was done in commit 9834ffd1ecc3 ("ASoC: omap-mcbsp: Add PM QoS support for McBSP to prevent glitches").
I guess you need to do it for mcpdm in this case?
Regards,
Tony
Hi Tony,
Am 06.11.2018 um 16:47 schrieb Tony Lindgren tony@atomide.com:
- H. Nikolaus Schaller hns@goldelico.com [181029 18:12]:
So we face this situation:
- CONFIG_CPU_IDLE=y reduces idle current of OMAP3 by 10mA (important for power saving! also for Pyra)
- CONFIG_CPU_IDLE=y breaks audio on OMAP5 + twl6040 (but not on Pandaboard ES OMAP4 + twl6040)
What fix is needed to make mainline (does not include AESS) OMAP5+TWL6040 work in combination with CONFIG_CPU_IDLE=y?
The audio components do not block deeper idle states in the hardware. So the driver needs to use PM QoS to prevent glitches, see for example what was done in commit 9834ffd1ecc3 ("ASoC: omap-mcbsp: Add PM QoS support for McBSP to prevent glitches").
I guess you need to do it for mcpdm in this case?
I have sketched some hack for first experiments (only compile-tested but not for function). Is this the correct direction?
BR, Nikolaus
diff --git a/sound/soc/omap/omap-mcpdm.c b/sound/soc/omap/omap-mcpdm.c index 4c1be36c2207..fb479b76f112 100644 --- a/sound/soc/omap/omap-mcpdm.c +++ b/sound/soc/omap/omap-mcpdm.c @@ -70,6 +70,9 @@ struct omap_mcpdm { int pm_active_count;
struct snd_dmaengine_dai_dma_data dma_data[2]; + + struct pm_qos_request pm_qos_req; + };
/* @@ -289,6 +292,8 @@ static void omap_mcpdm_dai_shutdown(struct snd_pcm_substream *substream, } }
+ /* FIXME: remove/update pm_qos */ + mutex_unlock(&mcpdm->mutex); }
@@ -359,6 +364,8 @@ static int omap_mcpdm_dai_hw_params(struct snd_pcm_substream *substream,
mcpdm->config[stream].link_mask = link_mask;
+ /* FIXME: handle latency calculations */ + return 0; }
@@ -367,6 +374,10 @@ static int omap_mcpdm_prepare(struct snd_pcm_substream *substream, { struct omap_mcpdm *mcpdm = snd_soc_dai_get_drvdata(dai);
+ #define latency 10 /* in usec https://www.kernel.org/doc/Documentation/power/pm_qos_interface.txt */ + + struct pm_qos_request *pm_qos_req = &mcpdm->pm_qos_req; + if (!omap_mcpdm_active(mcpdm)) { omap_mcpdm_start(mcpdm); omap_mcpdm_reg_dump(mcpdm); @@ -377,6 +388,12 @@ static int omap_mcpdm_prepare(struct snd_pcm_substream *substream, omap_mcpdm_reg_dump(mcpdm); }
+ /* FIXME: use different latencies for channels */ + if (pm_qos_request_active(pm_qos_req)) + pm_qos_update_request(pm_qos_req, latency); + else + pm_qos_add_request(pm_qos_req, PM_QOS_CPU_DMA_LATENCY, latency); + return 0; }
@@ -424,6 +441,9 @@ static int omap_mcpdm_remove(struct snd_soc_dai *dai) { struct omap_mcpdm *mcpdm = snd_soc_dai_get_drvdata(dai);
+ if (pm_qos_request_active(&mcpdm->pm_qos_req)) + pm_qos_remove_request(&mcpdm->pm_qos_req); + free_irq(mcpdm->irq, (void *)mcpdm); pm_runtime_disable(mcpdm->dev);
Hi Nikolaus, Tony,
On 2018-11-12 21:40, H. Nikolaus Schaller wrote:
Hi Tony,
Am 06.11.2018 um 16:47 schrieb Tony Lindgren tony@atomide.com:
- H. Nikolaus Schaller hns@goldelico.com [181029 18:12]:
So we face this situation:
- CONFIG_CPU_IDLE=y reduces idle current of OMAP3 by 10mA (important for power saving! also for Pyra)
- CONFIG_CPU_IDLE=y breaks audio on OMAP5 + twl6040 (but not on Pandaboard ES OMAP4 + twl6040)
What fix is needed to make mainline (does not include AESS) OMAP5+TWL6040 work in combination with CONFIG_CPU_IDLE=y?
The audio components do not block deeper idle states in the hardware. So the driver needs to use PM QoS to prevent glitches, see for example what was done in commit 9834ffd1ecc3 ("ASoC: omap-mcbsp: Add PM QoS support for McBSP to prevent glitches").
I guess you need to do it for mcpdm in this case?
I have sketched some hack for first experiments (only compile-tested but not for function). Is this the correct direction?
I have similar patch which calculates the latency value (but gives 0 all the time :o) which should adopt to the stream. But the McPDM FIFO is pretty small, so we definitely going to see small number to place as latency requirement.
FWIW I have spent the whole day to bisect for omap-abe-twl6040 breakage in linux-next (just to prepare the mcpdm pdmclk/fclk patches)... It ended up blaming a merge commit and the diff showed only unrelated changes :o But I have the fix for it in the omap-abe-twl6040 card itself.
I'll try to send both tomorrow.
BR, Nikolaus
diff --git a/sound/soc/omap/omap-mcpdm.c b/sound/soc/omap/omap-mcpdm.c index 4c1be36c2207..fb479b76f112 100644 --- a/sound/soc/omap/omap-mcpdm.c +++ b/sound/soc/omap/omap-mcpdm.c @@ -70,6 +70,9 @@ struct omap_mcpdm { int pm_active_count;
struct snd_dmaengine_dai_dma_data dma_data[2];
struct pm_qos_request pm_qos_req;};
/* @@ -289,6 +292,8 @@ static void omap_mcpdm_dai_shutdown(struct snd_pcm_substream *substream, } }
/* FIXME: remove/update pm_qos */mutex_unlock(&mcpdm->mutex);}
@@ -359,6 +364,8 @@ static int omap_mcpdm_dai_hw_params(struct snd_pcm_substream *substream,
mcpdm->config[stream].link_mask = link_mask;
/* FIXME: handle latency calculations */return 0;}
@@ -367,6 +374,10 @@ static int omap_mcpdm_prepare(struct snd_pcm_substream *substream, { struct omap_mcpdm *mcpdm = snd_soc_dai_get_drvdata(dai);
#define latency 10 /* in usec https://www.kernel.org/doc/Documentation/power/pm_qos_interface.txt */struct pm_qos_request *pm_qos_req = &mcpdm->pm_qos_req;if (!omap_mcpdm_active(mcpdm)) { omap_mcpdm_start(mcpdm); omap_mcpdm_reg_dump(mcpdm);@@ -377,6 +388,12 @@ static int omap_mcpdm_prepare(struct snd_pcm_substream *substream, omap_mcpdm_reg_dump(mcpdm); }
/* FIXME: use different latencies for channels */if (pm_qos_request_active(pm_qos_req))pm_qos_update_request(pm_qos_req, latency);elsepm_qos_add_request(pm_qos_req, PM_QOS_CPU_DMA_LATENCY, latency);return 0;}
@@ -424,6 +441,9 @@ static int omap_mcpdm_remove(struct snd_soc_dai *dai) { struct omap_mcpdm *mcpdm = snd_soc_dai_get_drvdata(dai);
if (pm_qos_request_active(&mcpdm->pm_qos_req))pm_qos_remove_request(&mcpdm->pm_qos_req);free_irq(mcpdm->irq, (void *)mcpdm); pm_runtime_disable(mcpdm->dev);
- Péter
Texas Instruments Finland Oy, Porkkalankatu 22, 00180 Helsinki. Y-tunnus/Business ID: 0615521-4. Kotipaikka/Domicile: Helsinki
-
Andreas Kemnade -
Belisko Marek -
H. Nikolaus Schaller -
Michael Mrozek -
patrick -
Peter Ujfalusi -
Tony Lindgren