Hi Nikolaus,
On Fri, Sep 01, 2023 at 05:26:17PM +0200, H. Nikolaus Schaller wrote:
Hi Patrik,
Am 01.09.2023 um 16:46 schrieb Patrik Dahlström risca@dalakolonin.se:
Hello Nikolaus,
On Tue, Aug 29, 2023 at 06:00:46PM +0200, H. Nikolaus Schaller wrote:
Hi Patrik,
Am 29.08.2023 um 17:44 schrieb Patrik Dahlström risca@dalakolonin.se:
great! I never digged into PM topics and tools...
/* "tick" of the audio engine */
+#if 0 // makes /sys/kernel/debug/omap-aess/pmem inaccessible omap_aess_write_event_generator(aess, EVENT_TIMER); +#endif
printk("%s: 11\n", __func__);
iMac:master hns$
This is in omap_aess_pcm_probe()
Well, a theory could be that this EVENT_TIMER starts the AES DSP in a state where it is not ready to start...
We are getting closer to the road block but still do not understand which material it is off and which crane or explosive we need...
From the function description, I believe omap_aess_write_event_generator() is meant to start the DSP. You can see a call to stop the generator right after in the probe function. If it's not possible to start the DSP, it points to a real problem.
Yes, that made me think that the DSP is not up and running and why I disabled this step.
Hence we either get timeouts when trying to send audio data or get these pmem and other issues.
However, I was able to read the PMEM area without starting the DSP in my test module. Or maybe the startup was hidden from my eyes? hwmods might have activated some clocks behind my back.
I've been digging into hwmods most of today. I had this idea of writing a minimal kernel module which would simply ioremap() the PMEM memory area and try and read it. Then I could bisect the kernel history to see where things went wrong.
Great idea!
I decided to start with the 4.18 kernel. That's the last kernel version I could find before they started removing hwmods.
It immediately got "Unhandled fault: imprecise external abort".
That occurs if processor or some peripheral tries to access non-existing memory.
Maybe we're missing some memory muxing and that's why this fault happens without hwmods?
Did some digging, and it turns out that OMAP5 did not get the hwmods ported for aess. I copied the definitions from omap_hwmod_44xx_data.c and it all started to work. I could read from the PMEM region without getting any faults any more! Please note that this is without loading any FW so the output is garbage.
I uploaded the code here: https://github.com/Risca/dump_pmem
I'll look into that over the weekend. Since this area is shared with OMAP4 and OMAP5 it should also work on the PandaES.
My experience is that the data in omap_hwmod_44xx_data.c will not be loaded for OMAP5. The Linux-3.8 tree I have is from TI, and that tree have a lot more in omap_hwmod_54xx_data.c. That explains why reading PMEM works in that tree, but not in any mainline tree.
omap_hwmod_44xx_data.c also provides things like OMAP4_RM_ABE_AESS_CONTEXT_OFFSET which is a hint about these context things...
Other things are IMHO available in modern device tree based setups, like mapping of l3/l4 paths and clocks. But this seems to have been removed in v5.6.
OMAP4_RM_ABE_AESS_CONTEXT_OFFSET was last used in arch/arm/mach-omap2/omap_hwmod_44xx_data.c in v5.9 and last defined in arch/arm/mach-omap2/prm44xx.h in v5.15.
Git log shows: c33ff4c864d2b ARM: OMAP2+: Drop unused PRM defines for omap4
commit c33ff4c864d2b2511f43ec11cb23903b957b1159 Author: Tony Lindgren tony@atomide.com Date: Tue Oct 5 10:01:37 2021 +0300
ARM: OMAP2+: Drop unused PRM defines for omap4 These are unused and should be handled by drivers/soc/ti/omap_prm.c driver nowadays. Signed-off-by: Tony Lindgren <tony@atomide.com>
But drivers/soc/ti/omap_prm.c does not talk about AESS or ABE. Now it becomes understandable where our problems come from...
Indeed. I may take a look at omap_prm.c later.
Of course, there is no AESS driver in Linxu-4.18, but this is another clue towards what is missing in Linux-6.x! I decided that I don't have to bisect this any more. It's better to investigate what is the purpose of the aess hwmod. For instance, it calls this function:
/** * omap_hwmod_aess_preprogram - enable AESS internal autogating * @oh: struct omap_hwmod * * * The AESS will not IdleAck to the PRCM until its internal autogating * is enabled. Since internal autogating is disabled by default * after * AESS reset, we must enable autogating after the hwmod code resets * the AESS. Returns 0. */ int omap_hwmod_aess_preprogram(struct omap_hwmod *oh)
Ok, that is quite some description. This function was also removed in v5.6.
Maybe it is possible to translate all this into ti,sysc-omap2 settings.
I'll contact Tony for help how to translate asap (i.e. tomorrow).
Great idea!
I hope you don't mind that I put kernel@pyra-handheld.com in CC.
I have tried to manually insert equivalent code in the debugfs code (omap-aess-dbg.c), but still no luck in reading from PMEM.
Well, that may be in conflict with other initialization steps.
Yeah, it was a long shot. At this point I'm trying a bit of everything and see what works.
To be continued...
Best regards Patrik
So you have opened the magic door a little bit, so that we can already a glimpse into the treasure chamber behind it :)
:)
Best regards, Nikolaus
Hi Tony, we are still struggling with ABE/AESS on the OMAP4/5 (PandaES, Pyra, OMAP5UEVM, BT-200).
Symptoms are that * pmem access will be broken after initializing the ABE-DSP * it seems the AES DSP is not running (and leads to timeouts when sending audio data to the buffers) * boot issues on BT-200
What we have found is that a TI kernel v3.8 works on the OMAP5EVM but newer kernels start to fail.
A major observation ist that hwmods have been removed in smaller pieces and the last removal was in v5.6: c33ff4c864d2b ARM: OMAP2+: Drop unused PRM defines for omap4
There are also some other unknown factors in our code where we do not know how to port to modern kernels: * there is a context lost code but how to make use of it? * pmem fails unless we disable omap_aess_write_event_generator(aess, EVENT_TIMER);
It seems as if clocks and code like omap_hwmod_aess_preprogram() is missing. Especially for the omap4 we have found no equivalent to aess_fclk which exists for omap5 and dra7. Nowhere is a reference using the abe_iclk node.
This would mean that power/clock management for ABE/AESS is incomplete/missing.
So maybe can you review our device tree definitions? Our patched DTS is included here:
https://github.com/goldelico/letux-kernel/tree/letux/aess-v12 https://github.com/goldelico/letux-kernel/tree/letux/aess-v12/arch/arm/boot/...
and compare e.g. with the old hwmods code which is doing quite special AESS setup like
https://elixir.bootlin.com/linux/v4.18.20/source/arch/arm/mach-omap2/omap_hw... https://elixir.bootlin.com/linux/v4.18.20/source/include/sound/aess.h#L37
I hope that I properly summarized the discussions with Patrik and Andreas who please add if I forgot something important.
BR and thanks, Nikolaus
On Sat, 2 Sep 2023 09:27:41 +0200 "H. Nikolaus Schaller" hns@goldelico.com wrote:
Hi Tony, we are still struggling with ABE/AESS on the OMAP4/5 (PandaES, Pyra, OMAP5UEVM, BT-200).
Symptoms are that
- pmem access will be broken after initializing the ABE-DSP
- it seems the AES DSP is not running (and leads to timeouts when sending audio data to the buffers)
- boot issues on BT-200
What we have found is that a TI kernel v3.8 works on the OMAP5EVM but newer kernels start to fail.
A major observation ist that hwmods have been removed in smaller pieces and the last removal was in v5.6: c33ff4c864d2b ARM: OMAP2+: Drop unused PRM defines for omap4
There are also some other unknown factors in our code where we do not know how to port to modern kernels:
- there is a context lost code but how to make use of it?
- pmem fails unless we disable omap_aess_write_event_generator(aess, EVENT_TIMER);
It seems as if clocks and code like omap_hwmod_aess_preprogram() is missing. Especially for the omap4 we have found no equivalent to aess_fclk which exists for omap5 and dra7. Nowhere is a reference using the abe_iclk node.
for omap4 I guess the clocks = <&abe_clkctrl OMAP4_AESS_CLKCTRL 0>;
in omap4-l4-abe.dtsi should be enough and correcly referencing fclk?
Regards, Andreas
* Andreas Kemnade andreas@kemnade.info [230902 10:26]:
On Sat, 2 Sep 2023 09:27:41 +0200 "H. Nikolaus Schaller" hns@goldelico.com wrote:
Hi Tony, we are still struggling with ABE/AESS on the OMAP4/5 (PandaES, Pyra, OMAP5UEVM, BT-200).
Symptoms are that
- pmem access will be broken after initializing the ABE-DSP
- it seems the AES DSP is not running (and leads to timeouts when sending audio data to the buffers)
- boot issues on BT-200
What we have found is that a TI kernel v3.8 works on the OMAP5EVM but newer kernels start to fail.
OK. Sounds like the usual trying to catch up out of tree code with the mainline kernel :) We do have a better infrastructure in place now for using various accelerators with standard Linux generic frameworks now, so it should not be that hard to update the driver code.
A major observation ist that hwmods have been removed in smaller pieces and the last removal was in v5.6: c33ff4c864d2b ARM: OMAP2+: Drop unused PRM defines for omap4
There are also some other unknown factors in our code where we do not know how to port to modern kernels:
- there is a context lost code but how to make use of it?
For now, you can just check if context got lost on runtime PM resume based on some device registers. Linux does not have any framework available right now to make use of the context lost registers. And the context is only lost if the module power domain is shut off, which is mostly not happening with the mainline kernel either.
- pmem fails unless we disable omap_aess_write_event_generator(aess, EVENT_TIMER);
No idea about this one, but this might be doable with generic pwm code now with the dmtimers. See for example how the ir-rx51 is getting phased away and replaced with the generci pwm ir driver.
It seems as if clocks and code like omap_hwmod_aess_preprogram() is missing. Especially for the omap4 we have found no equivalent to aess_fclk which exists for omap5 and dra7. Nowhere is a reference using the abe_iclk node.
for omap4 I guess the clocks = <&abe_clkctrl OMAP4_AESS_CLKCTRL 0>;
in omap4-l4-abe.dtsi should be enough and correcly referencing fclk?
Yeah the clocks chould be there and should use addressing like Andreas is showing.
The replacement for omap_hwmod_aess_preprogam that should do the trick is sysc_module_enable_quirk_aess() based on AESS module detection. See "aess" in ti-sysc.c. If the AESS module has different revision register values other than 0x40000000, then new entries need to be added.
Not sure which dts file(s) to look at in your git tree, maybe send some RFC patches to the mailing lists on adding support?
Regards,
Tony
Hi Tony,
Am 04.09.2023 um 08:34 schrieb Tony Lindgren tony@atomide.com:
- Andreas Kemnade andreas@kemnade.info [230902 10:26]:
On Sat, 2 Sep 2023 09:27:41 +0200 "H. Nikolaus Schaller" hns@goldelico.com wrote:
Hi Tony, we are still struggling with ABE/AESS on the OMAP4/5 (PandaES, Pyra, OMAP5UEVM, BT-200).
Symptoms are that
- pmem access will be broken after initializing the ABE-DSP
- it seems the AES DSP is not running (and leads to timeouts when sending audio data to the buffers)
- boot issues on BT-200
What we have found is that a TI kernel v3.8 works on the OMAP5EVM but newer kernels start to fail.
OK. Sounds like the usual trying to catch up out of tree code with the mainline kernel :) We do have a better infrastructure in place now for using various accelerators with standard Linux generic frameworks now, so it should not be that hard to update the driver code.
Well, the driver seems to be working and upgraded as good as possible - except PM which should be managed by core.
A major observation ist that hwmods have been removed in smaller pieces and the last removal was in v5.6: c33ff4c864d2b ARM: OMAP2+: Drop unused PRM defines for omap4
There are also some other unknown factors in our code where we do not know how to port to modern kernels:
- there is a context lost code but how to make use of it?
For now, you can just check if context got lost on runtime PM resume based on some device registers. Linux does not have any framework available right now to make use of the context lost registers. And the context is only lost if the module power domain is shut off, which is mostly not happening with the mainline kernel either.
Ok, so we do not have to focus on that at the moment.
- pmem fails unless we disable omap_aess_write_event_generator(aess, EVENT_TIMER);
No idea about this one, but this might be doable with generic pwm code now with the dmtimers. See for example how the ir-rx51 is getting phased away and replaced with the generci pwm ir driver.
This is not PWM. It writes to some register shared with the AESS DSP and we assume that the DSP firmware should be started. I have put Péter on CC, maybe he knows something.
It seems as if clocks and code like omap_hwmod_aess_preprogram() is missing. Especially for the omap4 we have found no equivalent to aess_fclk which exists for omap5 and dra7. Nowhere is a reference using the abe_iclk node.
for omap4 I guess the clocks = <&abe_clkctrl OMAP4_AESS_CLKCTRL 0>;
in omap4-l4-abe.dtsi should be enough and correcly referencing fclk?
Yeah the clocks chould be there and should use addressing like Andreas is showing.
Well, according to my analysis the fclk may be there but the iclk is missing or not initialized. That could explain why we get L3 problems as soon as we try to start the AESS-DSP.
The replacement for omap_hwmod_aess_preprogam that should do the trick is sysc_module_enable_quirk_aess() based on AESS module detection. See "aess" in ti-sysc.c.
Ah, I see. I have compared sysc_module_enable_quirk_aess() and omap_hwmod_aess_preprogam() and they seem to be equivalent (assuming the base addresses are resolved in the same way).
If the AESS module has different revision register values other than 0x40000000, then new entries need to be added.
Not sure which dts file(s) to look at in your git tree, maybe send some RFC patches to the mailing lists on adding support?
Well, the patches itself are not cleaned up, but the result is what we think is clean (so a diff to v6.5 will show what we have changed):
Mainly of interest with changes compared to upstream are: https://github.com/goldelico/letux-kernel/blob/letux/aess-v12/arch/arm/boot/... https://github.com/goldelico/letux-kernel/blob/letux/aess-v12/arch/arm/boot/... https://github.com/goldelico/letux-kernel/blob/letux/aess-v12/arch/arm/boot/... https://github.com/goldelico/letux-kernel/blob/letux/aess-v12/arch/arm/boot/...
The key observation is that the abe_iclk references in the DTS seem to be nowhere referenced (which may or may not be an issue):
https://github.com/goldelico/letux-kernel/blob/letux/aess-v12/arch/arm/boot/... https://github.com/goldelico/letux-kernel/blob/letux/aess-v12/arch/arm/boot/...
The branch where all changes are sitting can be inspected here:
https://github.com/goldelico/letux-kernel/commits/letux/aess-v12?after=567e9...
They are all tagged ARM: DTS: omap4 or omap5.
Hope this helps. Otherwise we have to prepare a cleaned up version of the DTS changes as a patch series.
Best regards, Nikolaus
* H. Nikolaus Schaller hns@goldelico.com [230904 18:07]:
Am 04.09.2023 um 08:34 schrieb Tony Lindgren tony@atomide.com: No idea about this one, but this might be doable with generic pwm code now with the dmtimers. See for example how the ir-rx51 is getting phased away and replaced with the generci pwm ir driver.
This is not PWM. It writes to some register shared with the AESS DSP and we assume that the DSP firmware should be started. I have put Péter on CC, maybe he knows something.
Hmm sorry I somehow thought it was using a dmtimer. There are references to ABE_GPT5_FCLK to ABE_GPT8_FCLK in the TRM but maybe those are separate and the driver is not tinkering with the timer register directly hopefully.
It seems as if clocks and code like omap_hwmod_aess_preprogram() is missing. Especially for the omap4 we have found no equivalent to aess_fclk which exists for omap5 and dra7. Nowhere is a reference using the abe_iclk node.
for omap4 I guess the clocks = <&abe_clkctrl OMAP4_AESS_CLKCTRL 0>;
in omap4-l4-abe.dtsi should be enough and correcly referencing fclk?
Yeah the clocks chould be there and should use addressing like Andreas is showing.
Well, according to my analysis the fclk may be there but the iclk is missing or not initialized. That could explain why we get L3 problems as soon as we try to start the AESS-DSP.
OK
The key observation is that the abe_iclk references in the DTS seem to be nowhere referenced (which may or may not be an issue):
https://github.com/goldelico/letux-kernel/blob/letux/aess-v12/arch/arm/boot/... https://github.com/goldelico/letux-kernel/blob/letux/aess-v12/arch/arm/boot/...
So I guess the ick is in the dts the ocp_abe_iclk@528 for omap4 and abe_iclk@528 for omap5. Seems like the driver should request them, I recall that the interconnect target module does not need the ick to access sysc and revision registers.
The branch where all changes are sitting can be inspected here:
https://github.com/goldelico/letux-kernel/commits/letux/aess-v12?after=567e9...
They are all tagged ARM: DTS: omap4 or omap5.
Hmm the "we don't need separate target modules" patch is wrong, the modules may have separate clocks and power domains, and flushing a posted write to one module does not flush write to the other module. This can lead into hard to track down bugs accessing separate modules. The dts module data I generated from the hardware AP registers for each SoC so the module ranges should be correct.
Hope this helps. Otherwise we have to prepare a cleaned up version of the DTS changes as a patch series.
Yeah nice,
Tony
Hi Tony,
Am 05.09.2023 um 08:12 schrieb Tony Lindgren tony@atomide.com:
The key observation is that the abe_iclk references in the DTS seem to be nowhere referenced
(which may or may not be an issue):
https://github.com/goldelico/letux-kernel/blob/letux/aess-v12/arch/arm/boot/... https://github.com/goldelico/letux-kernel/blob/letux/aess-v12/arch/arm/boot/...
So I guess the ick is in the dts the ocp_abe_iclk@528 for omap4 and abe_iclk@528 for omap5. Seems like the driver should request them, I recall that the interconnect target module does not need the ick to access sysc and revision registers.
Yes, that is what I suspect but I don't know how to request them.
The branch where all changes are sitting can be inspected here:
https://github.com/goldelico/letux-kernel/commits/letux/aess-v12?after=567e9...
They are all tagged ARM: DTS: omap4 or omap5.
Hmm the "we don't need separate target modules" patch is wrong, the modules may have separate clocks and power domains, and flushing a posted write to
According to several TRM versions this is only one module (the Audio Engine AE module) which provides these RAM areas. "The AE subsystem includes an AE and has the following on-chip memories available: 64-KiB data memory (DMEM); 6-KiB coefficient memory (CMEM); 8-KiB program memory (PMEM); and 18-KiB sample memory (SMEM)." "Figure 13-1. Audio Back End" seems to integrate the whole AE into a single (target) module being part of the ABE together with McPDM, McBSP, WDT, GPTIMER.
There seems to be no separate dmem, cmem, smem, pmem modules which need hwmod or sysc control.
See also Table "Table 2-8. ABE DSP Memory Space Mapping" in the OMAP5432 TRM where DMEM and CMEM are called "Memory". But you are right the SMEM and PMEM are called "Module" like for example MCBSP2. But there is also an AESS configuration "module".
Finally "Table 3-243. CD_ABE Modules Clocks Association" seems to be quite cleart that there is AESS_GFCLK and ABE_GICLK. And clocks for all other modules - but no special ones for the memory blocks.
Maybe "Table 3-245. CD_ABE Modules Clock-Management Modes and Control" and "Table 3-246. CD_ABE Modules Slave Clock-Management Modes and Control" give more hints about the CLKCTRL we need:
CM_ABE_AESS_CLKCTRL[18] STBYST CM_ABE_AESS_CLKCTRL[17:16] IDLEST CM_ABE_L4_ABE_CLKCTRL[17:16] IDLEST CM_ABE_AESS_C LKCTRL[1:0] MODULEMODE CM_ABE_L4_ABE_ CLKCTRL[1:0] MODULEMODE
Table "Table 3-1074. ABE_CM_CORE_AON Register Mapping Summary" (all from OMAP5 TRM) also may be helpful.
one module does not flush write to the other module. This can lead into hard to track down bugs accessing separate modules. The dts module data I generated from the hardware AP registers for each SoC so the module ranges should be correct.
Finally all this was based on some 3.15 DTS entry by TI which has a single ti,hwmods = "aess"; entry.
I have not found a link to the original but have a copy in our tree:
https://git.goldelico.com/?p=letux-kernel.git;a=commit;h=ca9ee9532104eac5cfe...
It seems to be impossible to handle this with multiple target-modules.
Hope this helps. Otherwise we have to prepare a cleaned up version of the DTS changes as a patch series.
Yeah nice,
I have done this so that only the AESS specific DTS changes remain and I will post a short RFC series soon. This should make it easier for doing reviews.
BR and thanks, Nikolaus
On Tue, 5 Sep 2023 14:42:03 +0200 "H. Nikolaus Schaller" hns@goldelico.com wrote:
Hi Tony,
Am 05.09.2023 um 08:12 schrieb Tony Lindgren tony@atomide.com:
The key observation is that the abe_iclk references in the DTS seem to be nowhere referenced
(which may or may not be an issue):
https://github.com/goldelico/letux-kernel/blob/letux/aess-v12/arch/arm/boot/... https://github.com/goldelico/letux-kernel/blob/letux/aess-v12/arch/arm/boot/...
So I guess the ick is in the dts the ocp_abe_iclk@528 for omap4 and abe_iclk@528 for omap5. Seems like the driver should request them, I recall that the interconnect target module does not need the ick to access sysc and revision registers.
Yes, that is what I suspect but I don't know how to request them.
add ,"ick" to the clock-names line in the target-module? and ,<&ocp_abe_iclk> to the target module?
Regards, Andreas
Hi,
Am 05.09.2023 um 16:44 schrieb Andreas Kemnade andreas@kemnade.info:
On Tue, 5 Sep 2023 14:42:03 +0200 "H. Nikolaus Schaller" hns@goldelico.com wrote:
Hi Tony,
Am 05.09.2023 um 08:12 schrieb Tony Lindgren tony@atomide.com:
The key observation is that the abe_iclk references in the DTS seem to be nowhere referenced
(which may or may not be an issue):
https://github.com/goldelico/letux-kernel/blob/letux/aess-v12/arch/arm/boot/... https://github.com/goldelico/letux-kernel/blob/letux/aess-v12/arch/arm/boot/...
So I guess the ick is in the dts the ocp_abe_iclk@528 for omap4 and abe_iclk@528 for omap5. Seems like the driver should request them, I recall that the interconnect target module does not need the ick to access sysc and revision registers.
Yes, that is what I suspect but I don't know how to request them.
add ,"ick" to the clock-names line in the target-module? and ,<&ocp_abe_iclk> to the target module?
You mean
diff --git a/arch/arm/boot/dts/ti/omap/omap4-l4-abe.dtsi b/arch/arm/boot/dts/ti/omap/omap4-l4-abe.dtsi index 7ca7b369b4e59..aa140a8bb9f29 100644 --- a/arch/arm/boot/dts/ti/omap/omap4-l4-abe.dtsi +++ b/arch/arm/boot/dts/ti/omap/omap4-l4-abe.dtsi @@ -482,8 +482,8 @@ target-module@f1000 { /* 0x401f1000, ap 32 20.0 */ <SYSC_IDLE_NO>, <SYSC_IDLE_SMART>; /* Domains (V, P, C): iva, abe_pwrdm, abe_clkdm */ - clocks = <&abe_clkctrl OMAP4_AESS_CLKCTRL 0>; - clock-names = "fck"; + clocks = <&abe_clkctrl OMAP4_AESS_CLKCTRL 0>, <&ocp_abe_iclk>; + clock-names = "fck", "ick"; #address-cells = <1>; #size-cells = <1>;
diff --git a/arch/arm/boot/dts/ti/omap/omap5-l4-abe.dtsi b/arch/arm/boot/dts/ti/omap/omap5-l4-abe.dtsi index 611c92e04fe7a..647f34e38c431 100644 --- a/arch/arm/boot/dts/ti/omap/omap5-l4-abe.dtsi +++ b/arch/arm/boot/dts/ti/omap/omap5-l4-abe.dtsi @@ -421,8 +421,8 @@ target-module@f1000 { /* 0x401f1000, ap 32 20.0 */ <SYSC_IDLE_NO>, <SYSC_IDLE_SMART>; /* Domains (V, P, C): iva, abe_pwrdm, abe_clkdm */ - clocks = <&abe_clkctrl OMAP5_AESS_CLKCTRL 0>; - clock-names = "fck"; + clocks = <&abe_clkctrl OMAP5_AESS_CLKCTRL 0>, <&abe_iclk>; + clock-names = "fck", "ick"; #address-cells = <1>; #size-cells = <1>;
Well, certainly worth trying!
BR, Nikolaus
Am 05.09.2023 um 17:09 schrieb H. Nikolaus Schaller hns@goldelico.com:
Hi,
Am 05.09.2023 um 16:44 schrieb Andreas Kemnade andreas@kemnade.info:
On Tue, 5 Sep 2023 14:42:03 +0200 "H. Nikolaus Schaller" hns@goldelico.com wrote:
Hi Tony,
Am 05.09.2023 um 08:12 schrieb Tony Lindgren tony@atomide.com:
The key observation is that the abe_iclk references in the DTS seem to be nowhere referenced
(which may or may not be an issue):
https://github.com/goldelico/letux-kernel/blob/letux/aess-v12/arch/arm/boot/... https://github.com/goldelico/letux-kernel/blob/letux/aess-v12/arch/arm/boot/...
So I guess the ick is in the dts the ocp_abe_iclk@528 for omap4 and abe_iclk@528 for omap5. Seems like the driver should request them, I recall that the interconnect target module does not need the ick to access sysc and revision registers.
Yes, that is what I suspect but I don't know how to request them.
add ,"ick" to the clock-names line in the target-module? and ,<&ocp_abe_iclk> to the target module?
You mean
diff --git a/arch/arm/boot/dts/ti/omap/omap4-l4-abe.dtsi b/arch/arm/boot/dts/ti/omap/omap4-l4-abe.dtsi index 7ca7b369b4e59..aa140a8bb9f29 100644 --- a/arch/arm/boot/dts/ti/omap/omap4-l4-abe.dtsi +++ b/arch/arm/boot/dts/ti/omap/omap4-l4-abe.dtsi @@ -482,8 +482,8 @@ target-module@f1000 { /* 0x401f1000, ap 32 20.0 */ <SYSC_IDLE_NO>, <SYSC_IDLE_SMART>; /* Domains (V, P, C): iva, abe_pwrdm, abe_clkdm */
clocks = <&abe_clkctrl OMAP4_AESS_CLKCTRL 0>;
clock-names = "fck";
clocks = <&abe_clkctrl OMAP4_AESS_CLKCTRL 0>, <&ocp_abe_iclk>;
clock-names = "fck", "ick"; #address-cells = <1>; #size-cells = <1>;
Well, certainly worth trying!
A quick test on PandaES shows that the ick is (or becomes) active (I have not yet compared to without above patch):
root@letux:~# ls -l /sys/kernel/debug/clk/ocp_abe_iclk total 0 -r--r--r-- 1 root root 0 Jan 1 1970 clk_accuracy -r--r--r-- 1 root root 0 Jan 1 1970 clk_duty_cycle -r--r--r-- 1 root root 0 Jan 1 1970 clk_enable_count -r--r--r-- 1 root root 0 Jan 1 1970 clk_flags -r--r--r-- 1 root root 0 Jan 1 1970 clk_max_rate -r--r--r-- 1 root root 0 Jan 1 1970 clk_min_rate -r--r--r-- 1 root root 0 Jan 1 1970 clk_notifier_count -r--r--r-- 1 root root 0 Jan 1 1970 clk_parent -r--r--r-- 1 root root 0 Jan 1 1970 clk_phase -r--r--r-- 1 root root 0 Jan 1 1970 clk_prepare_count -r--r--r-- 1 root root 0 Jan 1 1970 clk_protect_count -r--r--r-- 1 root root 0 Jan 1 1970 clk_rate root@letux:~# cat /sys/kernel/debug/clk/ocp_abe_iclk/* 0 1/2 1 4294967295 0 0 abe-clkctrl:0008:24 0 1 0 49152000 root@letux:~#
But still:
root@letux:~# xxd /sys/kernel/debug/omap-aess/cmem | head -5 00000000: 0000 0000 0000 8000 0000 8000 0000 4000 ..............@. 00000010: 0000 0000 0000 0000 0000 0000 0010 0000 ................ 00000020: 0010 0000 0010 0000 0010 0000 0010 0000 ................ 00000030: 0010 0000 0010 0000 0010 1500 0010 2000 .............. . 00000040: 0000 0000 0010 0000 0010 0000 0010 0000 ................ root@letux:~# xxd /sys/kernel/debug/omap-aess/dmem | head -5 00000000: 0000 0000 0000 0000 0200 0400 0000 0000 ................ 00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000030: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ root@letux:~# xxd /sys/kernel/debug/omap-aess/smem | head -5 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000020: 0000 0000 0000 0000 037f 0400 0000 0000 ................ 00000030: 0282 0400 0000 0000 03ca 0600 0000 0000 ................ 00000040: 05cc 0400 0000 0000 04d1 0400 0000 0000 ................ root@letux:~# xxd /sys/kernel/debug/omap-aess/pmem | head -5 [ 526.023803] 8<--- cut here --- [ 526.027526] Unhandled fault: imprecise external abort (0x1406) at 0x004f62d4 [ 526.027526] [004f62d4] *pgd=bb1f8831 [ 526.038726] Internal error: : 1406 [#1] PREEMPT SMP ARM [ 526.038726] Modules linked in: usb_f_ecm g_ether usb_f_rndis u_ether libcomposite cmac bnep wl12xx wlcore mac80211 libarc4 smsc95xx usbnet cfg80211 snd_soc_spdif_tx snd_soc_omap_hdmi snd_soc_dmic snd_soc_omap_abe_twl6040 snd_soc_omap_mcpdm snd_soc_twl6040 omapdrm cec omap_sham emif omap_aes_driver omap2430 ehci_omap ohci_platform ohci_hcd bq27xxx_battery_hdq bq27xxx_battery wlcore_sdio omap_des crypto_engine omap_crypto libdes omap_hdq phy_twl6030_usb twl6030_gpadc musb_hdrc industrialio hci_uart btbcm bluetooth clk_twl6040 gpio_twl6040 ecdh_generic ecc snd_soc_omap_aess snd_soc_omap_mcbsp snd_soc_ti_sdma display_connector leds_gpio ti_tfp410 ti_tpd12s015 drm_kms_helper fuse drm drm_panel_orientation_quirks configfs ip_tables x_tables ipv6 autofs4 [ 526.107086] CPU: 0 PID: 3649 Comm: xxd Tainted: G W 6.5.0-letux+ #14905 [ 526.117065] Hardware name: Generic OMAP4 (Flattened Device Tree) [ 526.128936] PC is at __copy_to_user_std+0x74/0x374 [ 526.128936] LR is at 0x0 [ 526.128936] pc : [<c093cc44>] lr : [<00000000>] psr: 200c0013 [ 526.143249] sp : f05a9e88 ip : 00000000 fp : f05a9e98 [ 526.145202] r10: 00000000 r9 : 00000000 r8 : 00000000 [ 526.150787] r7 : 004f52d0 r6 : 00000000 r5 : 00000000 r4 : 00000000 [ 526.161102] r3 : 00000000 r2 : 00000f20 r1 : f0280060 r0 : 004f5310 [ 526.167968] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user [ 526.169311] Control: 10c5387d Table: 81e2c04a DAC: 00000055 [ 526.180236] Register r0 information: non-paged memory [ 526.181823] Register r1 information: 0-page vmalloc region starting at 0xf0280000 allocated at __devm_ioremap+0x68/0xa8 [ 526.197082] Register r2 information: non-paged memory [ 526.197082] Register r3 information: NULL pointer [ 526.207061] Register r4 information: NULL pointer [ 526.208465] Register r5 information: NULL pointer [ 526.213409] Register r6 information: NULL pointer [ 526.220733] Register r7 information: non-paged memory [ 526.228607] Register r8 information: NULL pointer [ 526.233398] Register r9 information: NULL pointer [ 526.238494] Register r10 information: NULL pointer [ 526.243530] Register r11 information: 2-page vmalloc region starting at 0xf05a8000 allocated at kernel_clone+0xc0/0x2e8 [ 526.247070] Register r12 information: NULL pointer [ 526.257080] Process xxd (pid: 3649, stack limit = 0xccd1bf74) [ 526.257080] Stack: (0xf05a9e88 to 0xf05aa000) [ 526.270568] 9e80: f05a9f70 c17c1040 f0280000 00000051 004f52d0 00001000 [ 526.270568] 9ea0: 00000000 00001000 004f5190 bf15b6b4 c471e540 bf15b768 c276c660 004f52d0 [ 526.283569] 9ec0: 00001000 f05a9f70 00000003 bf15b784 f0280000 00002000 00001000 c03bf80c [ 526.290069] 9ee0: c03bf7c4 00001000 c471e540 f05a9f70 c0e03fc8 004f52d0 00000003 c028ab88 [ 526.297058] 9f00: c1e2c010 00000000 00000000 00000000 ffefe3d8 efd5cf14 00000000 e519f734 [ 526.313568] 9f20: 00516000 00000817 f05a9fb0 004f62d4 00000817 c1a25e00 00000255 004f6000 [ 526.322174] 9f40: 00000000 c09718f8 0a891c34 e519f734 63a445cf c471e540 f05a9f70 004f52d0 [ 526.330780] 9f60: c0e03fc8 f05a9f7c 00001000 c028afe0 00000000 00000000 00000009 c471e540 [ 526.338226] 9f80: 00000000 e519f734 00000000 004f5190 000005e8 b6fcfbf0 00000003 c01002e8 [ 526.339385] 9fa0: c17d5780 c0100080 004f5190 000005e8 00000003 004f52d0 00001000 00000000 [ 526.354766] 9fc0: 004f5190 000005e8 b6fcfbf0 00000003 00000002 00000009 00000000 004f5190 [ 526.365173] 9fe0: 00000003 bea8fb60 b6f43d17 b6eca7e6 400c0030 00000003 00000000 00000000 [ 526.373779] __copy_to_user_std from aess_read_mem+0xf0/0x138 [snd_soc_omap_aess] [ 526.381744] aess_read_mem [snd_soc_omap_aess] from aess_read_pmem+0x1c/0x24 [snd_soc_omap_aess] [ 526.391052] aess_read_pmem [snd_soc_omap_aess] from full_proxy_read+0x48/0x5c [ 526.392395] full_proxy_read from vfs_read+0x140/0x180 [ 526.401733] vfs_read from ksys_read+0x68/0xb4 [ 526.408813] ksys_read from ret_fast_syscall+0x0/0x54 [ 526.410064] Exception stack(0xf05a9fa8 to 0xf05a9ff0) [ 526.419464] 9fa0: 004f5190 000005e8 00000003 004f52d0 00001000 00000000 [ 526.424072] 9fc0: 004f5190 000005e8 b6fcfbf0 00000003 00000002 00000009 00000000 004f5190 [ 526.433044] 9fe0: 00000003 bea8fb60 b6f43d17 b6eca7e6 [ 526.437072] Code: f5d1f05c f5d1f07c e8b15378 e2522020 (e8a05378) [ 526.437072] ---[ end trace 0000000000000000 ]--- [ 526.437072] note: xxd[3649] exited with irqs disabled [ 526.458831] 8<--- cut here --- [ 526.462066] Unhandled fault: imprecise external abort (0x1406) at 0x004f62d4 [ 526.467987] [004f62d4] *pgd=00000000 root@letux:~#
BR, Nikolaus
Am 05.09.2023 um 17:48 schrieb H. Nikolaus Schaller hns@goldelico.com:
Am 05.09.2023 um 17:09 schrieb H. Nikolaus Schaller hns@goldelico.com:
Hi,
Am 05.09.2023 um 16:44 schrieb Andreas Kemnade andreas@kemnade.info:
On Tue, 5 Sep 2023 14:42:03 +0200 "H. Nikolaus Schaller" hns@goldelico.com wrote:
Hi Tony,
Am 05.09.2023 um 08:12 schrieb Tony Lindgren tony@atomide.com:
The key observation is that the abe_iclk references in the DTS seem to be nowhere referenced
(which may or may not be an issue):
https://github.com/goldelico/letux-kernel/blob/letux/aess-v12/arch/arm/boot/... https://github.com/goldelico/letux-kernel/blob/letux/aess-v12/arch/arm/boot/...
So I guess the ick is in the dts the ocp_abe_iclk@528 for omap4 and abe_iclk@528 for omap5. Seems like the driver should request them, I recall that the interconnect target module does not need the ick to access sysc and revision registers.
Yes, that is what I suspect but I don't know how to request them.
add ,"ick" to the clock-names line in the target-module? and ,<&ocp_abe_iclk> to the target module?
You mean
diff --git a/arch/arm/boot/dts/ti/omap/omap4-l4-abe.dtsi b/arch/arm/boot/dts/ti/omap/omap4-l4-abe.dtsi index 7ca7b369b4e59..aa140a8bb9f29 100644 --- a/arch/arm/boot/dts/ti/omap/omap4-l4-abe.dtsi +++ b/arch/arm/boot/dts/ti/omap/omap4-l4-abe.dtsi @@ -482,8 +482,8 @@ target-module@f1000 { /* 0x401f1000, ap 32 20.0 */ <SYSC_IDLE_NO>, <SYSC_IDLE_SMART>; /* Domains (V, P, C): iva, abe_pwrdm, abe_clkdm */
clocks = <&abe_clkctrl OMAP4_AESS_CLKCTRL 0>;
clock-names = "fck";
clocks = <&abe_clkctrl OMAP4_AESS_CLKCTRL 0>, <&ocp_abe_iclk>;
clock-names = "fck", "ick"; #address-cells = <1>; #size-cells = <1>;
Well, certainly worth trying!
A quick test on PandaES shows that the ick is (or becomes) active (I have not yet compared to without above patch):
root@letux:~# ls -l /sys/kernel/debug/clk/ocp_abe_iclk total 0 -r--r--r-- 1 root root 0 Jan 1 1970 clk_accuracy -r--r--r-- 1 root root 0 Jan 1 1970 clk_duty_cycle -r--r--r-- 1 root root 0 Jan 1 1970 clk_enable_count -r--r--r-- 1 root root 0 Jan 1 1970 clk_flags -r--r--r-- 1 root root 0 Jan 1 1970 clk_max_rate -r--r--r-- 1 root root 0 Jan 1 1970 clk_min_rate -r--r--r-- 1 root root 0 Jan 1 1970 clk_notifier_count -r--r--r-- 1 root root 0 Jan 1 1970 clk_parent -r--r--r-- 1 root root 0 Jan 1 1970 clk_phase -r--r--r-- 1 root root 0 Jan 1 1970 clk_prepare_count -r--r--r-- 1 root root 0 Jan 1 1970 clk_protect_count -r--r--r-- 1 root root 0 Jan 1 1970 clk_rate root@letux:~# cat /sys/kernel/debug/clk/ocp_abe_iclk/* 0 1/2 1 4294967295 0 0 abe-clkctrl:0008:24 0 1 0 49152000 root@letux:~#
There is indeed a difference after removing, recompiling, rebooting:
root@letux:~# cat /sys/kernel/debug/clk/ocp_abe_iclk/* 0 1/2 0 <---- 4294967295 0 0 abe-clkctrl:0008:24 0 0 <---- 0 49152000 root@letux:~#
BR, Nikolaus
On Tue, Sep 05, 2023 at 02:42:03PM +0200, H. Nikolaus Schaller wrote:
Finally all this was based on some 3.15 DTS entry by TI which has a single ti,hwmods = "aess"; entry.
I have not found a link to the original but have a copy in our tree:
https://git.goldelico.com/?p=letux-kernel.git;a=commit;h=ca9ee9532104eac5cfe...
I've been working with the Linux-3.8.13 from GLSDK 6_03_00_01. The kernel sources in that SDK point to omapzoom.org, which is quite slow. I also found the same commit here: https://git.ti.com/cgit/android-sdk/kernel-omap/commit/?h=p-ti-glsdk-3.8.y&a...
On Sat, Sep 02, 2023 at 08:37:16AM +0200, Patrik Dahlström wrote:
Hi Nikolaus,
On Fri, Sep 01, 2023 at 05:26:17PM +0200, H. Nikolaus Schaller wrote:
Hi Patrik,
Am 01.09.2023 um 16:46 schrieb Patrik Dahlström risca@dalakolonin.se:
Hello Nikolaus,
On Tue, Aug 29, 2023 at 06:00:46PM +0200, H. Nikolaus Schaller wrote:
Hi Patrik,
Am 29.08.2023 um 17:44 schrieb Patrik Dahlström risca@dalakolonin.se:
great! I never digged into PM topics and tools...
/* "tick" of the audio engine */
+#if 0 // makes /sys/kernel/debug/omap-aess/pmem inaccessible omap_aess_write_event_generator(aess, EVENT_TIMER); +#endif
printk("%s: 11\n", __func__);
iMac:master hns$
This is in omap_aess_pcm_probe()
Well, a theory could be that this EVENT_TIMER starts the AES DSP in a state where it is not ready to start...
We are getting closer to the road block but still do not understand which material it is off and which crane or explosive we need...
From the function description, I believe omap_aess_write_event_generator() is meant to start the DSP. You can see a call to stop the generator right after in the probe function. If it's not possible to start the DSP, it points to a real problem.
Yes, that made me think that the DSP is not up and running and why I disabled this step.
Hence we either get timeouts when trying to send audio data or get these pmem and other issues.
However, I was able to read the PMEM area without starting the DSP in my test module. Or maybe the startup was hidden from my eyes? hwmods might have activated some clocks behind my back.
I have now tested my module on Linux-6.x and it dumps PMEM area fine. For this test I disabled sound support entirely in the kernel config and only used my test module. I can dump PMEM repeatedly by removing and inserting the module.
There must be something we do while loading some of the audio drivers that make this stop working.
I've been digging into hwmods most of today. I had this idea of writing a minimal kernel module which would simply ioremap() the PMEM memory area and try and read it. Then I could bisect the kernel history to see where things went wrong.
Great idea!
I decided to start with the 4.18 kernel. That's the last kernel version I could find before they started removing hwmods.
It immediately got "Unhandled fault: imprecise external abort".
That occurs if processor or some peripheral tries to access non-existing memory.
Maybe we're missing some memory muxing and that's why this fault happens without hwmods?
Did some digging, and it turns out that OMAP5 did not get the hwmods ported for aess. I copied the definitions from omap_hwmod_44xx_data.c and it all started to work. I could read from the PMEM region without getting any faults any more! Please note that this is without loading any FW so the output is garbage.
I uploaded the code here: https://github.com/Risca/dump_pmem
I'll look into that over the weekend. Since this area is shared with OMAP4 and OMAP5 it should also work on the PandaES.
My experience is that the data in omap_hwmod_44xx_data.c will not be loaded for OMAP5. The Linux-3.8 tree I have is from TI, and that tree have a lot more in omap_hwmod_54xx_data.c. That explains why reading PMEM works in that tree, but not in any mainline tree.
omap_hwmod_44xx_data.c also provides things like OMAP4_RM_ABE_AESS_CONTEXT_OFFSET which is a hint about these context things...
Other things are IMHO available in modern device tree based setups, like mapping of l3/l4 paths and clocks. But this seems to have been removed in v5.6.
OMAP4_RM_ABE_AESS_CONTEXT_OFFSET was last used in arch/arm/mach-omap2/omap_hwmod_44xx_data.c in v5.9 and last defined in arch/arm/mach-omap2/prm44xx.h in v5.15.
Git log shows: c33ff4c864d2b ARM: OMAP2+: Drop unused PRM defines for omap4
commit c33ff4c864d2b2511f43ec11cb23903b957b1159 Author: Tony Lindgren tony@atomide.com Date: Tue Oct 5 10:01:37 2021 +0300
ARM: OMAP2+: Drop unused PRM defines for omap4 These are unused and should be handled by drivers/soc/ti/omap_prm.c driver nowadays. Signed-off-by: Tony Lindgren <tony@atomide.com>
But drivers/soc/ti/omap_prm.c does not talk about AESS or ABE. Now it becomes understandable where our problems come from...
Indeed. I may take a look at omap_prm.c later.
Of course, there is no AESS driver in Linxu-4.18, but this is another clue towards what is missing in Linux-6.x! I decided that I don't have to bisect this any more. It's better to investigate what is the purpose of the aess hwmod. For instance, it calls this function:
/** * omap_hwmod_aess_preprogram - enable AESS internal autogating * @oh: struct omap_hwmod * * * The AESS will not IdleAck to the PRCM until its internal autogating * is enabled. Since internal autogating is disabled by default * after * AESS reset, we must enable autogating after the hwmod code resets * the AESS. Returns 0. */ int omap_hwmod_aess_preprogram(struct omap_hwmod *oh)
Ok, that is quite some description. This function was also removed in v5.6.
Maybe it is possible to translate all this into ti,sysc-omap2 settings.
I'll contact Tony for help how to translate asap (i.e. tomorrow).
Great idea!
I hope you don't mind that I put kernel@pyra-handheld.com in CC.
I have tried to manually insert equivalent code in the debugfs code (omap-aess-dbg.c), but still no luck in reading from PMEM.
Well, that may be in conflict with other initialization steps.
Yeah, it was a long shot. At this point I'm trying a bit of everything and see what works.
To be continued...
Best regards Patrik
So you have opened the magic door a little bit, so that we can already a glimpse into the treasure chamber behind it :)
:)
Best regards, Nikolaus
Hi Patrik,
Am 02.09.2023 um 09:44 schrieb Patrik Dahlström risca@dalakolonin.se:
However, I was able to read the PMEM area without starting the DSP in my test module. Or maybe the startup was hidden from my eyes? hwmods might have activated some clocks behind my back.
I have now tested my module on Linux-6.x and it dumps PMEM area fine. For this test I disabled sound support entirely in the kernel config and only used my test module. I can dump PMEM repeatedly by removing and inserting the module.
I had to add
#include <linux/mod_devicetable.h> /* Needed for struct of_device_id */
to compile in on v6.5. But I have not tested.
There must be something we do while loading some of the audio drivers that make this stop working.
Well, we can also read the pmem when disabling the EVENT_TIMER setup.
So it is the attempt to start the AESS DSP which finally breaks things. AFAIK the DSP has also access to the L3/L4 interchanges and may make a hickup (i.e. lock) there if the AESS clocks are not properly set up?
Let's see what Tony will comment.
BR, Nikolaus