Audio Driver (with bounty!)

hns · Mar 30, 2023

... or IRQ or DMA. Or something else.
With some kernel debugging enabled I get:

Code:

[  331.359714] omap-abe-twl6040 sound: playback write error (DMA or IRQ trouble?)

which comes from the function wait_for_avail().
This is now hardest rock to break through.
Well, a known brute force trick is to devmem2 (or kernel equivalent) dump important registers in clock setup, interface setup, DMA setup, AESS setup etc. and compare bit by bit between v3.15 and v6.3.
Will take some time to set up and interpret findings of course.

Edit: I just observed that trying to play through AESS once also makes the legacy interface get stuck (which works before trying AESS). Which means we also have to look at the McPDM setup.

Risca · Mar 30, 2023

hns said:
... or IRQ or DMA. Or something else.
With some kernel debugging enabled I get:

Code:

[ 331.359714] omap-abe-twl6040 sound: playback write error (DMA or IRQ trouble?)

which comes from the function wait_for_avail().
This is now hardest rock to break through.
Well, a known brute force trick is to devmem2 (or kernel equivalent) dump important registers in clock setup, interface setup, DMA setup, AESS setup etc. and compare bit by bit between v3.15 and v6.3.
Will take some time to set up and interpret findings of course.

Edit: I just observed that trying to play through AESS once also makes the legacy interface get stuck (which works before trying AESS). Which means we also have to look at the McPDM setup.

For the clock setup, there's a nice interface in debugfs so see what's going on: /sys/kernel/debug/clk/clk_summary

Bosbeetle · Mar 30, 2023

I don't have anything to add but this thread reads like a good detective novel, keep up the great work!
If you get to a stage where stuff is testable in a regular pyra setting just let me know what tests you need.

ElPoco · Mar 30, 2023

Bosbeetle said:
I don't have anything to add but this thread reads like a good detective novel

Debugging, the detective novel where you're the detective, the victim and the criminal.

jedi23 · Mar 30, 2023

IMHO, the criminal is cleary TI who totally abandoned this SoC and we the users are the victims suffering from this mess of not getting the full potential out of it.
Glad there are detectives on the case!
The suspense factor in this thread is in deed quite high

hns · Mar 30, 2023

Well, it's a mixture of detective work, medicine and archeology - and actually just a computer game

hns · Apr 2, 2023

Risca said:
For the clock setup, there's a nice interface in debugfs so see what's going on: /sys/kernel/debug/clk/clk_summary

Some quick continuation of the novel:
The issue with /sys/kernel/debug/clk is that the output format has significantly changed between v3.15 and v6.3 so that I can't really compare that way.
What I have started is to really printk() the register contents. Most values I have checked so far are reasonable but there are some differences to be studied. And I have not checked all of them.
And one observation: there are no AESS interrupts in /proc/interrupts on the v6.3 kernel. So maybe the AE is simply not running and notifying the A15 cores that it has processed some audio samples from the ping-pong FIFO buffers.

Risca · Apr 19, 2023

I was browsing through the device tree for linux-5.6.19-daveiii-pyradef-aufs, the current Linux version of the pyra, and I could find references toaess_fclk, but nothing which directly uses it. The abe_iclk node has a reference to it, but nothing has a reference to abe_iclk.

I couldn't see that the omap4-aess driver requests any clocks or does any clk_prepare_enable() either.lk

Does this peripheral not need any clock to be enabled?

Furthermore, the aess driver has this code:

C:

        /* aess_clk has to be enabled to access hal register.
         * Disable the clk after it has been used.
         */
        pm_runtime_get_sync(aess->dev);
...
        pm_runtime_put_sync(aess->dev);

but I don't see the driver defining anything useful for pm_runtime. I might have overlooked stuff though. Looks like the aess driver private data has a clk pointer, but I don't ever see it get set or used anywhere:

C:

/*
 * AESS private data.
 */
struct omap_aess {
        struct device *dev;

        struct clk *clk;
...

if anything, I guess the clock should be requested in omap_aess_engine_probe() along with the rest of the resources it needs.

That's just a brain dump before I head for bed. Hopefully, @hns can make some use of it =)

Farox · Apr 20, 2023

i was testing the FastTracker 2 clone and i saw many errors on the shell (probably related to pulseaudio and the incomplete AESS implementation)..
the error messages are:

pa_write() failed while trying to wake up the mainloop: Broken pipe
pa_write() failed while trying to wake up the mainloop: Broken pipe
pa_write() failed while trying to wake up the mainloop: Broken pipe
pa_write() failed while trying to wake up the mainloop: Broken pipe
pa_write() failed while trying to wake up the mainloop: Broken pipe
pa_write() failed while trying to wake up the mainloop: Broken pipe
pa_write() failed while trying to wake up the mainloop: Broken pipe
pa_write() failed while trying to wake up the mainloop: Broken pipe

now if i see with the command pactl list sources short:

sudo pactl list sources short

0 alsa_output.hw_CARD_L15_DEV_0.monitor module-alsa-sink.c s32le ch 4 88200 Hz SUSPENDED
1 speakers.monitor module-remap-sink.c s32le ch 2 88200 Hz SUSPENDED
2 headphones.monitor module-remap-sink.c s32le ch 2 88200 Hz SUSPENDED

instead on my linux PC i have

0 alsa_output.pci-0000_00_14.2.analog-stereo.monitor module-alsa-card.c s16le ch 2 44100 Hz IDLE
1 alsa_input.pci-0000_00_14.2.analog-stereo module-alsa-card.c s16le ch 2 44100 Hz SUSPENDED

it's really strange have 88200Hz and s32le or not?

Thanks

Risca · Apr 20, 2023

I did a quick comparison of the aess_fclk between old (working) kernel:

Code:

root@old:~# cat /sys/kernel/debug/clk/virt_19200000_ck/sys_clkin/abe_dpll_clk_mux/dpll_abe_ck/dpll_abe_x2_ck/dpll_abe_m2x2_ck/abe_clk/aess_fclk/clk_prepare_count
1
root@old:~# cat /sys/kernel/debug/clk/virt_19200000_ck/sys_clkin/abe_dpll_clk_mux/dpll_abe_ck/dpll_abe_x2_ck/dpll_abe_m2x2_ck/abe_clk/aess_fclk/clk_enable_count 
0
<start playing something>
root@old:~# cat /sys/kernel/debug/clk/virt_19200000_ck/sys_clkin/abe_dpll_clk_mux/dpll_abe_ck/dpll_abe_x2_ck/dpll_abe_m2x2_ck/abe_clk/aess_fclk/clk_prepare_count
1
root@old:~# cat /sys/kernel/debug/clk/virt_19200000_ck/sys_clkin/abe_dpll_clk_mux/dpll_abe_ck/dpll_abe_x2_ck/dpll_abe_m2x2_ck/abe_clk/aess_fclk/clk_enable_count 
1

And something newer:

Code:

root@new:~# cat /sys/kernel/debug/clk/clk_summary
                                 enable  prepare  protect                                duty  hardware
   clock                          count    count    count        rate   accuracy phase  cycle    enable
-------------------------------------------------------------------------------------------------------
...
                   aess_fclk          0        0        0   196608000          0     0  50000         Y

I didn't try to play anything on a new kernel. I don't have the latest Letux kernel up and running.

However, something enables aess_fclk when I play audio. I just haven't figured out how it happens

hns · May 14, 2023

Risca said:

I did a quick comparison of the aess_fclk between old (working) kernel:

Code:

root@old:~# cat /sys/kernel/debug/clk/virt_19200000_ck/sys_clkin/abe_dpll_clk_mux/dpll_abe_ck/dpll_abe_x2_ck/dpll_abe_m2x2_ck/abe_clk/aess_fclk/clk_prepare_count
1
root@old:~# cat /sys/kernel/debug/clk/virt_19200000_ck/sys_clkin/abe_dpll_clk_mux/dpll_abe_ck/dpll_abe_x2_ck/dpll_abe_m2x2_ck/abe_clk/aess_fclk/clk_enable_count
0
<start playing something>
root@old:~# cat /sys/kernel/debug/clk/virt_19200000_ck/sys_clkin/abe_dpll_clk_mux/dpll_abe_ck/dpll_abe_x2_ck/dpll_abe_m2x2_ck/abe_clk/aess_fclk/clk_prepare_count
1
root@old:~# cat /sys/kernel/debug/clk/virt_19200000_ck/sys_clkin/abe_dpll_clk_mux/dpll_abe_ck/dpll_abe_x2_ck/dpll_abe_m2x2_ck/abe_clk/aess_fclk/clk_enable_count
1

And something newer:

Code:

root@new:~# cat /sys/kernel/debug/clk/clk_summary
                                 enable  prepare  protect                                duty  hardware
   clock                          count    count    count        rate   accuracy phase  cycle    enable
-------------------------------------------------------------------------------------------------------
...
                   aess_fclk          0        0        0   196608000          0     0  50000         Y

I didn't try to play anything on a new kernel. I don't have the latest Letux kernel up and running.

However, something enables aess_fclk when I play audio. I just haven't figured out how it happens

Finally found time to check this (with letux-6.4-rc1):

Code:

root@letux:~# cat /sys/kernel/debug/clk/abe_clk/clk_prepare_count
1
root@letux:~# cat /sys/kernel/debug/clk/abe_clk/clk_enable_count
1
root@letux:~# cat /sys/kernel/debug/clk/clk_summary |fgrep abe_
       abe_dpll_bypass_clk_mux_ck       0        0        0    38400000          0     0  50000         Y
    abe_dpll_refclk_mux_ck            1        1        0       32768          0     0  50000         Y
       dpll_abe_ck                    1        1        0    98304000          0     0  50000         ?
          dpll_abe_m2_ck              0        0        0    98304000          0     0  50000         Y
             per_abe_nc_fclk          0        0        0    98304000          0     0  50000         Y
             per_abe_24m_fclk         0        0        0    24576000          0     0  50000         Y
          dpll_abe_x2_ck              1        1        0   196608000          0     0  50000         Y
             dpll_abe_m3x2_ck         0        0        0    98304000          0     0  50000         Y
             dpll_abe_m2x2_ck         1        2        0    98304000          0     0  50000         Y
                abe_clk               1        1        0    98304000          0     0  50000         Y
                      ocp_abe_iclk       0        0        0    49152000          0     0  50000         Y
                abe_24m_fclk          0        6        0    12288000          0     0  50000         Y
root@letux:~#

Looks good to me on first inspection. Maybe ocp_abe_iclk isn't enabled. And the clocks have been renamed.

matzesu · May 14, 2023

Great!! Allthought I don’t like Bounty .. hope there Who found a solution can chose in the end a different chocolate bar ^^

hns · May 14, 2023

Playing sound hangs in the function wait_for_avail() where I dump some interesting registers.

Code:

[ 4602.150329] wait_for_avail:
[ 4602.153259] dump all relevant OMAP4 AESS registers
[ 4602.158264] ABE_PRM:
[ 4602.160583] 4a306500: 03 01 33 00 37 03 00 00 00 00 00 00 00 00 00 00
[ 4602.167358] 4a306510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 4602.174102] 4a306520: 00 00 00 00 00 00 00 00 00 00 00 00 01 01 00 00
[ 4602.180877] 4a306530: 00 00 00 00 01 01 00 00 00 00 00 00 01 01 00 00
[ 4602.187652] 4a306540: 00 00 00 00 01 00 00 00 00 00 00 00 01 01 00 00
[ 4602.194427] 4a306550: 00 00 00 00 01 01 00 00 00 00 00 00 01 01 00 00
[ 4602.201171] 4a306560: 00 00 00 00 01 01 00 00 00 00 00 00 01 00 00 00
[ 4602.207946] 4a306570: 00 00 00 00 01 00 00 00 00 00 00 00 01 00 00 00
[ 4602.214721] 4a306580: 00 00 00 00 01 00 00 00 01 00 00 00 01 00 00 00
[ 4602.221466] ABE_CM1:
[ 4602.223785] 4a004500: 02 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 4602.230560] 4a004510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 4602.237304] 4a004520: 01 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00
[ 4602.244079] 4a004530: 02 00 00 00 00 00 00 00 00 00 03 00 00 00 00 00
[ 4602.250854] 4a004540: 00 00 03 00 00 00 00 00 00 00 03 00 00 00 00 00
[ 4602.257598] 4a004550: 00 00 03 00 00 00 00 00 00 00 03 00 00 00 00 00
[ 4602.264373] 4a004560: 00 00 03 00 00 00 00 00 00 00 03 00 00 00 00 00
[ 4602.271148] 4a004570: 00 00 03 00 00 00 00 00 00 00 03 00 00 00 00 00
[ 4602.277923] 4a004580: 00 00 03 00 00 00 00 00 00 00 03 00
[ 4602.283569] MCPDM:
[ 4602.285705] 49032000: 00 08 00 50 00 00 00 00 00 00 00 00 00 00 00 00
[ 4602.292480] 49032010: 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 4602.299255] 49032020: 00 00 00 00 00 04 00 00 00 00 00 00 0a 0a 00 00
[ 4602.306030] 49032030: 0a 0a 00 00 03 00 00 00 03 00 00 00 03 00 00 00
[ 4602.312774] 49032040: 03 00 00 00 fb 42 00 00 00 00 00 00 00 45 bb 9e
[ 4602.319549] 49032050: 02 00 00 00 02 00 00 00 1f 01 00 00 00 00 00 00
[ 4602.326324] 49032060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 4602.333099] 49032070: 00 00 00 00 00 00 00 00

The same on 3.15:

Code:

[  148.717407] wait_for_avail:
[  148.723846] dump all relevant OMAP4 AESS registers
[  148.728881] ABE_PRM:
[  148.731201] 4a306500: 01 01 33 00 37 03 00 01 00 00 00 00 00 00 00 00
[  148.737976] 4a306510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[  148.744750] 4a306520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[  148.751525] 4a306530: 00 00 00 00 00 00 00 00 00 00 00 00 01 01 00 00
[  148.758300] 4a306540: 00 00 00 00 01 00 00 00 00 00 00 00 01 01 00 00
[  148.765075] 4a306550: 00 00 00 00 01 01 00 00 00 00 00 00 01 01 00 00
[  148.771850] 4a306560: 00 00 00 00 01 01 00 00 00 00 00 00 01 00 00 00
[  148.778625] 4a306570: 00 00 00 00 01 00 00 00 00 00 00 00 01 00 00 00
[  148.785400] 4a306580: 00 00 00 00 01 00 00 00 01 00 00 00 01 00 00 00
[  148.792175] ABE_CM1:
[  148.792175] 4a004500: 03 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[  148.792175] 4a004510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[  148.792175] 4a004520: 01 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00
[  148.814788] 4a004530: 02 00 00 00 00 00 00 00 00 00 03 00 00 00 00 00
[  148.821563] 4a004540: 00 00 03 00 00 00 00 00 00 00 03 00 00 00 00 00
[  148.828338] 4a004550: 00 00 03 00 00 00 00 00 00 00 03 00 00 00 00 00
[  148.828338] 4a004560: 00 00 03 00 00 00 00 00 00 00 03 00 00 00 00 00
[  148.841888] 4a004570: 00 00 03 00 00 00 00 00 00 00 03 00 00 00 00 00
[  148.841888] 4a004580: 00 00 03 00 00 00 00 00 00 00 03 00
[  148.841888] MCPDM:
[  149.096496] 49032000: 00 08 00 50 00 00 00 00 00 00 00 00 00 00 00 00
[  149.096496] 49032010: 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[  149.096496] 49032020: 00 00 00 00 00 00 00 00 00 00 00 00 0a 0a 00 00
[  149.110046] 49032030: 0a 0a 00 00 03 00 00 00 03 00 00 00 03 00 00 00
[  149.110046] 49032040: 03 00 00 00 fb 42 00 00 00 00 00 ff 00 00 00 80
[  149.130340] 49032050: 42 51 00 00 42 10 00 00 00 00 00 00 00 00 00 00
[  149.137115] 49032060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[  149.137115] 49032070: 00 00 00 00 00 00 00 00

Risca · May 15, 2023

I did a quick check with the reference manual and the registers that I found which differs in your output are either "reserved" and/or something about "context lost" status bits. I'm not sure about the endianess of your output, but in either case I found nothing ground breaking

hns · May 15, 2023

something about "context lost" status bits.

Thanks for doing some research!
Indeed I had to comment out some code fragments that handle lost context registers because it is no longer supported by modern Kernel APIs.
Maybe we have to ask Tony Lindgren (omap maintainer) about this.

directive0 · Jun 28, 2023

I think putting this up for bounty was a good idea!

I would love to be able to contribute to the bounties for bugs. If I we as Pyra owners could put money up towards whatever feature or bug is bothering each of us the most that may help push things along and also help indicate community priority? In any case, it may take a small part of the financial burden off ED.

ClockworkCoder · Jun 28, 2023

directive0 said:
I would love to be able to contribute to the bounties for bugs

Second this, although I might be reticent to contribute much until I actually receive a Pyra... :-(

hns · Jun 30, 2023

Well, it is certainly time to write a small update. Well, there is not much update to report.
Code sill compiles fine, creates all necessary components from firmware, behaves identical on omap4 and omap5 but does not talk to the AESS signal processor.
Otherwise I am currently preparing to rebase for v6.5-rc1 which heavily restructures the device tree files for ARM processors. This will be a good thing.
But the main reason is that I did not find time (i.e. a week) without many disruptions an other unfinished taks to focus on this. Better time will come. So stay tuned...

Risca · Jul 8, 2023

@hns, is there a pre-compiled AESS firmware blob (omap_aess-adfw.bin) available for download anywhere?

As I understand it, the firmware have been remade to work on more modern kernels

I tried compiling it from source in letux-kernel:aess-6.4-rc3 branch, but got a lot of compiler errors:

Code:

$ make
Making all in src
make[1]: Entering directory '/storage/dev/omap5432_evm/letux-kernel/sound/soc/ti/aess/firmware/abefw/src'
gcc -DPACKAGE_NAME=\"abegen\" -DPACKAGE_TARNAME=\"abegen\" -DPACKAGE_VERSION=\"0.1\" -DPACKAGE_STRING=\"abegen\ 0.1\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DHAVE_STDIO_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_STRINGS_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_UNISTD_H=1 -DHAVE_WCHAR_H=1 -DSTDC_HEADERS=1 -D_ALL_SOURCE=1 -D_DARWIN_C_SOURCE=1 -D_GNU_SOURCE=1 -D_HPUX_ALT_XOPEN_SOCKET_API=1 -D_NETBSD_SOURCE=1 -D_OPENBSD_SOURCE=1 -D_POSIX_PTHREAD_SEMANTICS=1 -D__STDC_WANT_IEC_60559_ATTRIBS_EXT__=1 -D__STDC_WANT_IEC_60559_BFP_EXT__=1 -D__STDC_WANT_IEC_60559_DFP_EXT__=1 -D__STDC_WANT_IEC_60559_FUNCS_EXT__=1 -D__STDC_WANT_IEC_60559_TYPES_EXT__=1 -D__STDC_WANT_LIB_EXT2__=1 -D__STDC_WANT_MATH_SPEC_FUNCS__=1 -D_TANDEM_SOURCE=1 -D__EXTENSIONS__=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -I.    -I../src -I../tasks -Werror -Wall -I /storage/dev/omap5432_evm/letux-kernel/ -I /storage/dev/omap5432_evm/letux-kernel/arch/arm/include -I /storage/dev/omap5432_evm/letux-kernel/arch/arm/include/generated -I /storage/dev/omap5432_evm/letux-kernel/include -I /storage/dev/omap5432_evm/letux-kernel/arch/arm/include/uapi -I /storage/dev/omap5432_evm/letux-kernel/arch/arm/include/generated/uapi -I /storage/dev/omap5432_evm/letux-kernel/include/uapi -I /storage/dev/omap5432_evm/letux-kernel/include/generated/uapi -include /storage/dev/omap5432_evm/letux-kernel/include/linux/compiler-version.h -include /storage/dev/omap5432_evm/letux-kernel/include/linux/kconfig.h -include /storage/dev/omap5432_evm/letux-kernel/include/linux/compiler_types.h -include /storage/dev/omap5432_evm/letux-kernel/include/linux/types.h -D__KERNEL__ -D__LINUX_ARM_ARCH__=7 -g -O2 -MT abegen-abegen.o -MD -MP -MF .deps/abegen-abegen.Tpo -c -o abegen-abegen.o `test -f 'abegen.c' || echo './'`abegen.c
In file included from /usr/include/stdlib.h:395,
                 from abegen.c:67:
/usr/include/x86_64-linux-gnu/sys/types.h:42:18: error: conflicting types for ‘loff_t’; have ‘__loff_t’ {aka ‘long int’}
   42 | typedef __loff_t loff_t;
      |                  ^~~~~~
In file included from <command-line>:
/storage/dev/omap5432_evm/letux-kernel/include/linux/types.h:46:33: note: previous declaration of ‘loff_t’ with type ‘loff_t’ {aka ‘long long int’}
   46 | typedef __kernel_loff_t         loff_t;
      |                                 ^~~~~~
...

I'm compiling on x86_64 Ubuntu server machine.

I don't have a native compiler available on the target omap5432 uEVM board, unfortunately. Buildroot doesn't support it :/

Risca · Jul 8, 2023

I used the Letux makesd utility to create a SD card and could extract a firmware from there:

Code:

DEV=/dev/sdc ./makesd pyra

However, it also fails to load:

Code:

[   23.993652] omap-abe-twl6040 sound: ASoC: invalid header size for type 0 at offset 0x0 size 0x1a5e8.
[   24.003191] omap-abe-twl6040 sound: ASoC: topology: invalid header: -22
[   24.010098] aess 401f1000.aess: loading toplogy from AESS FW failed -22
[   24.016742] aess 401f1000.aess: ASoC: error at snd_soc_component_probe on 401f1000.aess: -22
[   24.025986] omap-abe-twl6040 sound: ASoC: failed to instantiate card -22
[   24.033822] omap-abe-twl6040 sound: card registration failed: -22
[   24.040233] omap-abe-twl6040: probe of sound failed with error -22

Doing a quick glance at the files they seem quite similar. Only 8 bytes difference in size and content looks similar as well

Audio Driver (with bounty!)

Well-Known Member

Very Active Member

Terminally lost

Hardcore Member

Very Active Member

Well-Known Member

Well-Known Member

Very Active Member

Certified Guru

Very Active Member

Well-Known Member

Forum Addict!

Well-Known Member

Very Active Member

Well-Known Member

Very Active Member

Chaotic Neutral

Well-Known Member

Very Active Member

Very Active Member

Similar threads