Am 08.11.2016 um 19:35 schrieb Michael Mrozek EvilDragon@openpandora.org:
Hi,
unfortunately, even with the new CPU boards that had the SMPS3 trace fixed (supposedly), we still have freezes with frequencies higher than 1GHz.
It COULD be that the manufacturer accidentally used the wrong gerber files to produce the boards and the fix hasn't been implemented, but it's unlikely. Nikolaus will test that within the next few days though by measuring an unpopulated board.
What do we know?
- The system runs perfectly stable with both processors up to 1GHz.
- The system runs stable with ONE processor enabled up to 1,5GHz (this
hasn't been tested with the current board, but it had been tested before)
I had configured the kernel for SMP=n
- With more than 1GHz and 2 CPUs, the system freezes as soon as the
kernel raises the frequency (the default startup speed of the OMAP5 is 1GHz). This happens after about 6 - 10 seconds.
- Sometimes with 1,25 GHz (one out of 15 times), it boots up and then
even runs the high-load test for a while (so it's unlikely to be a power supply issue).
Right now we think a power supply issue can mostly be ruled out.
Why?
- With 1GHz, I made the following test:
Power Supply: Via USB (5V) and the charger chip, no battery supplied. System running with ALL LEDs at full strength, Display connected (with full brightness), Wifi enabled and downloading data (apt-get upgrade) AND a script that puts the CPU under 100% constant load.
No crash for over 6 hours (then I shut it down).
With 1,5GHz enabled, it crashes even when no display, no LED, no Wifi, etc. is enabled and running. So the system is using a LOT less power.
- Nikolaus measured VSYS during the tests and it doesn't break down,
so it doesn't seem to have a power issue.
At least no significant variations. I can not 100% exclude short (<1µs) spikes but there are quite some decoupling capacitors on VSYS.
So IF there is a problem, it has most likely to do with the CPU-Board, the OMAP5 itself or the software configuration (or a mixture of everything).
Interestingly, someone had a similar issue with a DRA7: e2e.ti.com/support/arm/sitara_arm/f/791/p/527643/2023441
I have also seen the spurious "Unable to handle kernel paging request at virtual address" in one of the 10 attempts at 1.5GHz.
Some ideas, but not solved in any way.
I even asked my question there and already received a reply: https://e2e.ti.com/support/arm/sitara_arm/f/791/p/527643/2023458#2023458
How can we find out what causes this and prevent it?
Apparently, something goes wrong when the kernel changes the frequency. The question is: What and how can we debug this? And why does it only happen when two CPUs are enabled?
Some testing ideas I had:
- Is it possible to only enable frequencies up to 1GHz in the board
files and then later (after the boot) add 1,25 and 1,5GHz? Then we could test whether there is some weird conflict or race condition while booting.
- Is it possible to only enable 1CPU during boot and add the second CPU
later on?
- Is it possible to run a kernel that runs slower and spits out more
debug messages, so we can narrow the results?
* another idea is to try to setup JTAG to find out in which state of the OMAP5 is (and through I2C) the Palmas as well.
This is really tricky since we do not have wired out the official JTAG lines, except that we have some JTAG-self-test using GPIO mode. But this assumes that the Cortex-A15 (or some Cortex M4) are working.
But JTAG is possible through the SDCARD interface which is the right SD slot where we currently boot from. As far as I understand:
- we need to be able to boot from eMMC or the µSD (needs proper control of the TXS02612 switch (in MLO, U-Boot and Kernel) - we need a SD shaped hardware adapter to get the JTAG signals to a probe maybe with level-shifters (I don't know) - we need an MLO/U-Boot/Kernel variant that switches pinmux of the SD interface
Alternatively it could be possible to boot from SATA (if we can get MLO on a hard drive). I have fixed the SATA adapter today, so that I know the SATA hardware works.
IMHO this has a high chance to find out in which state the Palmas is. My hope is that it reports some error condition to the OMAP which does not handle it any more.
So we need someone who owns the required JTAG equipment and is experienced enough to go this way.
BR, Nikolaus