Hi, thanks to combining some test approaches we have recently developed together, I have done some new tests with new insights.
What I have played with are these factors: a) high-load with bash-loop vs. NEON-loop (the assembler program from Notaz) b) different VSYS settings in bq24297 (3.5V ... 3.85V), i.e. what is fed into the PALMAS c) different OMAP5 clock range (including/excluding the 1.25GHz OPP) d) try to evaluate at different temperatures (which is difficult since OMAP heats up fast)
All this was done on a single CPU board (C22 a 2GB RAM from the new production. This board has a retrofitted VDD-MPU bypass wire. I need to compare with other CPU boards as well).
This did lead to some interesting results and with them we can fortunately exclude some potential reasons. And it seems as if there is a combination which can trigger a freeze within seconds, so that we don't have to wait for 30 minutes if it occurs or not. This is important for quickly evaluating potential fixes.
Now here are the key results: * changing VSYS between 3.5 and 3.85V has no influence * running bash-loop works fine at 1.2GHz at any VSYS and temperature * VSYS goes down a little with rising temperature but this effect is much lower (less than 50mV) than on the EVM@12V * bash-loop heats up fast but does not freeze * it shows that the thermal throttling makes clock OPP go down to 750/500 MHz, sometimes in multiple steps * since this OPP change also includes VDD-MPU changes (between 1.25V and 0.85V) it shows that the Palmas regulation works fine * what makes a really big difference is running the bash-loop vs. the NEON-loop * high-load freezes almost immediately after switching from 1GHz to 1.25GHz I did not get more than two reports with 1250MHz. And this independent of VSYS or temperature! Suche a deciding factor is what I have been looking for for a while.
The freeze-matrix looks like:
| 1.0 GHz | 1.25GHz | | bash | NEON | NEON | bash | -------------------------+-------+-------+-------+-------+ low VSYS | ok | ok |freeze | ok | high ----------+-------+-------+-------+-------+ Temperature high VSYS | ok | ok |freeze | ok | -------------------------+-------+-------+-------+-------+ low high VSYS | ok | ok |freeze | ok | Temperature ----------+-------+-------+-------+-------+ low VSYS | ok | ok |freeze | ok | -------------------------+-------+-------+-------+-------+
This IMHO excludes: * it is not a temperature dependent effect (that would have been the worst case) * it is not a problem of the Palmas regulator switching between OPPs * it is indeed the Cortex A15 MPU/NEON core and not any peripheral on the OMAP5 SoC A weird reason could still be that the DRAM interface is used differently/faster if we run NEON at >1GHz. But the EMIF DRAM clock is constant if CPU clock changes. * it is unlikely a kernel software problem: the kernel does not explicitly control the combination of user-space NEON + CPU clock frequency * it is not a (direct) effect of the bq24297 VSYS output (although it might still be a short breakdown. I want to run another test which is much easier to do now with a known "freeze" command. Anyways this effect should be influenced by VSYS).
This means that the list of potential reasons is rapidly reducing.
As soon as there is only one remaining, we know where an improvement has the highest chance of fixing it with a single shot. It is more likely now that it is indeed a weakness in decoupling. The question will remain how small or big it is. This would define the amount of PCB change we need.
BR, Nikolaus
Here are some examples how the measurements looks like.
A) 1.25GHz enabled:
root@letux:~# ./high-load -n 100% load stress test for 2 cores Sun Jan 2 07:20:54 UTC 2000 39° 38° 39° 3816mV 1000MHz Sun Jan 2 07:20:54 UTC 2000 39° 38° 39° 3822mV 1000MHz Sun Jan 2 07:20:56 UTC 2000 40° 41° 39° 3808mV 1250MHz Sun Jan 2 07:20:57 UTC 2000 50° 42° 41° 3805mV 1250MHz -- freeze --
root@letux:~# i2cset -f -y 1 0x6b 0x01 0x1f root@letux:~# ./high-load 100% load stress test for 2 cores Sun Jan 2 07:32:57 UTC 2000 54° 56° 52° 3819mV 1000MHz Sun Jan 2 07:32:58 UTC 2000 54° 56° 52° 3819mV 1000MHz Sun Jan 2 07:32:59 UTC 2000 55° 55° 53° 3808mV 1250MHz Sun Jan 2 07:33:00 UTC 2000 65° 57° 57° 3808mV 1250MHz .... Sun Jan 2 07:35:01 UTC 2000 94° 87° 83° 3786mV 1250MHz Sun Jan 2 07:35:03 UTC 2000 96° 89° 85° 3787mV 1250MHz Sun Jan 2 07:35:04 UTC 2000 97° 91° 86° 3788mV 1250MHz Sun Jan 2 07:35:06 UTC 2000 99° 92° 87° 3799mV 1250MHz Sun Jan 2 07:35:07 UTC 2000 99° 93° 88° 3800mV 1250MHz Sun Jan 2 07:35:09 UTC 2000 99° 93° 88° 3813mV 750MHz Sun Jan 2 07:35:10 UTC 2000 100° 93° 89° 3826mV 500MHz Sun Jan 2 07:35:12 UTC 2000 101° 89° 83° 3824mV 750MHz Sun Jan 2 07:35:13 UTC 2000 89° 89° 83° 3804mV 1250MHz Sun Jan 2 07:35:15 UTC 2000 86° 87° 82° 3802mV 1250MHz Sun Jan 2 07:35:16 UTC 2000 94° 87° 84° 3783mV 1250MHz Sun Jan 2 07:35:18 UTC 2000 98° 91° 87° 3781mV 1250MHz Sun Jan 2 07:35:19 UTC 2000 99° 92° 86° 3781mV 1250MHz Sun Jan 2 07:35:21 UTC 2000 100° 93° 88° 3822mV 750MHz Sun Jan 2 07:35:22 UTC 2000 100° 93° 88° 3825mV 500MHz Sun Jan 2 07:35:24 UTC 2000 89° 90° 84° 3824mV 1000MHz Sun Jan 2 07:35:25 UTC 2000 89° 87° 81° 3794mV 1250MHz Sun Jan 2 07:35:27 UTC 2000 87° 87° 81° 3800mV 1250MHz Sun Jan 2 07:35:28 UTC 2000 94° 88° 84° 3780mV 1250MHz Sun Jan 2 07:35:30 UTC 2000 97° 90° 86° 3800mV 1250MHz ^Ckill 2554 2555
root@letux:~# ./high-load -n 100% load stress test for 2 cores Sun Jan 2 07:35:35 UTC 2000 87° 89° 84° 3815mV 1000MHz Sun Jan 2 07:35:35 UTC 2000 87° 89° 84° 3815mV 1000MHz -- freeze --
B) cores limited to 1GHz:
root@letux:~# ./high-load -n 100% load stress test for 2 cores Sun Jan 2 08:22:59 UTC 2000 45° 44° 43° 3825mV 750MHz Sun Jan 2 08:22:59 UTC 2000 45° 44° 43° 3826mV 750MHz Sun Jan 2 08:23:01 UTC 2000 43° 43° 42° 3821mV 1000MHz Sun Jan 2 08:23:02 UTC 2000 49° 44° 43° 3820mV 1000MHz Sun Jan 2 08:23:04 UTC 2000 52° 46° 46° 3822mV 1000MHz Sun Jan 2 08:23:05 UTC 2000 53° 48° 47° 3821mV 1000MHz .... Sun Jan 2 09:03:18 UTC 2000 99° 96° 90° 3807mV 1000MHz Sun Jan 2 09:03:20 UTC 2000 100° 95° 90° 3813mV 1000MHz Sun Jan 2 09:03:21 UTC 2000 100° 95° 90° 3814mV 1000MHz Sun Jan 2 09:03:23 UTC 2000 100° 95° 91° 3814mV 1000MHz Sun Jan 2 09:03:24 UTC 2000 100° 95° 91° 3814mV 1000MHz Sun Jan 2 09:03:26 UTC 2000 101° 96° 91° 3825mV 500MHz Sun Jan 2 09:03:27 UTC 2000 100° 95° 89° 3819mV 750MHz Sun Jan 2 09:03:29 UTC 2000 93° 94° 89° 3814mV 1000MHz Sun Jan 2 09:03:30 UTC 2000 92° 93° 87° 3811mV 1000MHz Sun Jan 2 09:03:32 UTC 2000 96° 93° 88° 3815mV 1000MHz Sun Jan 2 09:03:33 UTC 2000 98° 93° 89° 3814mV 1000MHz Sun Jan 2 09:03:35 UTC 2000 99° 94° 89° 3814mV 1000MHz Sun Jan 2 09:03:36 UTC 2000 99° 95° 89° 3814mV 1000MHz Sun Jan 2 09:03:38 UTC 2000 100° 95° 89° 3814mV 1000MHz Sun Jan 2 09:03:39 UTC 2000 100° 95° 90° 3820mV 1000MHz Sun Jan 2 09:03:41 UTC 2000 100° 96° 90° 3815mV 1000MHz -- no freeze yet --