Hi,
Am 19.11.2016 um 17:49 schrieb Grazvydas Ignotas notasas@gmail.com:
Hi,
On Fri, Nov 18, 2016 at 12:36 PM, H. Nikolaus Schaller hns@goldelico.com wrote:
What does "high-load" mean exactly?
Driving both cores to 100% load by running bash in loops:
http://git.goldelico.com/?p=gta04-kernel.git;a=blob;f=Letux/root/high-load;h...
Well, this might not be an optimal test since in theory every piece could fit in I/D-caches, but it is unlikely.
From our experience of overclocking pandora we know that "simple" loops like that don't really stress the CPU much, overclocked pandora will crash a lot sooner if NEON is used (and it will use more power). Try the attached program instead, it heats the CPU to 100C much quicker, meanwhile a bash loop takes a lot of time to do the same:
50C->100C on my uEVM (heatsink attached, room temperature 20C or so) bash: does not reach 100C (only does if the board is covered by antistatic bag)
Mine does reach 100°C in ca. 10 minutes @1.5Ghz.
Pyra stabilizes ~94°C @1Ghz after 1.5 hours running time.
neon: ~11min
Compile with: gcc -o neon_loop neon_loop.S -mfpu=neon -mcpu=native The program is probably far from a maximum stresser, but it at least tries to load all the CPU pipelines with work, including NEON.
Fine & thanks! I will give it a try asap.
That also brings to another observation: the current kernel is doing thermal throttling. If you watch current CPU frequency while the temperature is ~100C, you'll see the frequency sometimes dropping to 1GHz even with both cores at 100% and performance governor active.
Exactly. This lasts ca. 2-3 seconds and then it goes up to 1.5GHz again. And the temperature which has fallen to ~82°C goes up fast until it reaches 100°C again.
This cycle takes approx. 10 seconds.
And indeed, /sys/class/thermal/thermal_zone0/trip_point_0_temp specifies 100000, which must be the point it starts to throttle.
Well, then we at least know that this works!
Now, for my uEVM tests:
- my usual 12V supply: still rock stable on 4.9.0-rc5-letux+ (>1h test)
I've found a supply ED sent me long ago for GP2X (it needs some unusual voltage) which has a voltage switch:
- 4.5V setting, multimeter shows ~4.8V at the board connector on load
- hung after ~20min or so
Yes. That is the EVM@5V issue.
- 5V setting, ~5.3V actually measured on load - survived for 14h (2x
neon_loop + memtester, also tried bash loop + memtester for a few hours)
I now have found a full explanation for this EVM@5V phenomenon.
I have extended the ./temperatures script a little so that it reads the PALMAS gpadc VSYS channel and prints it as well.
The first result is that with 12V power input the VSYS_COBRA (input voltage of the PALMAS) is already a little lower than the designed 3.7V. And again lower with 5V.
VSYS is ca. 3.6V with 12V supply and 3.4V with 5V supply in console idle.
But as soon as I start the high-load script VSYS_COBRA breaks down even more. Not as much with 12V supply but significantly with 5V supply.
So this is half of the explanation, because it is still in a reasonable range.
Then, we have a second effect. Power consumption of the OMAP5 appears to increase with temperature, at least at high clock frequencies. I.e. as the OMAP5 is heating up, VSYS_COBRA goes down even more. Again much more with 5V supply than with 12V supply.
The lowest VSYS_COBRA I have seen was 3.088V. And this with a nominal 3.7V input voltage.
Interestingly, the voltage jumps up as soon as the clock frequency is thermally throttled to 1GHz and starts to go down again as soon as the governor switches back to 1.5GHz.
It appears that at some moment VSYS_COBRA becomes too low for the PALMAS regulators to continue. Or, the Palmas detects an undervoltage situation but unlike the twl4030 does not simply power off.
Or it submits an undervoltage notification interrupt that is not processed any more. That is something we could find in the kernel code.
This is the moment the EVM freezes.
So this is the exact explanation why the EVM is inherently more instable at 5V than at 12V.
The following factors are part of the explanation: * the DC/DC converter regulation on the EVM is much worse @5V than @12V - I have not tried to understand why it is so much worse because it does not help us - maybe the copper traces are too weak, or the DC/DC isn't designed for <4.5V input voltage and has much worse performance - and the resistance of the power cable may also play a significant role * the power demand goes up with OMAP5 load *and* temperature * if VSYS voltage becomes too low the OMAP hangs instead of PALMAS turning off NOTE: I have no exact definition and explanation what "too low" is and how it makes the OMAP freeze (SMPS123 regulation failure? PALMAS lockup?)
Now what does that mean for the Pyra?
Unfortunately almost nothing. On the Pyra I see a quite stable 3.6V VSYS_COBRA almost independent on load (@1GHz even 1.25GHz) and temperature. The lowest value I did see was 3.588V. But the Pyra did still freeze @1.25Ghz after some minutes but not at 1GHz. I was not yet able to test 1.5GHz.
So we now only know that the Pyra freeze has a different reason than the EVM@5V freeze although they look very similar in results.
I hope your new NEON load test gives us new insights.
And I am in the middle of adding some missing driver code to be able to set the Pyra VSYS from /sys/bq24297. Then I can try to find out if freezes are more likely at lower or higher voltages. This might shed some light on the influence of the PALMAS DC/DC converters in this freeze issue.
The only question to the EVM where I still have no answer for is why RESET does not properly boot after a low-voltage freeze while it works fine otherwise. And why this persists for a while even if powered off.
This is something I have seen similarily on the Pyra - where it happens earlier (MLO complains and asks for a RESET).
The only speculation I have is that it is some setting in the PALMAS which survives by the backup battery.
BR and thanks, Nikolaus