Thanks, that would be great! If it is indeed your configuration, it might be interesting to compare it.

Maybe Nikolaus can provide his kernel/uboot as well. You've got the most recent EVM, we should check whether they've fixed a few things here...

On November 18, 2016 1:14:47 AM GMT+01:00, Matthijs van Duin matthijsvanduin@gmail.com wrote:

On 18 November 2016 at 00:59, Michael Mrozek EvilDragon@openpandora.org wrote:

...

The reboot... yes, I've seen that. I remember I always had plug out

the

...

power of the EVM completely, otherwise, it never really rebooted.

Has anyone looked into this? Does this still happen or does anyone know why?

I'm not 100% sure but I don't think I've ever really had any problems with rebooting. I'll try a couple of reboots when I get to the office again.

I'm using a 12V adapter, but if I understand you correctly that isn't a factor for this issue you're seeing. Perhaps it's due to differences in my kernel config or rest of system (debian stretch without initramfs).

Matthijs

Am 18.11.2016 um 07:15 schrieb H. Nikolaus Schaller hns@goldelico.com:

...

Am 18.11.2016 um 00:59 schrieb Michael Mrozek EvilDragon@openpandora.org:

Hi,

as Nikolaus is currently working on improving the voltage connections on the CPU board, it came to our mind that the EVM (which we based our design on) also might have some bugs.

Unfortunately, I don't have an EVM here anymore (it's at the touchscreen manufacturers place so they could test the screen), so I can't help, but as I know some here have an EVM and / or an IGEPv5, maybe you can help.

Here are the issues Nikolaus had that occur both on the EVM AND on the Pyra.

• Unstable at 1,5GHz. Crashes under high-load and when doing memtest.

I have tried with letux-4.9-rc5 (letux defconfig) linus/v4.9-rc5 (omap2plus defconfig) ti/v4.8.3-rt6

I have compiled some ti/v3.8.something but it did not boot (might need different bootargs and no DT)

...

• Doesn't properly reboot after a reset

but only after such a crash.

...

As far as I know, the instability only happens whn using an 5V adaptor on the EVM (at least I ran it under high-load with a 12V AC adaptor for quite a while). Can someone test and confirm this? Also, does memtest (maybe Nikolaus can provide the compiled binary for that) also crash when using 5V?

apt-get install memtester

$ memtester 1500M

...

The reboot... yes, I've seen that. I remember I always had plug out the power of the EVM completely, otherwise, it never really rebooted.

For me it reboots if I normally do a reset. Only in that special case of a CPU crash something in the EVM seems to be locked up so that pressing the reset button makes the kernel boot process randomly hang between 5 and 10 seconds. U-Boot always starts properly.

I have a new interesting observation.

In multiple of these reset+mlo+u-boot+kernel-stuck sequences the last activity I see is like this:

[ 4.360252] ehci-omap: OMAP-EHCI Host Controller driver [ 4.375779] ehci-omap 4a064c00.ehci: EHCI Host Controller [ 4.381644] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1 [ 4.398233] ehci-omap 4a064c00.ehci: irq 404, io mem 0x4a064c00

Tony reported some time ago a potential stability issue with EHCI. I will try to follow this scent.

This also persists if I unlug the power supply for short time. It appears that I must unplug >30 seconds and then everything works fine again.

...

Has anyone looked into this? Does this still happen or does anyone know why? And if yes, does that happen on the igepV5 as well?

It's interesting that our Pyra boards behave a bit like the EVM when powered with 5V. I hope there's no bug in the power-supply circuit which we cloned into our CPU board.

Then it would be how the Palmas is set up.

...

Thanks for any help here :)

Of course, Nikolaus is welcome to add some comments in case I've forgotten something.

-- Mit freundlichen Grüßen,

Michael Mrozek

---

OpenPandora GmbH Geschäftsführer: Michael Mrozek

Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt

---

eMail: mrozek@openpandora.org _______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel

---

Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel

Hi,

Am 18.11.2016 um 11:26 schrieb Grazvydas Ignotas notasas@gmail.com:

On Fri, Nov 18, 2016 at 8:15 AM, H. Nikolaus Schaller hns@goldelico.com wrote:

...

...

Am 18.11.2016 um 00:59 schrieb Michael Mrozek EvilDragon@openpandora.org:

Hi,

as Nikolaus is currently working on improving the voltage connections on the CPU board, it came to our mind that the EVM (which we based our design on) also might have some bugs.

Unfortunately, I don't have an EVM here anymore (it's at the touchscreen manufacturers place so they could test the screen), so I can't help, but as I know some here have an EVM and / or an IGEPv5, maybe you can help.

Here are the issues Nikolaus had that occur both on the EVM AND on the Pyra.

• Unstable at 1,5GHz. Crashes under high-load and when doing memtest.

I have tried with letux-4.9-rc5 (letux defconfig) linus/v4.9-rc5 (omap2plus defconfig) ti/v4.8.3-rt6

That is with DDR voltage fix, right?

yes.

What does "high-load" mean exactly?

Driving both cores to 100% load by running bash in loops:

http://git.goldelico.com/?p=gta04-kernel.git;a=blob;f=Letux/root/high-load;h...

Well, this might not be an optimal test since in theory every piece could fit in I/D-caches, but it is unlikely.

BR, Nikolaus

Hi,

Am 19.11.2016 um 17:49 schrieb Grazvydas Ignotas notasas@gmail.com:

Hi,

On Fri, Nov 18, 2016 at 12:36 PM, H. Nikolaus Schaller hns@goldelico.com wrote:

...

...

What does "high-load" mean exactly?

Driving both cores to 100% load by running bash in loops:

http://git.goldelico.com/?p=gta04-kernel.git;a=blob;f=Letux/root/high-load;h...

Well, this might not be an optimal test since in theory every piece could fit in I/D-caches, but it is unlikely.

From our experience of overclocking pandora we know that "simple" loops like that don't really stress the CPU much, overclocked pandora will crash a lot sooner if NEON is used (and it will use more power). Try the attached program instead, it heats the CPU to 100C much quicker, meanwhile a bash loop takes a lot of time to do the same:

50C->100C on my uEVM (heatsink attached, room temperature 20C or so) bash: does not reach 100C (only does if the board is covered by antistatic bag)

Mine does reach 100°C in ca. 10 minutes @1.5Ghz.

Pyra stabilizes ~94°C @1Ghz after 1.5 hours running time.

neon: ~11min

Compile with: gcc -o neon_loop neon_loop.S -mfpu=neon -mcpu=native The program is probably far from a maximum stresser, but it at least tries to load all the CPU pipelines with work, including NEON.

Fine & thanks! I will give it a try asap.

That also brings to another observation: the current kernel is doing thermal throttling. If you watch current CPU frequency while the temperature is ~100C, you'll see the frequency sometimes dropping to 1GHz even with both cores at 100% and performance governor active.

Exactly. This lasts ca. 2-3 seconds and then it goes up to 1.5GHz again. And the temperature which has fallen to ~82°C goes up fast until it reaches 100°C again.

This cycle takes approx. 10 seconds.

And indeed, /sys/class/thermal/thermal_zone0/trip_point_0_temp specifies 100000, which must be the point it starts to throttle.

Well, then we at least know that this works!

Now, for my uEVM tests:

• my usual 12V supply: still rock stable on 4.9.0-rc5-letux+ (>1h test)

I've found a supply ED sent me long ago for GP2X (it needs some unusual voltage) which has a voltage switch:

• 4.5V setting, multimeter shows ~4.8V at the board connector on load
• hung after ~20min or so

Yes. That is the EVM@5V issue.

• 5V setting, ~5.3V actually measured on load - survived for 14h (2x

neon_loop + memtester, also tried bash loop + memtester for a few hours)

I now have found a full explanation for this EVM@5V phenomenon.

I have extended the ./temperatures script a little so that it reads the PALMAS gpadc VSYS channel and prints it as well.

The first result is that with 12V power input the VSYS_COBRA (input voltage of the PALMAS) is already a little lower than the designed 3.7V. And again lower with 5V.

VSYS is ca. 3.6V with 12V supply and 3.4V with 5V supply in console idle.

But as soon as I start the high-load script VSYS_COBRA breaks down even more. Not as much with 12V supply but significantly with 5V supply.

So this is half of the explanation, because it is still in a reasonable range.

Then, we have a second effect. Power consumption of the OMAP5 appears to increase with temperature, at least at high clock frequencies. I.e. as the OMAP5 is heating up, VSYS_COBRA goes down even more. Again much more with 5V supply than with 12V supply.

The lowest VSYS_COBRA I have seen was 3.088V. And this with a nominal 3.7V input voltage.

Interestingly, the voltage jumps up as soon as the clock frequency is thermally throttled to 1GHz and starts to go down again as soon as the governor switches back to 1.5GHz.

It appears that at some moment VSYS_COBRA becomes too low for the PALMAS regulators to continue. Or, the Palmas detects an undervoltage situation but unlike the twl4030 does not simply power off.

Or it submits an undervoltage notification interrupt that is not processed any more. That is something we could find in the kernel code.

This is the moment the EVM freezes.

So this is the exact explanation why the EVM is inherently more instable at 5V than at 12V.

The following factors are part of the explanation: * the DC/DC converter regulation on the EVM is much worse @5V than @12V - I have not tried to understand why it is so much worse because it does not help us - maybe the copper traces are too weak, or the DC/DC isn't designed for <4.5V input voltage and has much worse performance - and the resistance of the power cable may also play a significant role * the power demand goes up with OMAP5 load *and* temperature * if VSYS voltage becomes too low the OMAP hangs instead of PALMAS turning off NOTE: I have no exact definition and explanation what "too low" is and how it makes the OMAP freeze (SMPS123 regulation failure? PALMAS lockup?)

Now what does that mean for the Pyra?

Unfortunately almost nothing. On the Pyra I see a quite stable 3.6V VSYS_COBRA almost independent on load (@1GHz even 1.25GHz) and temperature. The lowest value I did see was 3.588V. But the Pyra did still freeze @1.25Ghz after some minutes but not at 1GHz. I was not yet able to test 1.5GHz.

So we now only know that the Pyra freeze has a different reason than the EVM@5V freeze although they look very similar in results.

I hope your new NEON load test gives us new insights.

And I am in the middle of adding some missing driver code to be able to set the Pyra VSYS from /sys/bq24297. Then I can try to find out if freezes are more likely at lower or higher voltages. This might shed some light on the influence of the PALMAS DC/DC converters in this freeze issue.

The only question to the EVM where I still have no answer for is why RESET does not properly boot after a low-voltage freeze while it works fine otherwise. And why this persists for a while even if powered off.

This is something I have seen similarily on the Pyra - where it happens earlier (MLO complains and asks for a RESET).

The only speculation I have is that it is some setting in the PALMAS which survives by the backup battery.

BR and thanks, Nikolaus

Hi Roy,

Am 18.11.2016 um 15:25 schrieb Roy Gillotti gillottir@gmail.com:

Is there anything I can test?, I know I did that experiment a few months ago, bumping up the supply voltage and what not. I was seeing an instability while running on 5V.

Ok!

I haven't tried to power up the devboard while limiting the CPU to a lower clock frequency.

I'm also still using an older kernel where the PVR is working properly, so I haven't played with any of the newer kernels yet.

The strategy I am currently trying to follow is to run the EVM at 1.5GHz@5V and find some working setup that does not hang. If necessary with a very old kernel.

If we can manage to find such a config we can try that on the Pyra hardware and see if we still have the problem there. If it is also gone we finally know that there is some hidden instability in modern kernels - although we have no explanation. Then we can look for it and fix the kernel.

If we still have the problem on the Pyra it is very likely a hardware difference which can not be fixed by software changes.

So if you could invest some time to locate a working old (3.8 to 3.12) kernel, that would be of big help. I have tried tag v3.8.13.9-rt20 from git://git.ti.com/ti-linux-kernel/ti-linux-kernel.git but it did hang after "Starting Kernel...". Maybe it was just the console setup or bootargs but I don't know.

BR and thanks, Nikolaus

On Thu, Nov 17, 2016 at 6:59 PM, Michael Mrozek EvilDragon@openpandora.org wrote: Hi,

as Nikolaus is currently working on improving the voltage connections on the CPU board, it came to our mind that the EVM (which we based our design on) also might have some bugs.

Unfortunately, I don't have an EVM here anymore (it's at the touchscreen manufacturers place so they could test the screen), so I can't help, but as I know some here have an EVM and / or an IGEPv5, maybe you can help.

Here are the issues Nikolaus had that occur both on the EVM AND on the Pyra.

• Unstable at 1,5GHz. Crashes under high-load and when doing memtest.
• Doesn't properly reboot after a reset

As far as I know, the instability only happens whn using an 5V adaptor on the EVM (at least I ran it under high-load with a 12V AC adaptor for quite a while). Can someone test and confirm this? Also, does memtest (maybe Nikolaus can provide the compiled binary for that) also crash when using 5V?

The reboot... yes, I've seen that. I remember I always had plug out the power of the EVM completely, otherwise, it never really rebooted.

Has anyone looked into this? Does this still happen or does anyone know why? And if yes, does that happen on the igepV5 as well?

It's interesting that our Pyra boards behave a bit like the EVM when powered with 5V. I hope there's no bug in the power-supply circuit which we cloned into our CPU board.

Thanks for any help here :)

Of course, Nikolaus is welcome to add some comments in case I've forgotten something.

-- Mit freundlichen Grüßen,

Michael Mrozek

---

OpenPandora GmbH Geschäftsführer: Michael Mrozek

Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt

---

eMail: mrozek@openpandora.org _______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel

---

Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel

New test with the EVM and official Pandora 5V adapter, the red warning LED shows up on the board. I boot with the pre-compiled Letux-4.9.0-rc5 on Slackware -current armhf.

3 neon loops, 1 memtester on 2G. No problem after more than 30 hours, dmesg is clean. cpufreq stats: 1000 MHz:0.15%, 1.50 GHz:99.85% 64°C

Do I let it run ?

Le 2016-11-23 08:41, H. Nikolaus Schaller a écrit :

Hi,

...

Am 22.11.2016 um 23:03 schrieb Linux-SWAT openpandora@free.fr:

New test with the EVM and official Pandora 5V adapter, the red warning LED shows up on the board.

I am not sure if it is intended to be a "warning" LED... (Table 1 of the EVM description calls it "Red Power Input Voltage Indicator").

It shows up only with 5V plugged in, not with 12V.

...

I boot with the pre-compiled Letux-4.9.0-rc5 on Slackware -current armhf.

3 neon loops, 1 memtester on 2G. No problem after more than 30 hours, dmesg is clean. cpufreq stats: 1000 MHz:0.15%, 1.50 GHz:99.85% 64°C

Interesting! That seems to be fine!

...

Do I let it run ?

Can you also check the VSYS_COBRA voltage of your setup? It might be higher than mine @5V. It should by theory be 3.7V but in practice I have seen it to go down to 3.088V.

Here is a script:

http://git.goldelico.com/?p=gta04-kernel.git;a=blob_plain;f=Letux/root/tempe...

which locates the gpadc inside the PALMAS and reads the VSYS voltage.

Here it is : Wed Nov 23 09:13:24 -00 2016 64° 47° 51° 3449mV 1500MHz

And still running fine : $ uptime 09:17:06 up 1 day, 18:06, 1 user, load average: 4.38, 4.40, 4.37

LS

I removed the ldrd and strd instructions lines, then started 3 neon loops on EVM at 1.7GHz 5V, no memory test.

Dunno if it's useful, but anyway here's some results before going to sleep :

$ sh findthermal.sh Thu Nov 24 00:44:13 -00 2016 71° 50° 54° 3319mV 1700MHz Thu Nov 24 00:44:22 -00 2016 70° 49° 53° 3308mV 1700MHz Thu Nov 24 00:44:25 -00 2016 71° 49° 53° 3301mV 1700MHz Thu Nov 24 00:44:33 -00 2016 71° 50° 54° 3321mV 1700MHz Thu Nov 24 00:44:37 -00 2016 71° 50° 54° 3316mV 1700MHz Thu Nov 24 00:44:46 -00 2016 71° 50° 54° 3296mV 1700MHz Thu Nov 24 00:44:51 -00 2016 71° 50° 54° 3318mV 1700MHz Thu Nov 24 00:45:07 -00 2016 71° 50° 54° 3316mV 1700MHz Thu Nov 24 00:45:14 -00 2016 72° 50° 54° 3306mV 1700MHz Thu Nov 24 00:45:18 -00 2016 70° 50° 55° 3316mV 1700MHz Thu Nov 24 00:45:21 -00 2016 72° 51° 55° 3312mV 1700MHz Thu Nov 24 00:45:23 -00 2016 71° 50° 55° 3316mV 1700MHz Thu Nov 24 00:45:25 -00 2016 70° 51° 54° 3305mV 1700MHz Thu Nov 24 00:45:28 -00 2016 71° 50° 54° 3310mV 1700MHz Thu Nov 24 00:47:11 -00 2016 72° 51° 55° 3305mV 1700MHz Thu Nov 24 00:47:17 -00 2016 73° 51° 55° 3291mV 1700MHz Thu Nov 24 00:48:05 -00 2016 72° 52° 55° 3304mV 1700MHz Thu Nov 24 00:48:17 -00 2016 72° 52° 56° 3295mV 1700MHz Thu Nov 24 00:52:15 -00 2016 74° 53° 57° 3294mV 1700MHz Thu Nov 24 00:52:25 -00 2016 74° 53° 57° 3286mV 1700MHz Thu Nov 24 00:54:10 -00 2016 75° 54° 57° 3278mV 1700MHz Thu Nov 24 00:54:52 -00 2016 75° 53° 58° 3282mV 1700MHz Thu Nov 24 00:54:54 -00 2016 75° 53° 57° 3290mV 1700MHz Thu Nov 24 00:56:03 -00 2016 75° 53° 57° 3288mV 1700MHz Thu Nov 24 00:56:07 -00 2016 75° 54° 58° 3289mV 1700MHz Thu Nov 24 00:56:09 -00 2016 75° 53° 58° 3277mV 1700MHz Thu Nov 24 00:57:31 -00 2016 76° 54° 58° 3291mV 1700MHz Thu Nov 24 00:57:34 -00 2016 74° 54° 58° 3288mV 1700MHz Thu Nov 24 00:58:21 -00 2016 75° 54° 58° 3282mV 1700MHz Thu Nov 24 00:58:35 -00 2016 76° 54° 58° 3286mV 1700MHz Thu Nov 24 00:59:31 -00 2016 76° 55° 58° 3274mV 1700MHz Thu Nov 24 00:59:50 -00 2016 76° 53° 58° 3257mV 1700MHz Thu Nov 24 01:02:04 -00 2016 76° 55° 59° 3283mV 1700MHz Thu Nov 24 01:02:28 -00 2016 77° 54° 59° 3269mV 1700MHz Thu Nov 24 01:02:40 -00 2016 76° 54° 59° 3268mV 1700MHz Thu Nov 24 01:02:49 -00 2016 76° 55° 59° 3256mV 1700MHz Thu Nov 24 01:04:03 -00 2016 77° 53° 59° 3267mV 1700MHz Thu Nov 24 01:06:54 -00 2016 77° 55° 59° 3268mV 1700MHz Thu Nov 24 01:09:22 -00 2016 77° 55° 59° 3256mV 1700MHz

I don't know how to track an eventual CPU throttling, but as ED said, I should stay under the °C limit so it should not happen.

I finally had a 9V crash, it was with the pre-compiled Letux 4.9-rc5 @1.5GHz, but I wasn't trying to trigger it. I was simply using the EVM as distcc server (helping to compile a kernel on another board) while git cloning the github Letux kernel. Maybe the I/O "helped" here. Unfortunately, nothing was outputted in the serial console nor the syslogs.

New test with the EVM and official Pandora 5V adapter, the red warning LED shows up on the board. I boot with the pre-compiled Letux-4.9.0-rc5 on Slackware -current armhf.

3 neon loops, 1 memtester on 2G. No problem after more than 30 hours, dmesg is clean. cpufreq stats: 1000 MHz:0.15%, 1.50 GHz:99.85% 64°C

Is that with the Stock Heatsink, didn't you say on the forums that you replaced the heatsink when overclocking? Is it still passively cooled?

Do I let it run ?

On Tue, Nov 22, 2016 at 5:03 PM, Linux-SWAT openpandora@free.fr wrote:

New test with the EVM and official Pandora 5V adapter, the red warning LED shows up on the board. I boot with the pre-compiled Letux-4.9.0-rc5 on Slackware -current armhf.

3 neon loops, 1 memtester on 2G. No problem after more than 30 hours, dmesg is clean. cpufreq stats: 1000 MHz:0.15%, 1.50 GHz:99.85% 64°C

Do I let it run ?

---

Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel