Tests, tests, and more tests.


Swordfish II

Advanced Member
Joined
May 20, 2015
Messages
1,046
Well, unfortunately, it improved the situation but it didn't fix it.
After 3:10 hours, it froze again.

notaz also found out that the board doesn't seem to freeze when you run memtester and at the same time stress the CPU.
This probably also is the reason why the full apt-get upgrade for Nikolaus worked. As long as the Pyra is not idling, it doesn't freeze.
So it could be related to some powersaving that's triggering (with such system, powersaving already triggers when there's only 1ms of idling happening).

So the interesting new findings:

* Disabling the modules leads to the freeze occur less likely
* When stressing the memory and the CPU, it doesn't freeze
* Nikolaus mentioned that even a stable 2GB RAM LC15 Eval board freezes when using the LPAE kernel - but otherwise works fine.
* However, on our current 4GB RAM boards, using LPAE or non-LPAE doesn't make a difference

This seems to be a mix of multiple things leading to the freeze - annoying to pin-point, but we're working on that!

My (unqualified) guess is that it's a combination of both powersaving AND hardware being active at least a bit.

If it would only be powersaving, then there shouldn't be any difference whether we run it with all modules enabled or not.
When the modules are enabled, the system is doing more, but not enough to prevent the freeze from happening (as it still freezes within minutes).
When the modules are disabled, the system doesn't really do a lot - so whatever the combination of powersaving / and the other part is, does occur less likely.

I'll also try older kernels (where OMAP-Powersaving is not yet fully included) to see whether that changes anything.
Certainly sounds like a powersave issue. Modules/hardware loaded and trying to work without enough power freezes. Disabling the modules saves some power, but eventually some process needs more power then it gets and freeze.
 

Djoga'Ro

moonstruck
Joined
Apr 3, 2016
Messages
1,526
Certainly sounds like a powersave issue. Modules/hardware loaded and trying to work without enough power freezes. Disabling the modules saves some power, but eventually some process needs more power then it gets and freeze.
So, you're saying instead of a powersave issue it's more of a back-to-full-throttle issue?
 

levi

Still fresh, damnit!
Joined
Oct 6, 2008
Messages
12,885
Location
Somewhere off the coast of the EU
Well, unfortunately, it improved the situation but it didn't fix it.
This probably also is the reason why the full apt-get upgrade for Nikolaus worked. As long as the Pyra is not idling, it doesn't freeze.
So it could be related to some powersaving that's triggering (with such system, powersaving already triggers when there's only 1ms of idling happening).
You mentioned on the mailing list that the SoC gets quite hot while it's frozen, so it's almost certainly in a tight loop going round and round and doing nothing while stopping the SoC powering down at all. I don't think it's getting stuck in a low power mode, or receiving a dodgy clock that's stopped the pipelines triggering or anything like that, but it could still be preparing to powersave when it hits this loop.

Actually, I don't know if a loop tight enough to fit in the top level cache will be sufficient to make the unit settle on a warm enough equilibrium to feel positively hot. So it might need to be a slightly larger loop that causes cache flushes, or gets some power hungry component within the SoC into some indeterminate state that churns current.
 

ClockworkCoder

Chaotic Neutral
Joined
Jan 21, 2016
Messages
1,719
Location
Menzoberranzan
You mentioned on the mailing list that the SoC gets quite hot while it's frozen.
Took me a couple of reads before I made sense of that! :p

I wish I had more to offer in terms of useful input. I'm really impressed how hard people are working to resolve this. Times like this show how fantastic the community is here.
 

Silent-Hunter

Hardcore Member
Joined
May 29, 2010
Messages
3,220
I wish I could help too, but I only know a small amount of C and Fortran, I'm no kernel developer.
 

Djoga'Ro

moonstruck
Joined
Apr 3, 2016
Messages
1,526
How do power state changes work, anyway? Does the software issue a special instruction or write some specific value to a particular register and the SoC does the rest? Or does the software have to handle some management around it, too?
 

AndiTheBest

Active Member
Joined
Aug 29, 2011
Messages
81
Age
33
Location
Ried im Innkreis, Austria
How do you check the kernel log when its freezing? Look at the logfiles, or is a serial port connected? A serial log may put out more infos before freeze than writing a logfile.
 

bukkit

Member
Joined
Jun 7, 2010
Messages
221
Well, unfortunately, it improved the situation but it didn't fix it.
After 3:10 hours, it froze again.
That's sad, but it doesn't make your planned test (loading drivers, trying for 15 minutes) useless. You could still fence in one issue wit that method and reduce the problem's overall complexity.
 

levi

Still fresh, damnit!
Joined
Oct 6, 2008
Messages
12,885
Location
Somewhere off the coast of the EU
@AndiTheBest Serial console out over the microUSB2 charging/debug port, I think I've read.

How do power state changes work, anyway? Does the software issue a special instruction or write some specific value to a particular register and the SoC does the rest? Or does the software have to handle some management around it, too?
As far as I can tell, it's largely implementation dependent. The ARM core spec doesn't provide any commands to suspend anything or change the clock speed, but I guess TI have either implemented coprocessor instructions or a special location on the memory map that allows software to control this stuff. It's not like the i686 world where different suspend modes are implemented in the silicon, so you likely at least need to slow the clock, and bring various subsystems down from live. How much of that is implemented yet remains to be seen though; I remember Notaz got some extra hours of idle out from the Pandora by stopping powering the system's 5V rail (for USB power out mostly). As far as I remember during the OMAP3 days with the Pandora we got the dynamic clock through the ondemand governor, and everything else was coded by Notaz though, and it seems like linux these days is slightly more advanced at scaling things back when running on ARM boards out of the box.
 

zedr0k

Member
Joined
Oct 28, 2015
Messages
30
Location
Germany
ICQ
62214606
...keep my fingers crossed... great news

Gesendet von meinem Aquaris M5 mit Tapatalk
 

BeMoreViking

Member
Joined
Apr 30, 2016
Messages
30
Location
USA
Not sure I understand the post on the mailing list. Anybody know what "DLL timing" is, or what "disable internal DLL trick" means?
 
Top