Tests, tests, and more tests.

[Silent-Hunter]

Mine been running fine for a while on a 4.x kernel of some flavo(u)r released 6 months ago, I've haven't played with it in a while with newer kernels...

There has been a lot of activity kernel wise with the OMAP5 in the past year or so, likely someone trampled on something.

What about trying a really old kernel then?

 

[BeMoreViking]

Let's not forget we haven't entirely ruled the issue out of the 2GB configuration. Maybe we should test those a bit more extensively to see how prevalent it is in the 2GB configuration.

ED posted earlier in this thread:

"8. Some 2GB RAM boards freeze as well, some occassionally, some never. We thought it was an issue with the production run... but maybe not!"​

Also, do we know if this particular OMAP5 has been used in any production devices?

 

[TrashyMG]

What about trying a really old kernel then?

I mean How old?, remember I've been using one for 4 years now, some where ones I compiled myself prior to Nikolaus, aTc and other being involved. Some kernels Letux, aTc's or otherwise I could run with months of uptime, generally only rebooted when I accidentally unplugged it or I wanted to upgrade/switch the OS.

If you mean for the Pyra?, Well not sure if it's worth backporting a ton of pyra specific things instead of just figuring out what the problem is...

 

[Silent-Hunter]

What I mean is you said there's been a lot of activity kernel wise in the past year or so, and someone might have trampled something. So what about trying a kernel from before those changes?

 

[pmprog]

If you mean for the Pyra?, Well not sure if it's worth backporting a ton of pyra specific things instead of just figuring out what the problem is...

Sure, but if you can prove the hangups are 100% kernel software, then that helps eliminate any hardware problems and could mean you could ship units and fix them in the future

 

[TrashyMG]

Sure, but if you can prove the hangups are 100% kernel software, then that helps eliminate any hardware problems and could mean you could ship units and fix them in the future

Well yeah, I'm not saying that someone shouldn't use older kernels to try and figure where things have broke. I just meant by my words is to Imply that the Devboards have ran stable in the past and this likely rules out the idea that there is an inherent issue with the OMAP5 as a SoC as some have been trying to imply in the prior posts.

 

[Djhg2000]

For better condensation?

No not that long, just long enough to stretch the margins a bit. A deep frozen one would probably be out of spec on many components before they even start defrosting.

 

[EvilDragon]

Just a quick update:
Without loading the drivers except for those included in the kernel (I simply renamed the modules-directory), the board is now running fine for 40 minutes already.
I'll let it run overnight, but usually it freezes after 5 - 10 minutes.

So it could very well be that some driver is causing the issue.

My further plans are to write a small script that modprobes one driver after another, waits for 15 minutes and then loads the next driver - until we see a freeze.

 

[second exodous]

. . .it could very well be that some driver is causing the issue.

I have my fingers crossed, this would be awesome if it is just a software issue.

 

[Djhg2000]

Just a quick update:
Without loading the drivers except for those included in the kernel (I simply renamed the modules-directory), the board is not running fine for 40 minutes already.

I'm confused, do you mean it's now running fine? If so that's great news!

Edit:
Should've checked the mailing list first. It's a typo.

While I'm at it I can answer your question from there as well:

Question: Is there a way to find out what modules are being loaded normally? So I know which ones I need to load manually with the script?

Sounds to me like "lsmod" is what you're looking for. It also prints which modules are using a module as a dependency (for instance "usbcore" is being used by "xhci_hcd,usbhid,usb_storage,xhci_pci,uas" on one of my machines).

 

[levi]

My further plans are to write a small script that modprobes one driver after another, waits for 15 minutes and then loads the next driver - until we see a freeze.

Reading between the lines on the mailing list, it seems to me that it could well be specific combinations of modules that causes freezes. So to nail it down you may need to run your script multiple times inserting modules in a different order each time. There's probably an optimal strategy to figuring this out, but I'm not aware of what one is.

 

[Djoga'Ro]

While I'm at it I can answer your question from there as well:
Sounds to me like "lsmod" is what you're looking for. It also prints which modules are using a module as a dependency (for instance "usbcore" is being used by "xhci_hcd,usbhid,usb_storage,xhci_pci,uas" on one of my machines).

Has PoetterWare abandoned /etc/modules* ?

 

[hns]

Reading between the lines on the mailing list, it seems to me that it could well be specific combinations of modules that causes freezes. So to nail it down you may need to run your script multiple times inserting modules in a different order each time. There's probably an optimal strategy to figuring this out, but I'm not aware of what one is.

We already know from similar experiments I have done some weeks ago that the "pwm_omap_dmtimer" module makes the "reboot" (warm reset) issue appear. If we blacklist this module (and the twl6040 module), a "reboot" works fine. So I suspect that the "reboot" issue and the "freezes" have something in common. At least symptoms are similar (with the exception that setting earlycon reveals that the "reboot" hangs when trying to start/lock the abe_dpll).

 

[Bosbeetle]

At least this points to software for sure, that must be a relief... Or is there still a possibility it is hardware related?

 

[Djhg2000]

Has PoetterWare abandoned /etc/modules* ?

AFAIK /etc/modules* only handles which modules should be loaded and how, not which ones are actually loaded? Are you perhaps thinking of /sys/module*?

 

[Djoga'Ro]

AFAIK /etc/modules* only handles which modules should be loaded and how, not which ones are actually loaded? Are you perhaps thinking of /sys/module*?

No, I just learnt about it, setting up a little VM with Alpine Linux, which I needed to initialize fuse stuff at boot. I just threw in the question in hopes of learning something. Thought, what should be loaded, is what's of interest, though.

 

[EvilDragon]

Well, unfortunately, it improved the situation but it didn't fix it.
After 3:10 hours, it froze again.

notaz also found out that the board doesn't seem to freeze when you run memtester and at the same time stress the CPU.
This probably also is the reason why the full apt-get upgrade for Nikolaus worked. As long as the Pyra is not idling, it doesn't freeze.
So it could be related to some powersaving that's triggering (with such system, powersaving already triggers when there's only 1ms of idling happening).

So the interesting new findings:

* Disabling the modules leads to the freeze occur less likely
* When stressing the memory and the CPU, it doesn't freeze
* Nikolaus mentioned that even a stable 2GB RAM LC15 Eval board freezes when using the LPAE kernel - but otherwise works fine.
* However, on our current 4GB RAM boards, using LPAE or non-LPAE doesn't make a difference

This seems to be a mix of multiple things leading to the freeze - annoying to pin-point, but we're working on that!

My (unqualified) guess is that it's a combination of both powersaving AND hardware being active at least a bit.

If it would only be powersaving, then there shouldn't be any difference whether we run it with all modules enabled or not.
When the modules are enabled, the system is doing more, but not enough to prevent the freeze from happening (as it still freezes within minutes).
When the modules are disabled, the system doesn't really do a lot - so whatever the combination of powersaving / and the other part is, does occur less likely.

I'll also try older kernels (where OMAP-Powersaving is not yet fully included) to see whether that changes anything.

 

[mclien]

Additionally, chinese companies support chinese companies - so they give them priority in production and delivery.

Maybe it's a good idea to ask bunny, if he knows tricks to get that speed up a bit? Most likely he has an open ear, plus it is not impissible to meet him in germany at the next ccc (35c3).

 

[vcoleiro1]

Well although the freezing issue is annoying, it's great to see the community coming together and helping with ideas.

 

[Swordfish II]

Well, unfortunately, it improved the situation but it didn't fix it.
After 3:10 hours, it froze again.

notaz also found out that the board doesn't seem to freeze when you run memtester and at the same time stress the CPU.
This probably also is the reason why the full apt-get upgrade for Nikolaus worked. As long as the Pyra is not idling, it doesn't freeze.
So it could be related to some powersaving that's triggering (with such system, powersaving already triggers when there's only 1ms of idling happening).

So the interesting new findings:

* Disabling the modules leads to the freeze occur less likely
* When stressing the memory and the CPU, it doesn't freeze
* Nikolaus mentioned that even a stable 2GB RAM LC15 Eval board freezes when using the LPAE kernel - but otherwise works fine.
* However, on our current 4GB RAM boards, using LPAE or non-LPAE doesn't make a difference

This seems to be a mix of multiple things leading to the freeze - annoying to pin-point, but we're working on that!

My (unqualified) guess is that it's a combination of both powersaving AND hardware being active at least a bit.

If it would only be powersaving, then there shouldn't be any difference whether we run it with all modules enabled or not.
When the modules are enabled, the system is doing more, but not enough to prevent the freeze from happening (as it still freezes within minutes).
When the modules are disabled, the system doesn't really do a lot - so whatever the combination of powersaving / and the other part is, does occur less likely.

I'll also try older kernels (where OMAP-Powersaving is not yet fully included) to see whether that changes anything.

Certainly sounds like a powersave issue. Modules/hardware loaded and trying to work without enough power freezes. Disabling the modules saves some power, but eventually some process needs more power then it gets and freeze.