Tests, tests, and more tests.

Discussion in 'Pyra News' started by EvilDragon, Sep 15, 2018.

  1. Grench

    Grench Forum Addict!

    Joined:
    Oct 3, 2008
    Messages:
    5,761
    Jumping entirely out of the box here, so this is a bit convoluted and maybe a bit frightening.

    ***** Thought 1 **********
    What other platforms/systems/things use these exact same memory modules?
    What do those systems use for a CPU & OS? Is there any chance to see how they are addressing this RAM?

    ***** Thought 2 **********
    If I recall correctly, there are two types of RAM surface mount patterns that the Pyra's SoC board can accept. 4x512 uses one pattern and the 4x1024 uses another pattern. Heat means thermal expansion. Is there any chance whatsoever that one or more of those connections might be a bit tenuous when the board heats up?

    ***** Thought 3 **********
    I don't know if it was established yet - does the 4x512 configuration reliably work? We know that the 4x1024 configuration works until it mysteriously doesn't. But - are we 100% certain that this issue does not affect the 4x512? Are there 4x1024 modules that use the same mount pattern as the 4x512 which could be purchased in test quantities? The idea here is to ensure that the issue isn't with thermal expansion or issues with the mounting patterns.

    ***** Thought 4 **********
    This is being chased as if it is a RAM issue. Is there any chance that this is an OMAP432 issue? I.e. is there a condition set up in the OMAP5 when it sees 4GB of RAM that causes instability? Not that we would have any way to find this.

    ***** Thought 5 **********
    RAM is expensive. SoC modules are expensive. In the event of OMAP5+4GB=fail, why trash either? Stick with me through this chain of thought... If the OMAP5 is stable and working well with 2GB, make 1st generation Pyras with the OMAP5 and 2GB and ship them. Find a different (newer?) SoC that can also take those same 4x1024 modules and begin work on the 2nd generation Pyra SoC boards with <SoC TBD> and 4GB. (This would be a real convoluted long shot and gambles a big pile of EUR.)
     
    Silent-Hunter and Djoga'Ro like this.
  2. Djoga'Ro

    Djoga'Ro moonstruck

    Joined:
    Apr 3, 2016
    Messages:
    672
    IIRC, the purchased RAM is EOL. So, maybe no base to build the upgrade from.
    I don't know much about the SoC business, but I would imagine, current models rather use DDR4. ?
     
  3. Silent-Hunter

    Silent-Hunter Advanced Member

    Joined:
    May 29, 2010
    Messages:
    2,792
    What about trying a really old kernel then?
     
  4. BeMoreViking

    BeMoreViking Still Fresh

    Joined:
    Apr 30, 2016
    Messages:
    24
    Location:
    USA
    Let's not forget we haven't entirely ruled the issue out of the 2GB configuration. Maybe we should test those a bit more extensively to see how prevalent it is in the 2GB configuration.

    ED posted earlier in this thread:
    "8. Some 2GB RAM boards freeze as well, some occassionally, some never. We thought it was an issue with the production run... but maybe not!"​

    Also, do we know if this particular OMAP5 has been used in any production devices?
     
  5. TrashyMG

    TrashyMG Sarcasm Dispenser Staff Member

    Joined:
    Jan 18, 2010
    Messages:
    10,226
    I mean How old?, remember I've been using one for 4 years now, some where ones I compiled myself prior to Nikolaus, aTc and other being involved. Some kernels Letux, aTc's or otherwise I could run with months of uptime, generally only rebooted when I accidentally unplugged it or I wanted to upgrade/switch the OS.

    If you mean for the Pyra?, Well not sure if it's worth backporting a ton of pyra specific things instead of just figuring out what the problem is...
     
    Last edited: Oct 3, 2018
    FBnil likes this.
  6. Silent-Hunter

    Silent-Hunter Advanced Member

    Joined:
    May 29, 2010
    Messages:
    2,792
    What I mean is you said there's been a lot of activity kernel wise in the past year or so, and someone might have trampled something. So what about trying a kernel from before those changes?
     
    rygD likes this.
  7. pmprog

    pmprog Advanced Member

    Joined:
    Apr 25, 2011
    Messages:
    3,618
    Sure, but if you can prove the hangups are 100% kernel software, then that helps eliminate any hardware problems and could mean you could ship units and fix them in the future
     
    Swordfish II and Silent-Hunter like this.
  8. TrashyMG

    TrashyMG Sarcasm Dispenser Staff Member

    Joined:
    Jan 18, 2010
    Messages:
    10,226
    Well yeah, I'm not saying that someone shouldn't use older kernels to try and figure where things have broke. I just meant by my words is to Imply that the Devboards have ran stable in the past and this likely rules out the idea that there is an inherent issue with the OMAP5 as a SoC as some have been trying to imply in the prior posts.
     
    pmprog and Silent-Hunter like this.
  9. Djhg2000

    Djhg2000 Very Active Member

    Joined:
    Jul 22, 2014
    Messages:
    151
    Location:
    Sweden
    No not that long, just long enough to stretch the margins a bit. A deep frozen one would probably be out of spec on many components before they even start defrosting.
     
  10. EvilDragon

    EvilDragon Administrator Staff Member

    Joined:
    Mar 4, 2003
    Messages:
    19,420
    Location:
    Ingolstadt
    Just a quick update:
    Without loading the drivers except for those included in the kernel (I simply renamed the modules-directory), the board is now running fine for 40 minutes already.
    I'll let it run overnight, but usually it freezes after 5 - 10 minutes.

    So it could very well be that some driver is causing the issue.

    My further plans are to write a small script that modprobes one driver after another, waits for 15 minutes and then loads the next driver - until we see a freeze.
     
    Last edited: Oct 4, 2018
    rSl, error, spud42 and 17 others like this.
  11. second exodous

    second exodous Advanced Member

    Joined:
    Sep 27, 2005
    Messages:
    2,894
    Location:
    Utah, USA
    I have my fingers crossed, this would be awesome if it is just a software issue.
     
    rSl, Drezrek and Yorizuka like this.
  12. Djhg2000

    Djhg2000 Very Active Member

    Joined:
    Jul 22, 2014
    Messages:
    151
    Location:
    Sweden
    I'm confused, do you mean it's now running fine? If so that's great news!

    Edit:
    Should've checked the mailing list first. It's a typo.

    While I'm at it I can answer your question from there as well:
    Sounds to me like "lsmod" is what you're looking for. It also prints which modules are using a module as a dependency (for instance "usbcore" is being used by "xhci_hcd,usbhid,usb_storage,xhci_pci,uas" on one of my machines).
     
    Last edited: Oct 4, 2018
  13. levi

    levi Still fresh, damnit!

    Joined:
    Oct 6, 2008
    Messages:
    9,768
    Location:
    Somewhere off the coast of the EU
    Reading between the lines on the mailing list, it seems to me that it could well be specific combinations of modules that causes freezes. So to nail it down you may need to run your script multiple times inserting modules in a different order each time. There's probably an optimal strategy to figuring this out, but I'm not aware of what one is.
     
  14. Djoga'Ro

    Djoga'Ro moonstruck

    Joined:
    Apr 3, 2016
    Messages:
    672
    Has PoetterWare abandoned /etc/modules* ?
     
  15. hns

    hns Very Active Member

    Joined:
    Jul 16, 2012
    Messages:
    312
    Location:
    Oberhaching
    We already know from similar experiments I have done some weeks ago that the "pwm_omap_dmtimer" module makes the "reboot" (warm reset) issue appear. If we blacklist this module (and the twl6040 module), a "reboot" works fine. So I suspect that the "reboot" issue and the "freezes" have something in common. At least symptoms are similar (with the exception that setting earlycon reveals that the "reboot" hangs when trying to start/lock the abe_dpll).
     
    klapse, rSl, levi and 1 other person like this.
  16. Bosbeetle

    Bosbeetle Terminally lost

    Joined:
    Sep 7, 2008
    Messages:
    3,644
    Location:
    The Netherlands
    At least this points to software for sure, that must be a relief... Or is there still a possibility it is hardware related?
     
  17. Djhg2000

    Djhg2000 Very Active Member

    Joined:
    Jul 22, 2014
    Messages:
    151
    Location:
    Sweden
    AFAIK /etc/modules* only handles which modules should be loaded and how, not which ones are actually loaded? Are you perhaps thinking of /sys/module*?
     
    Djoga'Ro likes this.
  18. Djoga'Ro

    Djoga'Ro moonstruck

    Joined:
    Apr 3, 2016
    Messages:
    672
    No, I just learnt about it, setting up a little VM with Alpine Linux, which I needed to initialize fuse stuff at boot. I just threw in the question in hopes of learning something. :) Thought, what should be loaded, is what's of interest, though.
     
  19. EvilDragon

    EvilDragon Administrator Staff Member

    Joined:
    Mar 4, 2003
    Messages:
    19,420
    Location:
    Ingolstadt
    Well, unfortunately, it improved the situation but it didn't fix it.
    After 3:10 hours, it froze again.

    notaz also found out that the board doesn't seem to freeze when you run memtester and at the same time stress the CPU.
    This probably also is the reason why the full apt-get upgrade for Nikolaus worked. As long as the Pyra is not idling, it doesn't freeze.
    So it could be related to some powersaving that's triggering (with such system, powersaving already triggers when there's only 1ms of idling happening).

    So the interesting new findings:

    * Disabling the modules leads to the freeze occur less likely
    * When stressing the memory and the CPU, it doesn't freeze
    * Nikolaus mentioned that even a stable 2GB RAM LC15 Eval board freezes when using the LPAE kernel - but otherwise works fine.
    * However, on our current 4GB RAM boards, using LPAE or non-LPAE doesn't make a difference

    This seems to be a mix of multiple things leading to the freeze - annoying to pin-point, but we're working on that!

    My (unqualified) guess is that it's a combination of both powersaving AND hardware being active at least a bit.

    If it would only be powersaving, then there shouldn't be any difference whether we run it with all modules enabled or not.
    When the modules are enabled, the system is doing more, but not enough to prevent the freeze from happening (as it still freezes within minutes).
    When the modules are disabled, the system doesn't really do a lot - so whatever the combination of powersaving / and the other part is, does occur less likely.

    I'll also try older kernels (where OMAP-Powersaving is not yet fully included) to see whether that changes anything.
     
    bennr01, rSl, CommanderB and 12 others like this.
  20. mclien

    mclien Very Active Member

    Joined:
    Apr 16, 2012
    Messages:
    435
    Location:
    Hannover
    Maybe it's a good idea to ask bunny, if he knows tricks to get that speed up a bit? Most likely he has an open ear, plus it is not impissible to meet him in germany at the next ccc (35c3).
     
    comradekingu, Coupo and xnopasaranx like this.

Share This Page

Loading...