Tests, tests, and more tests.

Discussion in 'Pyra News' started by EvilDragon, Sep 15, 2018.

  1. Grench

    Grench Forum Addict!

    Joined:
    Oct 3, 2008
    Messages:
    5,761
    Is there any chance that this is unrelated to the SoC or the RAM themselves? I.e. could there be a capacitor or voltage regulator in the power circuit somewhere that works until certain conditions are met (temp? current high/low?). Maybe you got a bad batch of -a- component that isn't working within spec X% of the time?

    I know that there was a lot of talk about whether or not to use Tantalum capacitors - could be completely unrelated of course.

    A lot of the features on these boards are -really- tiny. Finding the slightly bent needle in the needle stack could be frustrating - if there even is one.
     
  2. Askarus

    Askarus Hardcore Member

    Joined:
    Sep 28, 2011
    Messages:
    4,192
    Location:
    Germany
    I've asked that question on the Kernel Mailing List.
    Nikolaus told me they did try it several times without success.
    No luck ...
     
    Djhg2000 likes this.
  3. Silent-Hunter

    Silent-Hunter Advanced Member

    Joined:
    May 29, 2010
    Messages:
    2,792
    What did they try? Sorry, I don't understand.
     
  4. Askarus

    Askarus Hardcore Member

    Joined:
    Sep 28, 2011
    Messages:
    4,192
    Location:
    Germany
    To boot the 4GB boards with only 2GB enabled.

    ED stated: "7. 4GB RAM boards were working with only 2GB RAM enabled (as far as I remember - as that was the reason I ordered the RAM modules back then!)"

    And I was asking myself why no one is talking about that. Might have been an overlooked solution.
     
  5. Silent-Hunter

    Silent-Hunter Advanced Member

    Joined:
    May 29, 2010
    Messages:
    2,792
    But you just said Nikolaus said they tried that and it didn't work. Or am I misunderstanding something?
     
  6. den Tim

    den Tim Still Fresh

    Joined:
    May 15, 2018
    Messages:
    2
    I read the same as you: with only 2GB RAM enabled the 4GB boards worked as expected.
    What I now believe he meant was: the 4GB boards only had 2GB enabled and still had issues the 2GB boards didn't have.
     
  7. Djoga'Ro

    Djoga'Ro moonstruck

    Joined:
    Apr 3, 2016
    Messages:
    672
    To my understanding those test, when the bigger RAM chips worked while only half the capacity was enabled, were under different conditions. Different software and maybe different board revision.
     
  8. EvilDragon

    EvilDragon Administrator Staff Member

    Joined:
    Mar 4, 2003
    Messages:
    19,420
    Location:
    Ingolstadt
    No, they were working withous issues.
    I would've never ordered the RAM Chips if I knew they weren't working.
    I ordered them because they worked and we only had to enable the 4GB in software.
     
    Djhg2000 likes this.
  9. Nilsnsn

    Nilsnsn Member

    Joined:
    Jun 3, 2012
    Messages:
    75
    Location:
    Braunschweig
    I've read Nikolaus answer on the mailing list. Actually he doesn't say it was without success - he just states they tried it several times. No success might be implied, but i don't see it that clear.
    Additionally he talks about populating only half the chips, which in his opinion won't work.
    https://www.pyra-handheld.com/pipermail/kernel/2018-October/002818.html
     
  10. BeMoreViking

    BeMoreViking Still Fresh

    Joined:
    Apr 30, 2016
    Messages:
    24
    Location:
    USA
    Here's a couple other shots in the dark that I'll offer. They may or may not be feasible to test. Or maybe they've already been tested.

    1. Is it possible to get some engineering samples of a different memory chip to test out? I'm not suggesting that we'd want to switch to a different chip for production purposes (except as a very last resort), but it might be informative to know if the crashes are related to this particular chip model or not. We've certainly been through the ringer with them.

    2. Is it possible to test a stripped-down version of the PCB with all non-essential traces, subsystems, and components removed? Maybe order a couple stripped down PCBs from one of those hobbyist PCB fabrication services for testing?

    3. How about testing a super-stripped down kernel? Don't build in drivers for anything that's not absolutely essential (e.g. disable even video/display, use only the video out, or connect to it remotely, etc).

    4. Is it worth another audit of relevant kernel drivers for more 'B' vs '8' hex constants?

    5. These hangs aren't reproducible with our OS/kernel running on omap development boards right?

    6. Are there any physical interference details worth consideration? As a possibly unrelated example, USB3 and 2.4Ghz wifi are known to interfere with each-other. Any chance we have something similar? E.g. two components that interfere with each-other in too-close of proximity?
     
    Djhg2000 and Silent-Hunter like this.
  11. hns

    hns Very Active Member

    Joined:
    Jul 16, 2012
    Messages:
    312
    Location:
    Oberhaching
    We did have 4GB chip samples from different manufacturer and tried with the non-optimized board design. AFAIR, we did not see an improvement. It was only one CPU board where the chips were replaced and this was before memory timing improvements were understood. So this is not really comparable.
    We still need a 10 layer PCB and that is not available from any hobbyist PCB production. Expect 5000€ to build such a prototype.
    We already have built-in only those drivers that are essential (or we think are essential) so that you still can boot after
    Code:
    rm -rf /lib/modules
    . I am not sure that there was a systematic test yet. But AFAIR there are still freezes with all loadable modules removed.
    Yes, sure. If we find someone experienced and having spare time to do this.
    This can't be easily answered. I have two OMAP5EVM board with hangs. One developed them in the past 6 months without having them before. But it might be completely unrelated (e.g. bad solder joint, whiskers, wear-out by thermal stress etc.). I also tried with a 2 years old u-boot+kernel+rootfs setup but this didn't improve.
    Some boards freeze before the drivers for these components are loaded and powered on by the driver.
     
    rSl, Grench, Swordfish II and 3 others like this.
  12. Djhg2000

    Djhg2000 Very Active Member

    Joined:
    Jul 22, 2014
    Messages:
    151
    Location:
    Sweden
    I didn't see it explicitly mentioned anywhere, but have you tried booting with super conservative timings? As in double (or multiplied by another scalar significantly larger than 1) all of them? Performance will obviously suffer but for debugging purposes it might be useful.

    Since the boards behave themselves in mtest that shouldn't be it, but it could be that the RAM is sensitive to a particular pattern that mtest simply doesn't try. I've seen this before on x86 machines, they can work just fine, even for hours of mixed loads, and then suddenly crash. Then they become rock solid with a different batch of the same RAM or slower timings.

    If that's the culprit then it would explain the randomness. Not sure how we would even handle bad RAM chips, but I think that's a discussion we'll have if and when we get there.
     
    Swordfish II likes this.
  13. crazyhorse

    crazyhorse Member

    Joined:
    Sep 9, 2009
    Messages:
    207
    This can't be easily answered. I have two OMAP5EVM board with hangs. One developed them in the past 6 months without having them before. But it might be completely unrelated (e.g. bad solder joint, whiskers, wear-out by thermal stress etc.). I also tried with a 2 years old u-boot+kernel+rootfs setup but this didn't improve.

    This sounds worrying :( Hope you find out whats happening here
     
  14. CommanderB

    CommanderB Member

    Joined:
    Mar 1, 2014
    Messages:
    87
    Maybe the OMAP5 is for some reasons unstable in general?!
    It is really alarming, that even the official EVMs develop such symptoms and no completely stable configuration could been found after such a long development time

    Gesendet von meinem SM-N9005 mit Tapatalk
     
  15. TrashyMG

    TrashyMG Sarcasm Dispenser Staff Member

    Joined:
    Jan 18, 2010
    Messages:
    10,226
    Mine been running fine for a while on a 4.x kernel of some flavo(u)r released 6 months ago, I've haven't played with it in a while with newer kernels...

    There has been a lot of activity kernel wise with the OMAP5 in the past year or so, likely someone trampled on something.
     
    ClockworkCoder and Swordfish II like this.
  16. EvilDragon

    EvilDragon Administrator Staff Member

    Joined:
    Mar 4, 2003
    Messages:
    19,420
    Location:
    Ingolstadt
    Yes, it definitely seems to depend a lot on the kernel... so something is wrong there.
     
    rSl and crazyhorse like this.
  17. BeMoreViking

    BeMoreViking Still Fresh

    Joined:
    Apr 30, 2016
    Messages:
    24
    Location:
    USA
    Now that's interesting. How far does it get in the boot process? Do you get any text output before it locks up?
     
  18. Djhg2000

    Djhg2000 Very Active Member

    Joined:
    Jul 22, 2014
    Messages:
    151
    Location:
    Sweden
    Oh by the way, have you guys tried the old school debug procedure of putting it in the freezer before testing?
     
  19. Djoga'Ro

    Djoga'Ro moonstruck

    Joined:
    Apr 3, 2016
    Messages:
    672
    For better condensation?
     
  20. BeMoreViking

    BeMoreViking Still Fresh

    Joined:
    Apr 30, 2016
    Messages:
    24
    Location:
    USA
    I'd be curious if we can reliably reproduce this on a dev board. That would probably point us away from pyra-specific issues and more to kernel misconfiguration during initialization (I would think, anyways).

    Out of curiosity, what's the recommended dev board model to use?
     

Share This Page

Loading...