Tests, tests, and more tests.

[Silent-Hunter]

I've asked that question on the Kernel Mailing List.
Nikolaus told me they did try it several times without success.
No luck ...

What did they try? Sorry, I don't understand.

 

[Askarus]

What did they try? Sorry, I don't understand.

To boot the 4GB boards with only 2GB enabled.

ED stated: "7. 4GB RAM boards were working with only 2GB RAM enabled (as far as I remember - as that was the reason I ordered the RAM modules back then!)"

And I was asking myself why no one is talking about that. Might have been an overlooked solution.

 

[Silent-Hunter]

But you just said Nikolaus said they tried that and it didn't work. Or am I misunderstanding something?

 

[den Tim]

ED stated: "7. 4GB RAM boards were working with only 2GB RAM enabled (as far as I remember - as that was the reason I ordered the RAM modules back then!)"

I read the same as you: with only 2GB RAM enabled the 4GB boards worked as expected.
What I now believe he meant was: the 4GB boards only had 2GB enabled and still had issues the 2GB boards didn't have.

 

[Djoga'Ro]

To my understanding those test, when the bigger RAM chips worked while only half the capacity was enabled, were under different conditions. Different software and maybe different board revision.

 

[EvilDragon]

I read the same as you: with only 2GB RAM enabled the 4GB boards worked as expected.
What I now believe he meant was: the 4GB boards only had 2GB enabled and still had issues the 2GB boards didn't have.

No, they were working withous issues.
I would've never ordered the RAM Chips if I knew they weren't working.
I ordered them because they worked and we only had to enable the 4GB in software.

 

[Nilsnsn]

I've asked that question on the Kernel Mailing List.
Nikolaus told me they did try it several times without success.
No luck ...

I've read Nikolaus answer on the mailing list. Actually he doesn't say it was without success - he just states they tried it several times. No success might be implied, but i don't see it that clear.
Additionally he talks about populating only half the chips, which in his opinion won't work.
https://www.pyra-handheld.com/pipermail/kernel/2018-October/002818.html

 

[BeMoreViking]

Here's a couple other shots in the dark that I'll offer. They may or may not be feasible to test. Or maybe they've already been tested.

1. Is it possible to get some engineering samples of a different memory chip to test out? I'm not suggesting that we'd want to switch to a different chip for production purposes (except as a very last resort), but it might be informative to know if the crashes are related to this particular chip model or not. We've certainly been through the ringer with them.

2. Is it possible to test a stripped-down version of the PCB with all non-essential traces, subsystems, and components removed? Maybe order a couple stripped down PCBs from one of those hobbyist PCB fabrication services for testing?

3. How about testing a super-stripped down kernel? Don't build in drivers for anything that's not absolutely essential (e.g. disable even video/display, use only the video out, or connect to it remotely, etc).

4. Is it worth another audit of relevant kernel drivers for more 'B' vs '8' hex constants?

5. These hangs aren't reproducible with our OS/kernel running on omap development boards right?

6. Are there any physical interference details worth consideration? As a possibly unrelated example, USB3 and 2.4Ghz wifi are known to interfere with each-other. Any chance we have something similar? E.g. two components that interfere with each-other in too-close of proximity?

 

[hns]

Here's a couple other shots in the dark that I'll offer. They may or may not be feasible to test. Or maybe they've already been tested.
1. Is it possible to get some engineering samples of a different memory chip to test out? I'm not suggesting that we'd want to switch to a different chip for production purposes (except as a very last resort), but it might be informative to know if the crashes are related to this particular chip model or not. We've certainly been through the ringer with them.

We did have 4GB chip samples from different manufacturer and tried with the non-optimized board design. AFAIR, we did not see an improvement. It was only one CPU board where the chips were replaced and this was before memory timing improvements were understood. So this is not really comparable.

2. Is it possible to test a stripped-down version of the PCB with all non-essential traces, subsystems, and components removed? Maybe order a couple stripped down PCBs from one of those hobbyist PCB fabrication services for testing?

We still need a 10 layer PCB and that is not available from any hobbyist PCB production. Expect 5000€ to build such a prototype.

3. How about testing a super-stripped down kernel? Don't build in drivers for anything that's not absolutely essential (e.g. disable even video/display, use only the video out, or connect to it remotely, etc).

We already have built-in only those drivers that are essential (or we think are essential) so that you still can boot after

rm -rf /lib/modules

. I am not sure that there was a systematic test yet. But AFAIR there are still freezes with all loadable modules removed.

4. Is it worth another audit of relevant kernel drivers for more 'B' vs '8' hex constants?

Yes, sure. If we find someone experienced and having spare time to do this.

5. These hangs aren't reproducible with our OS/kernel running on omap development boards right?

This can't be easily answered. I have two OMAP5EVM board with hangs. One developed them in the past 6 months without having them before. But it might be completely unrelated (e.g. bad solder joint, whiskers, wear-out by thermal stress etc.). I also tried with a 2 years old u-boot+kernel+rootfs setup but this didn't improve.

6. Are there any physical interference details worth consideration? As a possibly unrelated example, USB3 and 2.4Ghz wifi are known to interfere with each-other. Any chance we have something similar? E.g. two components that interfere with each-other in too-close of proximity?

Some boards freeze before the drivers for these components are loaded and powered on by the driver.

 

[Djhg2000]

I didn't see it explicitly mentioned anywhere, but have you tried booting with super conservative timings? As in double (or multiplied by another scalar significantly larger than 1) all of them? Performance will obviously suffer but for debugging purposes it might be useful.

Since the boards behave themselves in mtest that shouldn't be it, but it could be that the RAM is sensitive to a particular pattern that mtest simply doesn't try. I've seen this before on x86 machines, they can work just fine, even for hours of mixed loads, and then suddenly crash. Then they become rock solid with a different batch of the same RAM or slower timings.

If that's the culprit then it would explain the randomness. Not sure how we would even handle bad RAM chips, but I think that's a discussion we'll have if and when we get there.

 

[crazyhorse]

This can't be easily answered. I have two OMAP5EVM board with hangs. One developed them in the past 6 months without having them before. But it might be completely unrelated (e.g. bad solder joint, whiskers, wear-out by thermal stress etc.). I also tried with a 2 years old u-boot+kernel+rootfs setup but this didn't improve.

This sounds worrying Hope you find out whats happening here

 

[CommanderB]

This can't be easily answered. I have two OMAP5EVM board with hangs. One developed them in the past 6 months without having them before. But it might be completely unrelated (e.g. bad solder joint, whiskers, wear-out by thermal stress etc.). I also tried with a 2 years old u-boot+kernel+rootfs setup but this didn't improve.

This sounds worrying Hope you find out whats happening here

Maybe the OMAP5 is for some reasons unstable in general?!
It is really alarming, that even the official EVMs develop such symptoms and no completely stable configuration could been found after such a long development time

Gesendet von meinem SM-N9005 mit Tapatalk

 

[TrashyMG]

Mine been running fine for a while on a 4.x kernel of some flavo(u)r released 6 months ago, I've haven't played with it in a while with newer kernels...

There has been a lot of activity kernel wise with the OMAP5 in the past year or so, likely someone trampled on something.

 

[EvilDragon]

Yes, it definitely seems to depend a lot on the kernel... so something is wrong there.

 

[BeMoreViking]

This can't be easily answered. I have two OMAP5EVM board with hangs. One developed them in the past 6 months without having them before. But it might be completely unrelated (e.g. bad solder joint, whiskers, wear-out by thermal stress etc.). I also tried with a 2 years old u-boot+kernel+rootfs setup but this didn't improve.

Some boards freeze before the drivers for these components are loaded and powered on by the driver.

Now that's interesting. How far does it get in the boot process? Do you get any text output before it locks up?

 

[Djhg2000]

Oh by the way, have you guys tried the old school debug procedure of putting it in the freezer before testing?

 

[Djoga'Ro]

Oh by the way, have you guys tried the old school debug procedure of putting it in the freezer before testing?

For better condensation?

 

[BeMoreViking]

I'd be curious if we can reliably reproduce this on a dev board. That would probably point us away from pyra-specific issues and more to kernel misconfiguration during initialization (I would think, anyways).

Out of curiosity, what's the recommended dev board model to use?

 

[Grench]

Jumping entirely out of the box here, so this is a bit convoluted and maybe a bit frightening.

***** Thought 1 **********
What other platforms/systems/things use these exact same memory modules?
What do those systems use for a CPU & OS? Is there any chance to see how they are addressing this RAM?

***** Thought 2 **********
If I recall correctly, there are two types of RAM surface mount patterns that the Pyra's SoC board can accept. 4x512 uses one pattern and the 4x1024 uses another pattern. Heat means thermal expansion. Is there any chance whatsoever that one or more of those connections might be a bit tenuous when the board heats up?

***** Thought 3 **********
I don't know if it was established yet - does the 4x512 configuration reliably work? We know that the 4x1024 configuration works until it mysteriously doesn't. But - are we 100% certain that this issue does not affect the 4x512? Are there 4x1024 modules that use the same mount pattern as the 4x512 which could be purchased in test quantities? The idea here is to ensure that the issue isn't with thermal expansion or issues with the mounting patterns.

***** Thought 4 **********
This is being chased as if it is a RAM issue. Is there any chance that this is an OMAP432 issue? I.e. is there a condition set up in the OMAP5 when it sees 4GB of RAM that causes instability? Not that we would have any way to find this.

***** Thought 5 **********
RAM is expensive. SoC modules are expensive. In the event of OMAP5+4GB=fail, why trash either? Stick with me through this chain of thought... If the OMAP5 is stable and working well with 2GB, make 1st generation Pyras with the OMAP5 and 2GB and ship them. Find a different (newer?) SoC that can also take those same 4x1024 modules and begin work on the 2nd generation Pyra SoC boards with <SoC TBD> and 4GB. (This would be a real convoluted long shot and gambles a big pile of EUR.)

 

[Djoga'Ro]

***** Thought 5 **********
RAM is expensive. SoC modules are expensive. In the event of OMAP5+4GB=fail, why trash either? Stick with me through this chain of thought... If the OMAP5 is stable and working well with 2GB, make 1st generation Pyras with the OMAP5 and 2GB and ship them. Find a different (newer?) SoC that can also take those same 4x1024 modules and begin work on the 2nd generation Pyra SoC boards with <SoC TBD> and 4GB. (This would be a real convoluted long shot and gambles a big pile of EUR.)

IIRC, the purchased RAM is EOL. So, maybe no base to build the upgrade from.
I don't know much about the SoC business, but I would imagine, current models rather use DDR4. ?