First, thank you all for your awesome support!
It's great to see you're all here with me
Now, it's time to reply to you guys and also give some more updates.
Can you explain how going to 2GB RAM would incur a loss of 80K Euro. For example, does that mean you had bought 5000 4GB RAM chips at 16Euro each or something?
Sure. We've got 4GB RAM chips for the first 1000 units lying around at GC, as these were parts with a long leadtime (and they're even EOL right now).
That would be around 40k EUR.
Buying the 2GB RAM chips would be roughly 20k EUR. Also, the prototype PCBs with the 4GB RAM boards can't be used, and these have also cost around 20k EUR.
So it adds up.
Of course, if we solve the issue and can use the 4GB RAM chips later, it would only be a temporary loss.
Or we might be able to sell the 4GB RAM chips to someone else.
But it's the current worst-case scenario.
If this problem cannot be solved easily and the money gets tight, maybe we could set up a community donation?
Then you could get the funds to either compensate the 80k loss or to hire a company to get the RAM working.
Unfortunately, I doubt that much would come from a community donation
Because nobody thought this could ever be a problem. Just blame TI for a crappy and not well documented SoC or something like that. If you build an SoC you have to guarantee certain specifications or warn to use XY specs like 4GB RAM. I would do so at least to prevent frustrated customers.
It's not a TI spec. The SoC is well documented, but that doesn't mean it's easy going. The manual is over 5000 pages big...
We know all the long lead times, I doubt ED already has all RAM purchased. So it may be a good idea to cancel some 4GB module orders in advance.
Because OF the long lead times, we have ordered them in advance.
Why do devs still use a 2GB pyra while more than thousand people ordered a 4GB version? There where 2 years in time to equip them with proper hardware...
Devs who are working on the OS (ptitSeb, aTc, Askarus) have a 2GB unit, as it doesn't make much sense to work on anything if the unit freezes regularly.
More lowlevel devs have a 4GB RAM board as well as a 2GB RAM board to compare, and these are also the ones working on it now (mostly Nikolaus and Tony).
You don't find devs with such experience round every corner though.
Also what is the difference with the above findings and the findings
here where the following was noted : "4GB RAM CPU Boards are working! Finally, after a long, long search for the reason, Tony Lindgren found the real issue for 4GB RAM crashes." .... "Nikolaus ran a 4GB Board without heatsink and 1,5GHz and did an apt-get upgrade which installed 400 new packages... without any issues. After a few hours, he shut it down again. Without any crash or overheating!"
Well, we're a bit further here. Tony has run the memtester with both an old 4GB RAM and new 4GB RAM board after finding that one typo.
He used a minimal setup just loading the kernel and doing the RAM Tester. Most of Pyras drivers were NOT loaded.
Nikolaus probably had a board that was working fine - because some seem to run more stable, some less.
More on that down below.
Have there been any hardware / OS revisions between the units tested in the 'Light at the end' news post and those tested recently? Seems to me that they key may lie in this area.
No more hardware revision, but they were probably using different kernels.
Does the fact the latest boards survived the MEM test mean that the timings are OK, i.e. does the mem test use the same Ram timings as the OS?
Yes, the timings are setup in the MLO (bootloader), even before U-Boot.
All boards I have here work fine with U-Boots memtester, so i THEORY all should be good. However, the system isn't doing much when we're running the memtester there, so it could also be noise, etc.
More on that further below.
Were any 2GB units produced using the latest board revision? If yes then do they work OK?
No, as the design itself hasn't been changed, only the layout has been improved (more vias, thicker traces).
It's basically the same as before, just with optimized traces to cause less voltage drop. So these would work.
How easy would it be to get Android or another OS running on a Pyra prototype? If this could be done then would this be useful for stability testing of the RAM or would the expectation be for similar lock ups / crashes?
It would still use the same kernel, and that's probably the issue.
We have different rootfs running on the Pyra, so it's not just that.
Can you tell us a bit more about this, i.e. would the switch to 2GB mean that the boards would have to be revised to use 4 X 512MB or would they use 2 X 1GB (or 4 X 512MB) with the current hardware revision?
The 4GB RAM units have the same amount of chips, the chips are just dual die (which means that it's basically two RAM chips stacked together).
The PCB is exactly the same, you just replace the RAM chips.
It seems to me that you should set a time limit to solve the 4GB issue and if that limit is exceeded then you should reach out to the pre orderers and enquire whether they will be happy with a 2GB version. I bet most would.
Yes, though that will take quite some organizational work to handle! It will be the last resort.
If it could work, you would enjoy half memory bandwidth speeds.
Actually not, as you would still use four RAM chips.
So much for replies, now for some updates regarding the issue.
What we know from tests / experiences:
1. It has nothing to do with temperature.
2. It has nothing to do with CPU stressed or not.
3. RAM works fine using mtest in U-Boot
4. RAM works fine with a simple kernel, stressed CPUs and memtest for Tony - even the old unoptimized 4GB RAM boards!
5. The unit simply freezes - no kernel panic, ooops or anything similar.
6. Sometimes the freeze happens after 10 minutes, sometimes after a few seconds. Regardless of the temperature, heatsink or whatever you do.
7. 4GB RAM boards were working with only 2GB RAM enabled (as far as I remember - as that was the reason I ordered the RAM modules back then!)
8. Some 2GB RAM boards freeze as well, some occassionally, some never. We thought it was an issue with the production run... but maybe not!
9. The Pyra freezes when you try to reboot it. Nikolaus is currently debugging it. It has do to with some driver setup - and that COULD also be the reason for freezes.
So, there are some interesting things.
4. and 5. are especially interesting:
If memory has corruption, it's likely the kernel will sometimes at least report that something is wrong before it freezes completely. This never happens.
This is very weird.
And with Tonys kernel, even a stressed out Pyra can do Memtests for ours without failure.
Then 7.:
Why would they freeze less likely (or never) with only 2GB RAM enabled? The timings are the same - the only difference is the kernel using LPAE.
And (8.) why do some 2GB RAM boards occassionally freeze as well?
It seems while the freeze is related to the RAM somehow, the reason is not directly the RAM.
It could be something in our OMAP5 / peripherals-setup that could trigger the freeze if a certain condition is met.
The condition ALWAYS seems to be met when we reboot the unit (which is good, as that can be reproduced and everything can be tested), and Nikolaus already found what's involved triggering the freeze (some LDO backlight, AFAIR), as he can workaround the freeze for the reboot after all.
He also managed to create a kernel that RELIABLY freezes even on the 2GB RAM boards each time you boot at the exact same time.
So, we have some clues what's involved, but we don't know what it's exactly, it's a bit like looking for the needle in the hay stack.
Tony will receive a JTAG adaptor from us, as he can then read out the state of the OMAP5 as well as the registers which can give valuable clues as to what fails exactly.
So yeah, we're at it - and at the moment, it doesn't seem to be the RAM, but some conflicts in the register / driver / whatever-Setup. This can be purely changed with software though, so everything might be good...
We'll keep you informed!
