Tests, tests, and more tests.


If it could work, you would enjoy half memory bandwidth speeds.

Not only that, but I suspect that whatever the issue is with the 4x1GB = 4GB configuration, it would likely persist in a 2x1GB = 2GB configuration. Same chips, quite possibly the same issues - whatever those issues are.

I'm really curious if there has been a -new from recent batch- board populated with a 4x512MB = 2GB. Different chips, different timings, addresses, etc. vs the 1GB chips. I suspect that with all of the attention being paid to the 4GB issues, there hasn't been much time to run the same series of tests in a 4x512MB = 2GB configuration.

If the 4x512MB = 2GB configuration passes the scenarios where the 4x1GB = 4GB configuration, the 4x512MB = 2GB could then potentially be used as a baseline to help figure out the 4GB issues?

Of course, if a current board revision gets populated with 4x512MB = 2GB also fails, then it points the research in another direction entirely.

But - I'm sure that the team has thought of the above and is far beyond my meager attempts of helping.

For the projects sake, I hope that the 4GB issues turn out to be something simple and easily fixed once found.

For my own Pyra, just make it the beastliest version that works please. I don't much care whether it arrives with 4GB or 2GB of RAM. I do care that the hardware itself works.
 
I'd gladly take a 4GB now and take my chances.

So if a pointer loses its value, something else overwrote the range it is in?

What happens when trying to allocate half of the 4GB?
Maybe the framebuffer/VRAM is addressing space taken by the system?
Set up uboot to boot from USB or TFTP to see if it is to do with the SD-card reader.
Try without the heatsink, or just leave it floating.
Non-bigmem kernel?
Don't load the graphics driver?

If it passes a memtest, maybe try Freedos or something like that to rule out Linux-specific bugs.
 
I was planning to build an external charger myself, but II would prefer to buy one from ed.
If you ever do end up making a external charger, can you please share the process of how you did it?
I'm interested in making one for my self, but have no idea where start. (I have never dealt with charging lithium nor do I want to, 9v are where its ha for personal electronic projects! :D)
 
Can whe please put this „Excternal Charger „ Idea in the Drawer until the 4GB Issue is solved??
There want be a need for a second battery if the Pyra ditnt come to live..

Also: im planing to use my Pyra in a more Intelligent way:
When I know, i have Days until i get my next changes to recharge the Battery, I would dime the Display, Dissable one Core and play maybe something that doesn’t need much power like Gameboy or Atari VCS (2600)
When the Battery Saver Options are fully functional, this should make the Battery Time quite long..
At least I hope so..
 
When I know, i have Days until i get my next changes to recharge the Battery, I would dime the Display, Dissable one Core and play maybe something that doesn’t need much power like Gameboy or Atari VCS (2600)
When the Battery Saver Options are fully functional, this should make the Battery Time quite long..
At least I hope so..

I plan to use mine in an even more intelligent way:
When I know, i have Days until i get my next changes to recharge the Battery, I'll get 1-2 spare batteries.
When one battery will be empty soon I'll stop playing for a minute and insert a new fully charged one.
Using multiple batteries should make the Battery Time quite long..
At least I hope so.
:p :p :p
 
Never needed a spare battery for my Pandora,
But I have a Batterycase for IPhone and Switch and a Big 6000 mäh Battery on my New 3DS XL..
The last time i was for longer time whiteout wallplug i wrote lots of Game of Thrones, and used my Powerbank..
but I want to use my Pyra more than my Pandora yet..
 
Yeah, the Pyra should be finished and out before any accessories are worked on. All the effort should go to the current project.
 
What I propose is this:

We radically alter the case and board layout (both CPU and Main) to facilitate the adoption of a margarita maker and hair dryer. I suspect since both products exist in separate forms it will be trivial to integrate them into the current design.

Maybe 6-8 months?
 
What I propose is this:

We radically alter the case and board layout (both CPU and Main) to facilitate the adoption of a margarita maker and hair dryer. I suspect since both products exist in separate forms it will be trivial to integrate them into the current design.

Maybe 6-8 months?
Please don't forget the all metal casing that rings like a cowbell when struck. Everything is better with more cowbell.
 
ED, if you change specs of shipping units, I will happily cover the cost of my units for the difference plus a little bit extra to cover the overhead of dealing with changes.

As well, I will also happily wait longer for delivery of 4g units when there is a fix.

Lastly, I will be pre-ordering a few more units in the next few months as well.

Just keep on plugging away. You guys are doing great.

Wow.
 
First, thank you all for your awesome support!
It's great to see you're all here with me :)

Now, it's time to reply to you guys and also give some more updates.

Can you explain how going to 2GB RAM would incur a loss of 80K Euro. For example, does that mean you had bought 5000 4GB RAM chips at 16Euro each or something?

Sure. We've got 4GB RAM chips for the first 1000 units lying around at GC, as these were parts with a long leadtime (and they're even EOL right now).
That would be around 40k EUR.

Buying the 2GB RAM chips would be roughly 20k EUR. Also, the prototype PCBs with the 4GB RAM boards can't be used, and these have also cost around 20k EUR.

So it adds up.

Of course, if we solve the issue and can use the 4GB RAM chips later, it would only be a temporary loss.
Or we might be able to sell the 4GB RAM chips to someone else.
But it's the current worst-case scenario.


If this problem cannot be solved easily and the money gets tight, maybe we could set up a community donation?
Then you could get the funds to either compensate the 80k loss or to hire a company to get the RAM working.

Unfortunately, I doubt that much would come from a community donation :)

Because nobody thought this could ever be a problem. Just blame TI for a crappy and not well documented SoC or something like that. If you build an SoC you have to guarantee certain specifications or warn to use XY specs like 4GB RAM. I would do so at least to prevent frustrated customers.

It's not a TI spec. The SoC is well documented, but that doesn't mean it's easy going. The manual is over 5000 pages big... :)

We know all the long lead times, I doubt ED already has all RAM purchased. So it may be a good idea to cancel some 4GB module orders in advance.

Because OF the long lead times, we have ordered them in advance.

Why do devs still use a 2GB pyra while more than thousand people ordered a 4GB version? There where 2 years in time to equip them with proper hardware...

Devs who are working on the OS (ptitSeb, aTc, Askarus) have a 2GB unit, as it doesn't make much sense to work on anything if the unit freezes regularly.
More lowlevel devs have a 4GB RAM board as well as a 2GB RAM board to compare, and these are also the ones working on it now (mostly Nikolaus and Tony).

You don't find devs with such experience round every corner though.

Also what is the difference with the above findings and the findings here where the following was noted : "4GB RAM CPU Boards are working! Finally, after a long, long search for the reason, Tony Lindgren found the real issue for 4GB RAM crashes." .... "Nikolaus ran a 4GB Board without heatsink and 1,5GHz and did an apt-get upgrade which installed 400 new packages... without any issues. After a few hours, he shut it down again. Without any crash or overheating!"

Well, we're a bit further here. Tony has run the memtester with both an old 4GB RAM and new 4GB RAM board after finding that one typo.
He used a minimal setup just loading the kernel and doing the RAM Tester. Most of Pyras drivers were NOT loaded.

Nikolaus probably had a board that was working fine - because some seem to run more stable, some less.
More on that down below.

Have there been any hardware / OS revisions between the units tested in the 'Light at the end' news post and those tested recently? Seems to me that they key may lie in this area.

No more hardware revision, but they were probably using different kernels.

Does the fact the latest boards survived the MEM test mean that the timings are OK, i.e. does the mem test use the same Ram timings as the OS?

Yes, the timings are setup in the MLO (bootloader), even before U-Boot.
All boards I have here work fine with U-Boots memtester, so i THEORY all should be good. However, the system isn't doing much when we're running the memtester there, so it could also be noise, etc.
More on that further below.

Were any 2GB units produced using the latest board revision? If yes then do they work OK?

No, as the design itself hasn't been changed, only the layout has been improved (more vias, thicker traces).
It's basically the same as before, just with optimized traces to cause less voltage drop. So these would work.

How easy would it be to get Android or another OS running on a Pyra prototype? If this could be done then would this be useful for stability testing of the RAM or would the expectation be for similar lock ups / crashes?

It would still use the same kernel, and that's probably the issue.
We have different rootfs running on the Pyra, so it's not just that.

Can you tell us a bit more about this, i.e. would the switch to 2GB mean that the boards would have to be revised to use 4 X 512MB or would they use 2 X 1GB (or 4 X 512MB) with the current hardware revision?

The 4GB RAM units have the same amount of chips, the chips are just dual die (which means that it's basically two RAM chips stacked together).
The PCB is exactly the same, you just replace the RAM chips.

It seems to me that you should set a time limit to solve the 4GB issue and if that limit is exceeded then you should reach out to the pre orderers and enquire whether they will be happy with a 2GB version. I bet most would.

Yes, though that will take quite some organizational work to handle! It will be the last resort.

If it could work, you would enjoy half memory bandwidth speeds.

Actually not, as you would still use four RAM chips.

So much for replies, now for some updates regarding the issue.

What we know from tests / experiences:

1. It has nothing to do with temperature.
2. It has nothing to do with CPU stressed or not.
3. RAM works fine using mtest in U-Boot
4. RAM works fine with a simple kernel, stressed CPUs and memtest for Tony - even the old unoptimized 4GB RAM boards!
5. The unit simply freezes - no kernel panic, ooops or anything similar.
6. Sometimes the freeze happens after 10 minutes, sometimes after a few seconds. Regardless of the temperature, heatsink or whatever you do.
7. 4GB RAM boards were working with only 2GB RAM enabled (as far as I remember - as that was the reason I ordered the RAM modules back then!)
8. Some 2GB RAM boards freeze as well, some occassionally, some never. We thought it was an issue with the production run... but maybe not!
9. The Pyra freezes when you try to reboot it. Nikolaus is currently debugging it. It has do to with some driver setup - and that COULD also be the reason for freezes.

So, there are some interesting things.
4. and 5. are especially interesting:
If memory has corruption, it's likely the kernel will sometimes at least report that something is wrong before it freezes completely. This never happens.
This is very weird.

And with Tonys kernel, even a stressed out Pyra can do Memtests for ours without failure.

Then 7.:
Why would they freeze less likely (or never) with only 2GB RAM enabled? The timings are the same - the only difference is the kernel using LPAE.
And (8.) why do some 2GB RAM boards occassionally freeze as well?

It seems while the freeze is related to the RAM somehow, the reason is not directly the RAM.

It could be something in our OMAP5 / peripherals-setup that could trigger the freeze if a certain condition is met.
The condition ALWAYS seems to be met when we reboot the unit (which is good, as that can be reproduced and everything can be tested), and Nikolaus already found what's involved triggering the freeze (some LDO backlight, AFAIR), as he can workaround the freeze for the reboot after all.
He also managed to create a kernel that RELIABLY freezes even on the 2GB RAM boards each time you boot at the exact same time.

So, we have some clues what's involved, but we don't know what it's exactly, it's a bit like looking for the needle in the hay stack.

Tony will receive a JTAG adaptor from us, as he can then read out the state of the OMAP5 as well as the registers which can give valuable clues as to what fails exactly.

So yeah, we're at it - and at the moment, it doesn't seem to be the RAM, but some conflicts in the register / driver / whatever-Setup. This can be purely changed with software though, so everything might be good...

We'll keep you informed!
 
Back
Top