The tale of the cases

[daveshah]

What will it recive? A digital half? Something between 0 and 1? The band between reception and no reception is very small. So why does it work for 2Gb? As an example you mention a problem where you were three to the power of ten away from the design guide? SRSLY?

I'm talking more about power supplies, not the data signals where 1s and 0s come into it - although they are a whole other can of worm - when it comes to high speed interfaces there is a much bigger gap between working all the time and not working at all. As an example, a DDR3 chip might register Vref+160mV as high and Vref-160mV as low. If at the time the clock transitions the input is only Vref+80mV eg because your trace delay is too high, it's anyone guess what happens - it could depend on chip to chip variations, temperature, etc.

Going back to power issues, I reiterate the problem is not the steady state DC voltage that you are talking about, which is what the DC resistance of the trace affects and I have no doubt is in spec, the likely problem is transient drops on the supply line as the devices switches. How the supply will respond to these depends on not just the length of the trace but also the presence and spacing to of a ground plane, decoupling capacitors and their placement and value.

The reason that this only affects 4GB boards is the larger memory devices have more transistors to switch, and therefore cause greater transients on the power supply lines, pushing them out of spec for a tiny period of time during that transient, enough to cause issues inside the chip.

The resistor issue I mentioned was not intended to be an example of how sensitive DDR chips are, just how hard it is to debug them and work out what is software and what is hardware.

Hence even I wouldn't be totally certain about the power issue but it seems pretty likely. Maybe you've done this and I've missed it, but I know you can buy special active scope probes for measuring power rail quality. They are expensive, and you would need access to a suitable good - also expensive, but easier to find someone with - oscilloscope but scrapping boards is also expensive... If you do get stuck it would be interesting to see. I saw some simulations have been done but sometimes measurement is the only way, particularly the exact characteristics of the capacitors and RAM supply demands won't be taken into account in a sim.

 

[Letalis Sonus]

Throwing DC formulas at high frequency signals is absolutely useless - digital or analogue, that doesn't work, you're effectively working with an AC signal that suffers the more from external influences the higher its frequency gets. The effective resistance that causes such a voltage drop is not just caused by the physical properties of the signal line, you have to take all of the electromagnetic influences around it into consideration as well, there's all sorts of virtual electronic components popping up in your calculations due to the effects that a high frequency signal has on nearby conductive elements. It can't be avoided that these effects bleed into the power supply lines as well, a high frequency signal powered by a DC voltage will inevitably cause high frequency noise on the supply line.
If you haven't already seen what's involved to calculate AC circuits, look up some basic formulas. That's some serious nightmare fuel for someone who hasn't studied math or physics.

So why does it work for 2Gb?

Have you ever spent more than a second to think about the idea that a DDR interface is more than a simple address/data bus? The actual structure of the available memory matters a lot, DDR memory is neither accessed atomically nor is it organized in a planar configuration. There's a reason motherboard manufacturers still provide a list of compatible RAM modules nowadays, every single parameter about DDR RAM that changes has a high probability to break things. Compared to things like USB 3 or PCIe lanes a DDR interface is still highly parallelized, the distortions that such a high amount of side by side signal lines cause to each other are enormous.

 

[daveshah]

Thinking about it some more, I saw at one point that turning off the DLL solves the problem. That could point further to a power issue, as it could be that the extra supply demand and hence noise caused by the extra logic of the DLL drives the power rail to the point of unreliability - clearly something is marginal and very much on the border of working. Equally I suppose it could also indicate a timing issue.

I know the people working on this have a lot more experience with this stuff than I do, and it sounds like what's happening so far is sensible. Although more decoupling capacitors aren't always a solution squeezing in more capacitors close to the memory would almost certainly help. Have you already checked if switching to a smaller size of capacitor so you can fit more closer to the memory would cost a lot more to assemble with GC?

 

[EvilDragon]

The same voltage drop that prevent the CPU from running with high clockrate? And where it turned out that it was a wrong setting?

Nope, totally different voltage line (VDDR3).
Besides, the VSYS (power for the OMAP5) was still having a lot of noise and voltage drops, so improving it was mandatory.
Otherwiese, it could still have caused instabilities when more power was actually being used (like NEON, 3D and a high clockrate together).

It's only guaranteed to work if everything is inside specs, and it clearly wasn't.

Voltage drop for pcb traces that are only a few millimeters long is COMPLETE BS! If i read the mailing list, i am now 99% sure that you have no clue what causes this issue.

Yeah, right.
It seems you don't even have the most basic understanding of electronics.

If a wire (or trace) is too thin the for power you want to squeeze through, it will have an increased resistance, which will lead to a voltage drop.
The thinner the wire is, the higher the drop - and that has NOTHING to do with the length of the trace and wire.

Besides, you could even see NO wire as a wire with a diameter of 0 (which is really really thin), and now please don't tell me you don't lose all the voltage if no wire is there.

Here is a nice result of the simulation.
Here it shows we've got too high noise on the VDDR3 plane:



And here is where you can clearly see the nice voltage drops:



But yeah, these professional simulations are all bullshit. They were probably created by us using Microsoft Paint so we have some pretty pictures.

What you do is not testing. It is called "playing around". With a test, you make predictions for the result before you do it. Not afterwards trying to explain and interpret what you have experienced.

What you are explaining is a COMPLETELY different type of tests. These are tests to prove your assumptions.

There are also tests that you make to find out exactly what's going on if there are multiple reasons something can happen. This is what we did.
Of course, you still need to know WHAT to test and how to interpret the results. Which we did.

What we knew:

• It could be a software issue (wrong timings, wrong setup, etc.)
• It could be a hardware issue (noise, voltage drop).
• It could be a general incompatibility between the memory chip and the OMAP5

So, simulation showed the hardware is out of specs. But as before with the OMAP5: Out of specs doesn't necessarily mean that's the problem. Because the specs GUARANTEE that everything will work. Therefore, the manufacturers add A LOT of tolerance to them.
So even if we're out of spec, it could've been that it would still be fine.

Therefore, we also played around with the timings and settings, as this can give more clues.
For example, as the RAM worked fine with higher voltage (with the same timings), it's very likely the voltage drop is indeed an issue.

Why it worked with DLL Off and not with DLL On (which is the default) is a bit trickier to figure out.
The most likely reason is that the too high noise affects the internal clock divider of the DRAM (which is the DLL).

To rule out an issue with the memory, we also tried SingleRank Samsung memory (the 2GB version of the 4GB RAM we're using) and DualRank IM memory (which is the 4GB version of the 2GB RAM we used).

The 2GB Samsung memory works fine, whereas the 4GB IM memory works a bit better than the 4GB Samsung, but still crashes and reports memory errors. So while it's not as sensitive as the Samsung memory, it still doesn't work properly.

All these tests showed that the memory itself is fine and the voltage drop and noise are too far out of specs. So that's what we need to fix.

Besides, we've got support from some memory experts (for example, this company: www.eyeknowhow.de, as well as Juliano who did the simulation for us). And even eyeknowhow could only give some assumptions but said without tests, nothing can be said for sure.
And that's what why we did the tests we could, and it showed us what the issue is.

If the rest of the world takes you as an example for german product development, then we are f*ed.

See? This is the reason why companies usually don't let anyone look into their development process, but rather present the finished product.

Do you think other companies don't have these things to figure out?
Why do you think companies like www.eyeknowhow.de do exist, if they don't have any clients (as apparently, it's not needed to run any tests and everything works without issues from the first try)?

Do you think companies like Nintendo or Sony don't do tons of redesigns, testings, etc. before they publish their final product?
Dream on!

Sure, they've got a LOT more experience and know-how than we do, as they have a lot more experts, a LOT more money and they have already produced TONS of products using memory, so they might have some reference designs already for that.

BTW: Nikolaus (the designer of the hardware) has worked for one of the largest electronic companies in Germany before... and they usually even had more prototype revisions than we have.

Sure, it's easier. Instead of spending the time to test whether it's the hardware, we could've just assumed it is the hardware (as it was the most likely case), produced new CPU boards and test if that would fix it.
But that's also a lot more expensive. No prob for a big company, but certainly not possible for a smaller one like us.

 

[EvilDragon]

The RAM chip works from 1.28V to 1.45V under all specified circumstances. So your voltage drop has to be around 110mV. And this has to be on every motherboard.

It's actually up to 300 - 350mV in worst case, if you take a look at the simulation I posted.
So yep, that's way too much.

Perhaps it was during such a moment, Mr Mrozek, that you ill-advisedly passed comment on a "joke", unwittingly reinforcing a racial stereotype.

I think you understood me wrong. This is not against any greek person. They're always friendly and nice, and the new case company is a good example.
However, I know FormAction was not an exception.

When we had lunch, it was nigh impossible for me to get an invoice. I had to do a lot of discussion, and when I received that invoice, it was Nr. 1, so the first invoice they issued for this day.
They probably calculated most of their prices without tax.
And the government seems to be unable to do something against that... that's a mess.

Additionally, I know that about 50% of the companies at least in the Thessaloniki area closed down during the last years, because of financial misorganization.
I've talked a lot about all these things with my contact, and I know most companies don't really care about bookings, tax or anything.

They just produce, sell (often without any invoice). They can do that because government doesn't really care.
But it doesn't work like that if you sell outside of Germany - as I HAVE to have an invoice for every payment I make.

So yes, sadly, it's not just FormAction. There's a reason these financial issues exist.
And yes, I know it's not the fault of most of the people living there. But it's still real. This has nothing to do with racism.
[doublepost=1508207671,1508207215][/doublepost]

What will it recive? A digital half? Something between 0 and 1? The band between reception and no reception is very small. So why does it work for 2Gb? As an example you mention a problem where you were three to the power of ten away from the design guide? SRSLY?

Actually, yes, with voltages involved, there is a rise and fall, so something between 0 and 1.
And it's actually important there's not too much noise in there, as this would cause issues the faster the timing settings are. That's the signal integrity.

The test for that is called an eye pattern, but we already knew that's not the issue, as that had been simulated before.

 

[Grench]

...
To rule out an issue with the memory, we also tried SingleRank Samsung memory (the 2GB version of the 4GB RAM we're using)
...
The 2GB Samsung memory works fine

If everything else is more or less working on that board with 2GB Samsung RAM, can I call dibs on it for my prototype?

 

[WizardStan]

can I call dibs on it for my prototype?

I'm honestly just a touch surprised you haven't gotten one yet. I know ED's trying to make them as good as possible, but they're prototypes, they're supposed to be questionably stable Otherwise what is even the point?

 

[levi]

I never considered, "Greek" to be racially different from any other predominantly or historically light beige skinned country and no different than English, French, Spanish, Norwegian, Swedish, Finish, Russian, Australian, American of European descent, etc... Does it make me racist if I don't see what you're on about?

The modern usage of the term 'racism' also applies if you're discriminating against a specific tribe, race or nationality, I've heard. Still, I miss the old 'irishman, scot and an englishman walk into a bar' jokes, but perhaps the internet has never really been the place for them.

OT: Thanks ED, for the pretty pictures!

 

[tarator]

Here the link to the video of the simulation of the memory voltage drop extracted from the picture above:

http://esss.com.br/www_cae/Ansoft_Install/video_suporte/pyra_ddr3_vdd/pyra_ddr3_vdd.mp4

 

[Zwerg01]

Interesting, that you exactly know what my profession is. It goes along with all the prejudice about people from Greek.

, why not suggest a solution?

Without a layout, a BOM, all datasheets and the information what is assembled, the only thing i can do is guess. And thats not my style.

Also interesting, that i get response to my posts soon, whereas other users here dont get an answer at all.

Now i am convinced. It has to be the voltage drop. Have fun.

 

[ClockworkCoder]

My entire family had to emigrate/flee from the GDR, why would anyone defend a country of origin that made one do that?
If Greece would be so great, you would hardly have gone away. oO
I fail to see the logic behind your patriotism (or patriotism in general tbh).



Ok, I'll bite. How in the hell can my question offend anyone or be in any way "racist"? Yet again I don't see the logic behind that, like - at all.
I'm actually curious why women are allowed to retire earlier when they even live longer in so many countries, it doesn't make any sense.

Guys, can we please keep this thread on topic? At least for now, while there are relevant and interesting conversations going on. Thanks.

 

[ashcloud]

Here the link to the video of the simulation of the memory voltage drop extracted from the picture above:

http://esss.com.br/www_cae/Ansoft_Install/video_suporte/pyra_ddr3_vdd/pyra_ddr3_vdd.mp4

Very interesting. Also, apart from the voltage drop, I see that there is a very long loop on the power supply line. Aren't you supposed to avoid something like that, to minimize induced currents/noise?
https://en.wikipedia.org/wiki/Ground_loop_(electricity)

 

[TrashyMG]

Only if we had all these arm chair engineers back when the CPU board was designed.

 

[Grench]

I'm honestly just a touch surprised you haven't gotten one yet. I know ED's trying to make them as good as possible, but they're prototypes, they're supposed to be questionably stable Otherwise what is even the point?

I suspect that some of the other prototype preorders in line ahead of me may be more interested in electronically & aesthetically 'perfect' units in unique colors with a 'prototype' pedigree.

My requirements are to have a unit where every piece physically works - i.e. electronically & mechanically sound. There are still a few pieces to the puzzle that I'm still uneasy about.
1. Thermal solution. We've seen a prototype of a heat pipe and a chunk of copper - so that has been worked on somewhat.
2. WiFi. It would be nice if I were able to get decent WiFi without using a USB dongle. I understand that has been worked on and may be pending a new version of the mainboard.
3. Siring capacitor? I want to be able to use my prototype in an office environment. If the device sits there and screams continuously at 18000Hz, not so much. I understand that there were options being looked into, but I'm not sure if the exact component that was singing was identified.

They are all known issues. There just doesn't seem to be enough human power to get those sorted while dealing with the 4GB and case issues.

The project is definitely getting close to where it would nee to be for my prototype. I'm not sure and can't say how close or far it is from acceptable for the other 7 people in line in front of me though.

 

[Splintercat]

So with things looking like a hardware issue, what's the plan?
It sounds like to make 4GB work the cpu boards will need another round of prototypes with some added capacitors.
So what about the ones that have already been produced? If the already produced 4GB boards are stable when limited to 2GB, I personally would be happy with that as my 4G order and would be fine with later paying additionally for a redesigned working 4G board.

What I'm saying is that I get that there's a limited budget and I'm open to options from the Pyra team that allows the project to keep moving.

 

[Silent-Hunter]

So with things looking like a hardware issue, what's the plan?
It sounds like to make 4GB work the cpu boards will need another round of prototypes with some added capacitors.
So what about the ones that have already been produced? If the already produced 4GB boards are stable when limited to 2GB, I personally would be happy with that as my 4G order and would be fine with later paying additionally for a redesigned working 4G board.

What I'm saying is that I get that there's a limited budget and I'm open to options from the Pyra team that allows the project to keep moving.

Count me in for this too.

 

[rygD]

If they get 4GB working, would 8GB work?

JK, I am fine with 2GB when the cases get made.

 

[levi]

So with things looking like a hardware issue, what's the plan?
It sounds like to make 4GB work the cpu boards will need another round of prototypes with some added capacitors.
So what about the ones that have already been produced?

As I'm reading it, it's looking more like it might be down to who's made the chips. AFAIK while PCBs have been made I don't think they've all been populated yet.

It's still not clear to me whether 4GB boards limited to 2GB would be more or less stable. Perhaps if they only use the chip nearer to the power supply, it'll be more stable, but remember AIUI it'll have half the memory bandwidth of genuine 2GB units, as those will have four 512MB chips (while 4GB have 4 1GB chips, with one pair disabled in your plan), giving you double the pin count.

Edit: Fixed; it's four chips of half the relative capacity (two pairs) not two chips. Thanks @Grench!

 

[Grench]

As I'm reading it, it's looking more like it might be down to who's made the chips. AFAIK while PCBs have been made I don't think they've all been populated yet.

It's still not clear to me whether 4GB boards limited to 2GB would be more or less stable. Perhaps if they only use the chip nearer to the power supply, it'll be more stable, but remember AIUI it'll have half the memory bandwidth of genuine 2GB units, as those will have two 1GB chips (while 4GB have 2 2GB chips, with one disabled in your plan), giving you double the pin count.

Just a clarification to what I understand... There are 2 controllers and 4 chip positions on the CPU card. Each controller has 2 chips. Each active controller needs 2 chips Possible configurations are:
(4) 512MB chips = 2GB full bandwidth
(2) 512MB chips = 1GB half bandwidth (one controller off) (This configuration does not exist to my knowledge, but could)
(4) 1GB chips = 4GB full bandwidth
(2) 1GB chips = 2GB half bandwidth (one controller off)

The working 2GB units are (4) 512MB chips.
The 4GB units are (4) 1GB chips.

There is no such animal as Pyra with (2) 2GB chips = 4GB.
There is no such animal as Pyra with (1) ANY chips = failure

Hypothetically a (4) 2GB chips = 8GB version could exist, but I don't think there are any compatible 2GB per chip parts to work from.

I don't think ED and crew have ever said anything about configuring any units to (2) 1GB chips for 2GB units.

 

[daveshah]

If the OMAP5 supports each channel having a different size, you could even create a 3GB Pyra - 2x1GB on one channel and 2x512MB on the other...