Next huge milestone reached!


EvilDragon

Administrator
Staff member
Joined
Mar 4, 2003
Messages
29,515
Age
42
Location
Ingolstadt
Time for some (very very good!) news!

While I've been in Greece, Nikolaus has been busy as well, working on the CPU boards.

There was one nasty issue, that was hard to trace, but which has finally been found this weekend!
Which means: We're now 99,99% sure that the CPU Board is working perfectly fine!
And the mainboard looks great as well, but this will get more extensive testing during the next week.

So, what was that issue?

Well, while the CPU boards were booting, there were some CRC errors when booting from the first MMC slot on the mainboard.

Interestingly, this never happened when using the debug MMC slot which can be soldered directly onto the CPU board.

So... the question to solve was: Why?

Want to know why?
Read the Spoiler!

Was that a reflection from the bus from that debug MMC slot?
Were the traces to the SD Card slot on the mainboard too long and caused issues?

Putting a finger onto the open pads of the debug MMC slot caused these issues to improve... adding a capacitor there didn't do any fixes (and we couldn't deliver the Pyra with a finger included ;))

Also, some PCBs were not impacted by that issue at all.
And to put some sugar on top: It only happened with the first four tries. After that, the errors were gone and everything worked as it should.

Taking all that into account, it sounded like some tolerance issues - something which is annoying to debug... VERY hard and annoying.

So, based on what we knew, we suspected the following:

1. Open pads causing a reflection / receiving some other distortion signals
2. Timing issues in the MMC driver (software setup). Could also be timing issues with the DRAM, as CRC errors could also be the DRAM and not the MMC.
3. Too long traces which cause a bit of a power loss (would also explain why some work and others didn't, as all PCBs have a bit of tolerances).
4. Some other software programming of the power chip (the PALMAS) that causes the voltage to drop.
5. Soldering process issues causing bad contacts (which could explain why the work after a few seconds, as it makes good contacts when the chip is heated up a bit).

That was quite a bit of possible causes, and finding the real one was a VERY time consuming task.

Nikolaus spent a lot of time studying datasheets of the battery charger chip and the PALMAS power management chip. As the EVM doesn't have a battery charger, the setup is different in our case, so some software configuration for the PALMAS could be wrong.
And while he found some things that needed to be changed in order for the charger to work properly together with the PALMAS, it wasn't the cause of the MMC CRC issues.
Well, at least the work was not useless :)

As we use a different DRAM than the EVM, Nikolaus rechecked the timing settings as well and made a small program to test the memory.
This was time consuming, as every single change in the timing settings needed a recompile of the bootloader and another long run with the memory tester.
After one day, we knew that our DRAM is working fine - but still the MMC issue remained.

Testing with different temperatures (a cooled PCB, etc.) showed us that the temperature doesn't affect it. There were only read errors for the first few tries, then everything worked fine.
So no temperature problem (and therefore no solder process issue, which would've been bad).

So, next up: Figuring out whether the longer traces could be the issue.
A good idea is testing whether the second SD Card slot also has CRC errors, as the trace length is similar.
However, as the EVM only has one SD Card slot, the second one was not yet included in the bootloader.
Another task for Nikolaus: Adding the second and third SD Card slot to the device tree.

Once that was done, testings showed that no other SD Card slot had any CRC issues, so it's unlikely that the trace length itself was the issue.

So... what COULD be the issue?
What's different using the SD Card slot on the mainboard compared to the one on the CPU board?

Finally, Nikolaus found ONE thing that's different:
There's one condensator on the MMC bus which had been chosen for optimum GPS reception.

And yes: After experimenting a bit, Nikolaus found out that it was IN FACT the condesator!
A tiny change: Using 15pf causes CRC errors, using 12pf already fixes everything!
We'll use 10pf (to give it a bit more tolerance), and that's still good enough to not worsen the GPS reception.

That reminds me a bit of the wrong resistor Michael had to replace on the first few hundred Pandora PCBs, that caused Wifi to be horrible!

I hope I explained everything correctly - I might have not understood everything exactly and maybe have forgotten something, but it should be mostly correct :)

Well, while it took a while resolving that issue, the good thing is that now all the other MMC buses are included in the device tree as well - that work had to be done anyways.

So, what's next?

Now that the CPU board is up and running, the next step is booting a real Linux system and testing all the remaining hardware components on the mainboard as well.

We don't expect many issues there, as a most of them had already been tested with the debug board and the remaining ones have already appeared on the bus they use, so that should be pretty straightforward.

After all, connecting and using sensors, Wifi modules or other chips to an existing I2C or USB bus is nothing hard (if Linux drivers exist).

I'll keep you informed, but it now put the biggest hurdle behing us!
 

Sebzy

Member
Joined
Oct 18, 2015
Messages
34
Thanks for the update
Would be great to see some videos of the final stages of getting the Pyra up and running. The whole development has been fascinating so far and I've even learnt a few things along the way. Thanks
 

EvilDragon

Administrator
Staff member
Joined
Mar 4, 2003
Messages
29,515
Age
42
Location
Ingolstadt
Thanks for the update
Would be great to see some videos of the final stages of getting the Pyra up and running. The whole development has been fascinating so far and I've even learnt a few things along the way. Thanks
You can be sure that happens once Nikolaus has fixed all boards and sends me a working set :)
 

jdb78

Still Fresh
Joined
Jul 30, 2010
Messages
16
Location
Germany
Great news! I'm so exited. But you changed the capacitor, not a condensator ;-)
 

TrashyMG

Sarcasm Dispenser
Staff member
Joined
Jan 18, 2010
Messages
10,736
Great news! I'm so exited. But you changed the capacitor, not a condensator ;-)
There is no difference between a condenser and a capacitor. Condenser used to be a more traditional term for a Capacitor years ago among English speaking countries. It's still used much in Germany if I'm not mistaken.
 
Last edited:

Zink

Member
Joined
Mar 8, 2012
Messages
170
It is "kondensator" in Poland too :)

Good to know everything is going forward.
 

rygD

Nihilistic Mystic
Joined
Feb 28, 2014
Messages
7,457
Location
Everywhere
Good call on not including a finger. Since I won't be getting one of the first couple handfuls of Pyras by the time you got around to mine you probably wouldn't have any fingers left to send out (or assemble, package and ship) my Pyra.

I am glad we have someone willing to put in the hours to figure things out the right way. Thanks, guys, and keep up the awesome work.
 
  • Like
Reactions: rSl

matzesu

Forum Addict!
Staff member
Joined
Oct 24, 2008
Messages
9,774
Age
35
Location
Germany,, Saarland, at home
This is indeat a huge milestone, 99,99 % working CPU Board sounds great,
And there allways the tiny parts who cause issues..
So hopefully,when the greece ship the Case Prototyphe to you, whe will mybe see the first working prototyphes whit all parts..
 

sepulep

Member
Joined
Nov 18, 2008
Messages
362
have you tested the 10pf cap.? if the issue is some resonant behaviour, its not given that 10 would be better than 12..not that i understand any of it:)
 

Farox

Certified Guru
Joined
Jan 8, 2009
Messages
2,142
Age
52
Location
Italy
Website
rbnet.it
Condensatore
also here in Italy.

Happy to read some good news about Pyra. I'm egearly waiting for a video too :cool:
 

AnimatedFreak

Very Active Member
Joined
Oct 29, 2011
Messages
306
Age
26
Awesome really happy that the project it moving a long :) hopefully we will see a video of Debian running on a dev board soon
 

levi

Still fresh, damnit!
Joined
Oct 6, 2008
Messages
12,485
Location
Somewhere off the coast of the EU
Sounds like you are working really well with Nikolaus, to get that level of trial and error reporting. Many engineers like to keep that stuff to themselves, and only report on the successes, but understanding some of the steps needed to resolve the problems helps us all appreciate the successes more.

FWIW, I'd recommend you translate 'Kondensator' as 'Capacitor' in English, if you can remember. 'Condensator' may be valid historical english usage, but it makes me do a double take as a modern english user. Of course, I know what you mean mind you.
 

fusion_power

Advanced Member
Joined
Dec 25, 2005
Messages
13,095
Location
germany
Website
Visit site
A tiny change: Using 15pf causes CRC errors, using 12pf already fixes everything!
That's it? Such a small change and such a huge effect? Wow, when I think how many components such an PCB has, it really can be searching for the needle in the hay stack. Good to have people that know where to search for.
 

hns

Well-Known Member
Joined
Dec 4, 2011
Messages
512
Location
Oberhaching
That's it? Such a small change and such a huge effect? Wow, when I think how many components such an PCB has, it really can be searching for the needle in the hay stack. Good to have people that know where to search for.
Yes... In fact the change is not huge. It is from 100.00% reliable to 99,95%. Isn't huge, but very significant for digital systems.

The detailled background is:
a) MMC/SD cards typically run at 25.000 MHz clock
b) the 63rd harmonic is therefore at 1575 MHz
c) this is the frequency where the GPS receiver is looking for very very weak signals
d) and the wire from a CPU to an SD slot is a short antenna
e) just some cm away

So running the SD/MMC with rectangular clock signals disturbs the GPS receiver.
We know this effect from both, the Openmoko Freerunner (GTA02) and the GTA04.

The solution is simple: make sure that the signals have weaker harmonics.
I.e. add a low pass filter that significantly attenuates 1575 MHz but not 25 MHz.

This is just this capacitor.

If its value is too big, a nasty effect occurs: the 25 MHz clock becomes a pure
sine wave. With not the full voltage swing. This deteriorates the timing with the
data lines.

So if the capacitor is too big, we get errors on the MMC interface. If it is too
small, we have too much noise for GPS.

Unfortunately we did choose 15pF - which is too big for the OMAP5 MMC interface.
And it was indeed finding the needle in the haystack. But that is the fun if we have it :)
 

rSl

i'm not tired. you're tired.
Joined
Nov 19, 2005
Messages
946
Location
homecomputer
down with caps, more fingers, please! ;)

thanks for these detailed reports on
curing the cpuboard.
i'm soo glad nikolaus understands this all
*verry* well and is doing a *really*
great job, huge THANKS!!!

for me it sounds like we don't need an
audio-circuit anymore, as we had these
sinewaves flowing the traces allready. ;)
 
Top