Next huge milestone reached!

Discussion in 'Pyra News' started by EvilDragon, Feb 1, 2016.

  1. EvilDragon

    EvilDragon Administrator Staff Member

    Mar 4, 2003
    Time for some (very very good!) news!

    While I've been in Greece, Nikolaus has been busy as well, working on the CPU boards.

    There was one nasty issue, that was hard to trace, but which has finally been found this weekend!
    Which means: We're now 99,99% sure that the CPU Board is working perfectly fine!
    And the mainboard looks great as well, but this will get more extensive testing during the next week.

    So, what was that issue?

    Well, while the CPU boards were booting, there were some CRC errors when booting from the first MMC slot on the mainboard.

    Interestingly, this never happened when using the debug MMC slot which can be soldered directly onto the CPU board.

    So... the question to solve was: Why?

    Want to know why?
    Read the Spoiler!

    Was that a reflection from the bus from that debug MMC slot?
    Were the traces to the SD Card slot on the mainboard too long and caused issues?

    Putting a finger onto the open pads of the debug MMC slot caused these issues to improve... adding a capacitor there didn't do any fixes (and we couldn't deliver the Pyra with a finger included ;))

    Also, some PCBs were not impacted by that issue at all.
    And to put some sugar on top: It only happened with the first four tries. After that, the errors were gone and everything worked as it should.

    Taking all that into account, it sounded like some tolerance issues - something which is annoying to debug... VERY hard and annoying.

    So, based on what we knew, we suspected the following:

    1. Open pads causing a reflection / receiving some other distortion signals
    2. Timing issues in the MMC driver (software setup). Could also be timing issues with the DRAM, as CRC errors could also be the DRAM and not the MMC.
    3. Too long traces which cause a bit of a power loss (would also explain why some work and others didn't, as all PCBs have a bit of tolerances).
    4. Some other software programming of the power chip (the PALMAS) that causes the voltage to drop.
    5. Soldering process issues causing bad contacts (which could explain why the work after a few seconds, as it makes good contacts when the chip is heated up a bit).

    That was quite a bit of possible causes, and finding the real one was a VERY time consuming task.

    Nikolaus spent a lot of time studying datasheets of the battery charger chip and the PALMAS power management chip. As the EVM doesn't have a battery charger, the setup is different in our case, so some software configuration for the PALMAS could be wrong.
    And while he found some things that needed to be changed in order for the charger to work properly together with the PALMAS, it wasn't the cause of the MMC CRC issues.
    Well, at least the work was not useless :)

    As we use a different DRAM than the EVM, Nikolaus rechecked the timing settings as well and made a small program to test the memory.
    This was time consuming, as every single change in the timing settings needed a recompile of the bootloader and another long run with the memory tester.
    After one day, we knew that our DRAM is working fine - but still the MMC issue remained.

    Testing with different temperatures (a cooled PCB, etc.) showed us that the temperature doesn't affect it. There were only read errors for the first few tries, then everything worked fine.
    So no temperature problem (and therefore no solder process issue, which would've been bad).

    So, next up: Figuring out whether the longer traces could be the issue.
    A good idea is testing whether the second SD Card slot also has CRC errors, as the trace length is similar.
    However, as the EVM only has one SD Card slot, the second one was not yet included in the bootloader.
    Another task for Nikolaus: Adding the second and third SD Card slot to the device tree.

    Once that was done, testings showed that no other SD Card slot had any CRC issues, so it's unlikely that the trace length itself was the issue.

    So... what COULD be the issue?
    What's different using the SD Card slot on the mainboard compared to the one on the CPU board?

    Finally, Nikolaus found ONE thing that's different:
    There's one condensator on the MMC bus which had been chosen for optimum GPS reception.

    And yes: After experimenting a bit, Nikolaus found out that it was IN FACT the condesator!
    A tiny change: Using 15pf causes CRC errors, using 12pf already fixes everything!
    We'll use 10pf (to give it a bit more tolerance), and that's still good enough to not worsen the GPS reception.

    That reminds me a bit of the wrong resistor Michael had to replace on the first few hundred Pandora PCBs, that caused Wifi to be horrible!

    I hope I explained everything correctly - I might have not understood everything exactly and maybe have forgotten something, but it should be mostly correct :)

    Well, while it took a while resolving that issue, the good thing is that now all the other MMC buses are included in the device tree as well - that work had to be done anyways.

    So, what's next?

    Now that the CPU board is up and running, the next step is booting a real Linux system and testing all the remaining hardware components on the mainboard as well.

    We don't expect many issues there, as a most of them had already been tested with the debug board and the remaining ones have already appeared on the bus they use, so that should be pretty straightforward.

    After all, connecting and using sensors, Wifi modules or other chips to an existing I2C or USB bus is nothing hard (if Linux drivers exist).

    I'll keep you informed, but it now put the biggest hurdle behing us!
    microdoge, _wb_, Linux-SWAT and 24 others like this.
  2. ekianjo

    ekianjo Hardcore Member

    May 7, 2012
    神戸市、日本 (Japan)
    Good to know things are progressing fast :) Looking forward to having a Pyra in my hands within this year!
  3. Sebzy

    Sebzy Member

    Oct 18, 2015
    Thanks for the update
    Would be great to see some videos of the final stages of getting the Pyra up and running. The whole development has been fascinating so far and I've even learnt a few things along the way. Thanks
  4. Jumpman

    Jumpman Advanced Member

    May 20, 2006
    Sin City
    My funds are ready, let's GO!

  5. EvilDragon

    EvilDragon Administrator Staff Member

    Mar 4, 2003
    You can be sure that happens once Nikolaus has fixed all boards and sends me a working set :)
    Link likes this.
  6. jdb78

    jdb78 Still Fresh

    Sep 12, 2014
    Great news! I'm so exited. But you changed the capacitor, not a condensator ;-)
  7. TrashyMG

    TrashyMG Sarcasm Dispenser Staff Member

    Jan 18, 2010
    There is no difference between a condenser and a capacitor. Condenser used to be a more traditional term for a Capacitor years ago among English speaking countries. It's still used much in Germany if I'm not mistaken.
    Last edited: Feb 1, 2016
    Raphael likes this.
  8. Zink

    Zink Member

    Mar 8, 2012
    It is "kondensator" in Poland too :)

    Good to know everything is going forward.
  9. rygD

    rygD Nihilistic Mystic

    Feb 28, 2014
    Good call on not including a finger. Since I won't be getting one of the first couple handfuls of Pyras by the time you got around to mine you probably wouldn't have any fingers left to send out (or assemble, package and ship) my Pyra.

    I am glad we have someone willing to put in the hours to figure things out the right way. Thanks, guys, and keep up the awesome work.
    rSl likes this.
  10. bzar

    bzar A Commando

    Sep 22, 2008
    Kondensaattori \o/

    You know what would be fun? A todo-list until launch that is updated each newspost :)
    Majin Wamu, comradekingu and DaMummy like this.
  11. matzesu

    matzesu Hardcore Member

    Jul 22, 2009
    Germany,, Saarland, at home
    This is indeat a huge milestone, 99,99 % working CPU Board sounds great,
    And there allways the tiny parts who cause issues..
    So hopefully,when the greece ship the Case Prototyphe to you, whe will mybe see the first working prototyphes whit all parts..
  12. sepulep

    sepulep Member

    Nov 18, 2008
    have you tested the 10pf cap.? if the issue is some resonant behaviour, its not given that 10 would be better than 12..not that i understand any of it:)
  13. Farox

    Farox Certified Guru

    Jan 8, 2009
    also here in Italy.

    Happy to read some good news about Pyra. I'm egearly waiting for a video too :cool:
  14. AnimatedFreak

    AnimatedFreak Very Active Member

    Oct 29, 2011
    Awesome really happy that the project it moving a long :) hopefully we will see a video of Debian running on a dev board soon
  15. levi

    levi Still fresh, damnit!

    Oct 6, 2008
    Somewhere off the coast of the EU
    Sounds like you are working really well with Nikolaus, to get that level of trial and error reporting. Many engineers like to keep that stuff to themselves, and only report on the successes, but understanding some of the steps needed to resolve the problems helps us all appreciate the successes more.

    FWIW, I'd recommend you translate 'Kondensator' as 'Capacitor' in English, if you can remember. 'Condensator' may be valid historical english usage, but it makes me do a double take as a modern english user. Of course, I know what you mean mind you.
  16. Foxgod

    Foxgod Active Member

    Jan 22, 2011
    Tnx a lot for the good news!
    Speaking of Kondensator, it reminds me of:

  17. Kev2442

    Kev2442 Still French

    Nov 4, 2014
    42 (Loire) is the answer.
    In France, we say "Condensateur".

    Too bad we can't have a Kidnapper Edition Pyra, but we have an almost-fully working prototype !
    ible likes this.
  18. fusion_power

    fusion_power Advanced Member

    Dec 11, 2006
    That's it? Such a small change and such a huge effect? Wow, when I think how many components such an PCB has, it really can be searching for the needle in the hay stack. Good to have people that know where to search for.
  19. hns

    hns Very Active Member

    Jul 16, 2012
    Yes... In fact the change is not huge. It is from 100.00% reliable to 99,95%. Isn't huge, but very significant for digital systems.

    The detailled background is:
    a) MMC/SD cards typically run at 25.000 MHz clock
    b) the 63rd harmonic is therefore at 1575 MHz
    c) this is the frequency where the GPS receiver is looking for very very weak signals
    d) and the wire from a CPU to an SD slot is a short antenna
    e) just some cm away

    So running the SD/MMC with rectangular clock signals disturbs the GPS receiver.
    We know this effect from both, the Openmoko Freerunner (GTA02) and the GTA04.

    The solution is simple: make sure that the signals have weaker harmonics.
    I.e. add a low pass filter that significantly attenuates 1575 MHz but not 25 MHz.

    This is just this capacitor.

    If its value is too big, a nasty effect occurs: the 25 MHz clock becomes a pure
    sine wave. With not the full voltage swing. This deteriorates the timing with the
    data lines.

    So if the capacitor is too big, we get errors on the MMC interface. If it is too
    small, we have too much noise for GPS.

    Unfortunately we did choose 15pF - which is too big for the OMAP5 MMC interface.
    And it was indeed finding the needle in the haystack. But that is the fun if we have it :)
    jdb78, iridepolarbears, _wb_ and 13 others like this.
  20. rSl

    rSl i'm not tired. you're tired.

    Dec 10, 2005
    down with caps, more fingers, please! ;)

    thanks for these detailed reports on
    curing the cpuboard.
    i'm soo glad nikolaus understands this all
    *verry* well and is doing a *really*
    great job, huge THANKS!!!

    for me it sounds like we don't need an
    audio-circuit anymore, as we had these
    sinewaves flowing the traces allready. ;)
    rygD and levi like this.

Share This Page