Time for some (very very good!) news!
While I've been in Greece, Nikolaus has been busy as well, working on the CPU boards.
There was one nasty issue, that was hard to trace, but which has finally been found this weekend!
Which means: We're now 99,99% sure that the CPU Board is working perfectly fine!
And the mainboard looks great as well, but this will get more extensive testing during the next week.
So, what was that issue?
Well, while the CPU boards were booting, there were some CRC errors when booting from the first MMC slot on the mainboard.
Interestingly, this never happened when using the debug MMC slot which can be soldered directly onto the CPU board.
So... the question to solve was: Why?
Want to know why?
Read the Spoiler!
Well, while it took a while resolving that issue, the good thing is that now all the other MMC buses are included in the device tree as well - that work had to be done anyways.
So, what's next?
Now that the CPU board is up and running, the next step is booting a real Linux system and testing all the remaining hardware components on the mainboard as well.
We don't expect many issues there, as a most of them had already been tested with the debug board and the remaining ones have already appeared on the bus they use, so that should be pretty straightforward.
After all, connecting and using sensors, Wifi modules or other chips to an existing I2C or USB bus is nothing hard (if Linux drivers exist).
I'll keep you informed, but it now put the biggest hurdle behing us!
While I've been in Greece, Nikolaus has been busy as well, working on the CPU boards.
There was one nasty issue, that was hard to trace, but which has finally been found this weekend!
Which means: We're now 99,99% sure that the CPU Board is working perfectly fine!
And the mainboard looks great as well, but this will get more extensive testing during the next week.
So, what was that issue?
Well, while the CPU boards were booting, there were some CRC errors when booting from the first MMC slot on the mainboard.
Interestingly, this never happened when using the debug MMC slot which can be soldered directly onto the CPU board.
So... the question to solve was: Why?
Want to know why?
Read the Spoiler!
Was that a reflection from the bus from that debug MMC slot?
Were the traces to the SD Card slot on the mainboard too long and caused issues?
Putting a finger onto the open pads of the debug MMC slot caused these issues to improve... adding a capacitor there didn't do any fixes (and we couldn't deliver the Pyra with a finger included
)
Also, some PCBs were not impacted by that issue at all.
And to put some sugar on top: It only happened with the first four tries. After that, the errors were gone and everything worked as it should.
Taking all that into account, it sounded like some tolerance issues - something which is annoying to debug... VERY hard and annoying.
So, based on what we knew, we suspected the following:
1. Open pads causing a reflection / receiving some other distortion signals
2. Timing issues in the MMC driver (software setup). Could also be timing issues with the DRAM, as CRC errors could also be the DRAM and not the MMC.
3. Too long traces which cause a bit of a power loss (would also explain why some work and others didn't, as all PCBs have a bit of tolerances).
4. Some other software programming of the power chip (the PALMAS) that causes the voltage to drop.
5. Soldering process issues causing bad contacts (which could explain why the work after a few seconds, as it makes good contacts when the chip is heated up a bit).
That was quite a bit of possible causes, and finding the real one was a VERY time consuming task.
Nikolaus spent a lot of time studying datasheets of the battery charger chip and the PALMAS power management chip. As the EVM doesn't have a battery charger, the setup is different in our case, so some software configuration for the PALMAS could be wrong.
And while he found some things that needed to be changed in order for the charger to work properly together with the PALMAS, it wasn't the cause of the MMC CRC issues.
Well, at least the work was not useless
As we use a different DRAM than the EVM, Nikolaus rechecked the timing settings as well and made a small program to test the memory.
This was time consuming, as every single change in the timing settings needed a recompile of the bootloader and another long run with the memory tester.
After one day, we knew that our DRAM is working fine - but still the MMC issue remained.
Testing with different temperatures (a cooled PCB, etc.) showed us that the temperature doesn't affect it. There were only read errors for the first few tries, then everything worked fine.
So no temperature problem (and therefore no solder process issue, which would've been bad).
So, next up: Figuring out whether the longer traces could be the issue.
A good idea is testing whether the second SD Card slot also has CRC errors, as the trace length is similar.
However, as the EVM only has one SD Card slot, the second one was not yet included in the bootloader.
Another task for Nikolaus: Adding the second and third SD Card slot to the device tree.
Once that was done, testings showed that no other SD Card slot had any CRC issues, so it's unlikely that the trace length itself was the issue.
So... what COULD be the issue?
What's different using the SD Card slot on the mainboard compared to the one on the CPU board?
Finally, Nikolaus found ONE thing that's different:
There's one condensator on the MMC bus which had been chosen for optimum GPS reception.
And yes: After experimenting a bit, Nikolaus found out that it was IN FACT the condesator!
A tiny change: Using 15pf causes CRC errors, using 12pf already fixes everything!
We'll use 10pf (to give it a bit more tolerance), and that's still good enough to not worsen the GPS reception.
That reminds me a bit of the wrong resistor Michael had to replace on the first few hundred Pandora PCBs, that caused Wifi to be horrible!
I hope I explained everything correctly - I might have not understood everything exactly and maybe have forgotten something, but it should be mostly correct
Were the traces to the SD Card slot on the mainboard too long and caused issues?
Putting a finger onto the open pads of the debug MMC slot caused these issues to improve... adding a capacitor there didn't do any fixes (and we couldn't deliver the Pyra with a finger included
)Also, some PCBs were not impacted by that issue at all.
And to put some sugar on top: It only happened with the first four tries. After that, the errors were gone and everything worked as it should.
Taking all that into account, it sounded like some tolerance issues - something which is annoying to debug... VERY hard and annoying.
So, based on what we knew, we suspected the following:
1. Open pads causing a reflection / receiving some other distortion signals
2. Timing issues in the MMC driver (software setup). Could also be timing issues with the DRAM, as CRC errors could also be the DRAM and not the MMC.
3. Too long traces which cause a bit of a power loss (would also explain why some work and others didn't, as all PCBs have a bit of tolerances).
4. Some other software programming of the power chip (the PALMAS) that causes the voltage to drop.
5. Soldering process issues causing bad contacts (which could explain why the work after a few seconds, as it makes good contacts when the chip is heated up a bit).
That was quite a bit of possible causes, and finding the real one was a VERY time consuming task.
Nikolaus spent a lot of time studying datasheets of the battery charger chip and the PALMAS power management chip. As the EVM doesn't have a battery charger, the setup is different in our case, so some software configuration for the PALMAS could be wrong.
And while he found some things that needed to be changed in order for the charger to work properly together with the PALMAS, it wasn't the cause of the MMC CRC issues.
Well, at least the work was not useless
As we use a different DRAM than the EVM, Nikolaus rechecked the timing settings as well and made a small program to test the memory.
This was time consuming, as every single change in the timing settings needed a recompile of the bootloader and another long run with the memory tester.
After one day, we knew that our DRAM is working fine - but still the MMC issue remained.
Testing with different temperatures (a cooled PCB, etc.) showed us that the temperature doesn't affect it. There were only read errors for the first few tries, then everything worked fine.
So no temperature problem (and therefore no solder process issue, which would've been bad).
So, next up: Figuring out whether the longer traces could be the issue.
A good idea is testing whether the second SD Card slot also has CRC errors, as the trace length is similar.
However, as the EVM only has one SD Card slot, the second one was not yet included in the bootloader.
Another task for Nikolaus: Adding the second and third SD Card slot to the device tree.
Once that was done, testings showed that no other SD Card slot had any CRC issues, so it's unlikely that the trace length itself was the issue.
So... what COULD be the issue?
What's different using the SD Card slot on the mainboard compared to the one on the CPU board?
Finally, Nikolaus found ONE thing that's different:
There's one condensator on the MMC bus which had been chosen for optimum GPS reception.
And yes: After experimenting a bit, Nikolaus found out that it was IN FACT the condesator!
A tiny change: Using 15pf causes CRC errors, using 12pf already fixes everything!
We'll use 10pf (to give it a bit more tolerance), and that's still good enough to not worsen the GPS reception.
That reminds me a bit of the wrong resistor Michael had to replace on the first few hundred Pandora PCBs, that caused Wifi to be horrible!
I hope I explained everything correctly - I might have not understood everything exactly and maybe have forgotten something, but it should be mostly correct
Well, while it took a while resolving that issue, the good thing is that now all the other MMC buses are included in the device tree as well - that work had to be done anyways.
So, what's next?
Now that the CPU board is up and running, the next step is booting a real Linux system and testing all the remaining hardware components on the mainboard as well.
We don't expect many issues there, as a most of them had already been tested with the debug board and the remaining ones have already appeared on the bus they use, so that should be pretty straightforward.
After all, connecting and using sensors, Wifi modules or other chips to an existing I2C or USB bus is nothing hard (if Linux drivers exist).
I'll keep you informed, but it now put the biggest hurdle behing us!
