Am 10.11.2016 um 04:20 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Wed, 9 Nov 2016 21:25:39 +0100 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:
Hi,
Well, the case was being built around the PCB design, so during the design phase, you could've easily added some things there. The PCB could've also been made more wide (there's space between the CPU board and the MicroSD/SIM Card reader).
That space would not have solved the problem. The space must be available around the OMAP and not at some edges of the PCB.
Wouldn't it have been possible to move the RAM away a bit further so we could have used more capacitors?
No they are squeezed between the B2B connectors and those are defined by the main board. And the main board is defined by the position of the Nubs.
Well, won't help now, as that's too much of a redesign anyways.
Well, I guess they the distance of every capacitor to the correct power pin. I guess the capacitors are not connected to multiple power pins, are they?
There are ca. 20 power pins near each other.
Yes, but you don't connect one capacitor to all 20 power pins, or do you?
It is only possible to connect the capacitor to all of them. The capacitor is almost the same dimension as the 20 pins. Well, the area is 3 times of the capacitor. But no chance to connect them in a useful way separately.
If you spend me >10 k€ for a high end tool like OrCAD or PADS and a modern Windows workstation, we can easily simulate this.
And there is no company that offers that service?
They offer it as part of an overall engineering service. It is like self-repairing the car and asking the garage to open one screw. They try everything to convince you that they should do the full repair.
That sounds pretty weird to me. It sounds like a typical service different PCBs designers would use.
Not something specifically bound to a design. More like something like XRay for an analysis. It's not their job to fix it, they only need to run the simulation and send the results.
That's more like an EMV test in my opinion...
The difference is that EMV tests or Xray are needed and useful for almost all projects.
The analyses we are talking about is only needed for really high-speed CPU designs. For example my OMAP3 designs didn't need them. How many developers in the world are doing such designs but don't own their own tool that has this capability? Probably just a handful (like us?) world-wide.
Apple, Samsung, Lenovo, ... they all simply buy the tool instead of asking for such a service.
And most (99%) other designers simply do not do such CPU designs. They buy standard modules which others have designed. And those others have the tools.
50%! This is quite a lot! Is there anything we can to to test that manually? (soldering a wire somewhere?)
Well, on the new board I can try to make another big connection between the Palmas and the capacitor.
Okay... if that has a similar effect. Reading the design guide made me think that every single thing can affect anything...
Yes. And we have so many "single things" which can potentially affect everything...
And yes, while we always need to hurry, we do not need to RUSH things. That's my motto: Hurry but don't rush. So if you told me you need two weeks more time to recheck the power supply circuit, then I would've said "Okay, do that".
Well, you said you urgently need the boards in June when I said I need more time. Without explicitly thinking about such power analysis. Then I submitted the files as they were - noting that there is some risk in it.
The CPU boards have been submitted in August, AFAIR, shortly before GamesCom and your holiday, not in June.
Yes, I wanted to have them in June so we could build units for GamesCom. But as you mentioned you couldn't do that in time, we decided to fix the boards we already had for GamesCom so you could take the time you need to finish the CPU Board design. It's more important to fully check them than to rush them out.
I honestly do not remember you noting that you did say anything about a risk because you didn't have time to recheck the power supply.
I did not specifically mention power supply but recheck everything.
I even remember that I was surprised when notaz found the SMPS3 trace in the schematics, as I assumed you triple-checked the power supply before submitting the CPU board.
I just cross-checked the schematics and removed the additional connection of the Palmas. Nothing else.
A basic principle of review is that you do peer-review in addition to self-review.
Because if you review the same thing triple time yourself you may three times not notice the bug.
Fortunately we are doing it now.
However, I also thought that such things can happen, as the design is more than just complex.
Sadly, I have no real clue about the full hardware design, so I can only trust your decisions. And if you decide that the files are ready to submit, then I hope that all needed things have been done.
Yes, I know there's always a remaining risk, but if you feel uneasy about submitting files, because you'd like to spend a bit more time to check something, feel free to tell me and ask me for a decision.
I feel uneasy that I have to check my work... Because there is the blind spot. You sometimes see your own mistakes, but sometimes you can't.
If you would've said "I haven't had the time to look properly at the full power supply issue yet, I'd like to have 2 more weeks to recheck it" then I would've answered: "Okay, do it!"
And yes, we're in a hurry. But 2 weeks more for checking is better than finding out it doesn't work, THEN needing 2 weeks for checking and afterwards 6 more weeks for another prototype run (with even more costs).
So please, if you ever feel uneasy about that: Let me know and decide whether it's worth the risk and submitting right away or whether some more time should've spent.
It won't help us now, and I don't even know if you felt uneasy (it sounds like it though), but please do remember that for the future.
I think we should discuss this privately.
Fixing this issue is mandatory, that's what I said in February as well.
To really fix it we must know the reason. And what we are discussing currently is still a guess.
I think it's already a bit more than a guess.
Software issue?
Not likely in my opinion, as it works with 1,5GHz and 1 CPU or up to 1GHz with 2 CPUs.
If there was a bug in the kernel switching to 1,5GHz, it would most likely also affect the EVM AND our own CPU boards when running as single-core.
That doesn't mean I assume there is no software solution available. Maybe changing the voltage in the board file can help. But that would be a workaround, to make an unoptimized hardware work.
I still think (like I thought in February) that there's something wrong with the power supply. Most likely the MPU, as the full system itself has no power issues.
And looking at the design guide it looks like the OMAP5 is extremely picky when it comes to the power supply.
Yes, we don't know if that's the cause. But I can't really think of anything else. Everything we tried so far didn't have any effect, and even the TI guy on the forums asked right away whether we're sure our MPU voltages are correct.
I sure hope we can find a fix with some software magic and minimum hardware changes (capacitors) so we can get it to work with our current PCB designs somehow... but we don't know that yet.
Hopefully, the Palmas I2C and OMAP5 JTAG will give us more clues as well.
True, but the more I read about it and think about it,
well, you can't find out by pure thinking :)
True. That's why I suggested trying higher and lower voltages for 1,25GHz as a test and trying different capacitor values (not just bigger ones but also smaller ones) to see if that changes anything.
Maybe we need at least 6 or 7 capacitors (compared to the 14 TI suggests) instead of 4, with lower values.
I can't do much more than thinking though. I have no clue how to change the voltages and compile the board files on order to test it with my SD Card.
And I have no clue which capacitor connects to where and which ones are the ones where I could try replacing them with more or smaller ones. (that's why I asked before :))
Who is assigned to do it?
That doesn't sound like a kernel issue but like a power supply issue to me.
I don't know how cache coherency and clock switching and SMP really work in the OMAP. So there are a lot of potential reasons.
I still don't think this is something where the kernel needs to be changed. It might lead to a workaround for not-optimally created boards, but the main issue would still be the hardware.
And it's especially suspicious that it works with one core.
Doesn't sound too complicated to understand to me, unless I understood it wrong: When you're using a via, the trace from the Power or ground pin should have the same length as the trace from the via (on the other side) to the capacitor pad. So e.g. if the distance from the power pin to the via is 100mil, it should also be 100mil from the via to the capacitor. It should not exceed 2:1 (so 200mil and 100mil, in our example).
Ah ok. That sounds like a reasonable explanation.
... or the way Matthijs explained it. Probably more correct than what I understood :)
Well, it seems to me the amount and size of capacitors DOES matter. In the PDF, 7.3.4 Dynamic PDN Analysis (Page 287) In their example, they use: 5x 100nF, 3x470nF, 4x1µF, 1x2.2µF, 1x22µF So they are suggesting 14 capacitors while you are using 4...
We don't have space for 14... So that is one of the compromises that have to be done because we can't fulfill all recommendations perfectly.
But 4 compared to 14 sounds like a big compromise to me :)
If you sum up the capacity you will find a total of 30.11µF. Not much more than a single 22µF.
What's with the empty area to the right of the OMAP (can be seen on the picture you sent)?
Can't we add some capacitors there?
Electrically yes if they are simply connected in parallel to the existing ones. But that is not the area where VDD-MPU is. So we need VIAs to connect them to VDD-MPU.
But: there is the NUB magnetic sensor chip on the mainboard so that there is not the same height for components as everywhere else.
It is a 3D puzzle...
It might be that we have choosen the wrong one.
Yes, hopefully. That would be my dream fix! Try a different capacitor and have it working out of the box!
So it's not only the total value of the capacitors that matters. It seems it's important to use multiple capacitors and make sure the maximum loop inductance is being kept. And that maximum loop inductance per capacitor...
Yes, that is the basic theory. But all theory doesn't help if there is not enough space around or to connect power supply and signals.
Both the EVM and the IGEPv5 have somehow managed to do it. Do they need that much more space?-
Looking at the EVM, it also has the RAM and MMC very close, just the PALMAS is a bit further away.
The layout is more like a square than a rectangle - so I guess that's what makes the difference for us?
Yes. We did have to move the OMAP to the PCB edge.
But just a couple of additional capacitors should not take a lot of space.
It is more than space to consider.
Just looking at the keyboard area of the Pandora makes me dizzy... MWeston has stacked tons of capacitors there :D
Or the backside of the EVM.
Maybe the capacitors we're using have too high impact on the inductance and that's why higher frequencies fail? >
Well, we have not tried different capacitors.
Which is something we really should do :)
- VDD-MPU and GND resistance between Palmas and OMAP is <0.3 Ohmm
(my Multimeters are not more precise)
Not sure what the maximum allowed value is. :)
Well, we can estimate. If the core draws 5A peak current I would say the 1.25V should not change by more than 0.05V. This means < 10 mOhm.
A rough estimate for standard copper traces on such PCBs is (I hope they did not make them thinner):
1cm wire 100µm wide means 50 mOhm.
So 500µm wide and 1cm long means 10mOhm. We might have 2cm...
But all this is rough guess.
So we might have 20mOhm here? And we need < 10mOhm?
Or did I understand something wrong?
(Yes, I know it's just an estimation)
It is an estimation of what we have, not what we certainly need.
BR, Nikolaus