Am Wed, 9 Nov 2016 18:43:16 +0100 hat "H. Nikolaus Schaller"
<hns@goldelico.com> geschrieben:

Hi,

 Hm, in the document Matthijs mentioned, I often fine "must be",
 etc.  
 Yes sometimes. But I see weakening words, e.g.:

   • Whenever possible
  • Minimizing the separation distance
  • as close as possible
    it is recommended to consider
  • must be as wide as possible

 In no such case it is clear when it is "good enough".

Well, yes, the guide first is an overall strategy guide for a design
based around the power. Later it has be more clear examples.

 Dealing with transients is still the job of decoupling caps as
 usual. TI gives quite a bit of guidance on the power distribution
 nets in general (sections 7.2.1-7.2.4, 7.3 of swps051k)    
 I didn't find that one but swps050i.  

 A quick google showed up this one:
 http://software-dl.ti.com/secure/techdocs/swps051k/swps051k?__gda__=1479563482_c6490a4ce11281967f82bf396cc380b5
 You just need to login with your TI account and tell them you don't
 want to nuclear bomb Trump.  
 Says "Access Denied". Maybe they know more about me than I do...

Well, probably because direct linking is not allowed ;)
But as mentioned entering swps051k in google gave me a direct link to
the TI site where you can download the document.
But I've attached it :)

 I had studied that as well as some Palmas Layout guide. But this is
2 years
ago so that I don't remember all details and especially I  
 do not remember the reasons for all decisions.

That's understandable.

 Not sure if there's a difference, but swps051k quite often has
 "must" in their text, not should be or similar.  
 "must be $value" is different from "must be as good as possible".

They have some values later in the guide, AFAIR, and it's mostly "must
be at least" or similar.

So this is something objective we can check.

 And we have quite strong space constraints so that not everything
 can be copied from the EVM (e.g. placing decoupling capacitors on
 the backside under the OMAP5) or guide lines (there isn't much
 space for adding more VIAs or moving traces to optimal
 positions).  
 The backside is completely unpopulated right now, but we have 0,5mm
 space between it and the case.
 Should be enough for capacitors, so why, for example, couldn't we
 put decoupling capacitors there?  
 Because
 1. we didn't know before :)
 2. we might at some time want to make the pyra thinner without
 limitations by the CPU board 3. we need to redesign the whole board

Well, the case was being built around the PCB design, so during
the
design phase, you could've easily added some things there.

The PCB could've also been made more wide (there's space between the
CPU board and the MicroSD/SIM Card reader). But we can't start to
redesign now.

 (This is just an example though, because according to the Datasheet:
 "2. Place Dcaps on “same-side” as component within their power plane
 outline to minimize “decoupling loop inductance”.  
 Target distance to
 power pin should be less than ~500mils depending upon PCB  
 500mil = 12.7mm

Thanks. I didn't find anything about that on the net :D

 This is one of these contradictions where we might have to trade off.
 With a chip dimension of 17x17 mm this more or less means "on the same
 side as the BGA balls and as close as possible to the edge of the
 chip"). It looks as we have that.

Good :)

 How many mils are we away from the power pin?  
 There is no single power pin...

Well, I guess they the distance of every capacitor to the correct power
pin.
I guess the capacitors are not connected to multiple power pins, are
they?

 If I understand correctly, what we need to check for are the IR
 drop and capacitance-inductance loop requirements... I have no idea
 how these can be checked though (I'm not really experienced here).
 Is that something you simulate from the PCB design files?  
 If you spend me >10 k€ for a high end tool like OrCAD or PADS and a
 modern Windows workstation, we can easily simulate this.

And there is no company that offers that service?
For our TV services, we often rent expensive tools (like 100k EUR
measurement devices) if we need them for a job.

For me it's totally normal booking external companies for services I
need... or do you plan to buy a population machine as well so we don't
need GC? ;)

"It is highly recommended to run board frequency domain
analysis on the
vdd_mpu, vdd_mm, and vdd_core power supplies to check if proper board
decoupling is implemented and confirm that the respective target
impedances are met. For more information on target impedance, see ,
External Capacitors." (Page 272)

"highly recommended" sounds like "it won't work otherwise" to me.
Does someone know a company that could check that and give us results?

 And something we should also check:
 "Use minimum 10mil trace width for all voltage and gnd planes
 connections (i.e. Dcap pads, component power pins, etc.)."   
 10 mil = 250 µm

 The minimum width used for all GND and power wires is 0.2 mm - but
 there is additional GND
filling.

Okay, that should be good then.

 I.e. big GND/power-strips like this for VDD-MPU:
 One quite weak thing I see now is that there is only a single VIA to
 connect this big area running in some inner layer to the surface. The
 problem is the space in this area is also heavily used for signal
 wiring:

Ouch... I really do NOT hope this is the issue!

The sentence "Do not share vias among multiple capacitors." on page 274
sounds like that could cause problems.

On that page they also show a nice "Bad design" and "Good design"
example, where the good design simply has more vias added.
They don't even cut any traces, they just add vias.

 Well it could be possible to squeeze in one or two more VIAs in there.
 And we could use one of the 2 layers we had added for getting the
 layout done to add another connection strip between Palmas and OMAP.
 This would reduce resistance definitively to less than 50% of the
 current value.

50%! This is quite a lot!
Is there anything we can to to test that manually? (soldering a wire
somewhere?)

 That such things are not optimal here is the result of finishing the
 CPU design in a hurry last year to get the layout done for the first
 attempt. And a second hurry this year as well. We likely have
 suboptimal trace layout because there was not enough time (and
 manpower) to think about and peer-review such details.
 And to be fair to ourselves - we didn't know
for sure that it could
 be a potential source of a problem.

Well, yes and no.
If you remember, the first thing we suspected after the production when
that happened were weaknessess in the power supply (that was in
February).

I asked you to triple check everything that has to do with the power
supply of the CPU board so we can be as sure as possible, it's not a
hardware issue.

And yes, while we always need to hurry, we do not need to RUSH things.
That's my motto: Hurry but don't rush.
So if you told me you need two weeks more time to recheck the power
supply circuit, then I would've said "Okay, do that".

Just do that next time!

Sadly, I'm not competent enough to do all that, I wouldn't even have
known where I'd have to look for such design guides, otherwise I
would've checked that earlier myself.

 Improvements here would need a change in board design. I am not sure
 if this is within financial and time budgets.

I can't possible produce 3000 Pyras which do not work as advertised.
Especially as it might be possible they don't even work reliable in all
situations with 1GHz.

Fixing this issue is mandatory, that's what I said in February as well.

Hopefully, we can find another workaround in software or without
changing the layout...

 And it could be a solution but as well it could remain unsolved if it
 is not the problem.

True, but the more I read about it and think about it, the more I think
this is the problem. Well, not particularly the design (maybe just the
capacitors), but
something with the power here.

Working with 1 Core at 1,5GHz, working with 2 cores at 1 GHz, but
crashing with 2 cores when going higher?

That doesn't sound like a kernel issue but like a power supply issue to
me. 

 "Use a “Power/Gnd pad/pin to via” ratio of 1:1 whenever possible.
 Do not exceed 2:1 ratio for small number of vias within restricted
 PCB areas (i.e. underneath BGA components)."   
 Maybe someone could help to translate this...

Doesn't sound too complicated to understand to me, unless I understood
it wrong:

When you're using a via, the trace from the Power or ground pin should
have the same length as the trace from the
via (on the other side) to
the capacitor pad.

So e.g. if the distance from the power pin to the via is 100mil, it
should also be 100mil from the via to the capacitor.

It should not exceed 2:1 (so 200mil and 100mil, in our example).

It can also be seen on page 274 (with examples).

 The recommend various decoupling capacitors, so we might have some
 ways to play around here.
 And capacitors can be changed without changing the board design.  
 Yes, indeed.

So this is basically our last straw trying to fix it here... :/

 "For power IC which can support more than 10uF close to processor, a
 bulk capacitor of at least 22uF is strongly recommended for VDD_MPU
 power domains."
 Do we have that? :)  
 I have checked
 1. we did have 3x10µF at the Palmas side (measured as 22µF)
 2. now we have 3x10µF at the Palmas side and 10µF at the OMAP side
 (measured as 34µF)
 Seems to be 20% below. Or my capacitance measurements is imprecise.
 So we have already increased the capacitors by ~50% with no
 significant success @ 1.5GHz.

Well, it seems to me the amount and size of capacitors DOES matter.
In the PDF, 7.3.4 Dynamic PDN Analysis (Page 287)

In their example, they use:
5x 100nF, 3x470nF, 4x1µF, 1x2.2µF, 1x22µF

So they are suggesting 14 capacitors while you are
using 4...

There has to be a difference between using one or more capacitors!

I guess these are some clues (all still page 287):
"Maximum Loop Inductance per Capacitor (nH) - Values do not include ESL
of the capacitor, and the ESL of the capacitor should not exceed 0.5nH"

And "Fpcb (Frequency of Interest) is defined to be a power rail’s max
frequency after which adding a reasonable number of decoupling
capacitors no longer significantly reduces the power rail impedance
below the desired impedance target (Zt2). This is due to the dominance
of the PCB’s parasitic planar spreading and internal package
inductances."

So it's not only the total value of the capacitors that matters. It
seems it's important to use multiple capacitors and make sure the
maximum loop inductance is being kept.
And that maximum loop inductance per capacitor...

Some more clues could be "PDN Impedance
characteristics":
"The PDN (Power Delivery Network) impedance characteristics are defined
versus the device activity (that runs at different frequency)
based on , Recommended Operating Conditions."

This tells us that different frequencies can have an effect of the
impedance, if I understand that correctly.

There's an interesting sentence on page 283:
"– At high frequencies current follows the path of least inductance.
– At low frequencies current follows the path of least resistance."

Could that be our effect, running at more than 1GHz?

There is more explanation on page 276/277:

"Decoupling capacitors must be mounted with minimum impact to
inductance. A real capacitor has characteristics not only of
capacitance but also inductance and resistance. Figure 7-10 shows the
parasitic model of a real capacitor. A real capacitor must be treated
as an RLC circuit with effective series
resistance (ESR) and effective
series inductance (ESL)."

Maybe the capacitors we're using have too high impact on the inductance
and that's why higher frequencies fail?

Maybe our capacitor value is TOO HIGH?
Or we are using too few capacitors.

If we're lucky, we might be able to fix that without extra vias...

Is it possible that we can test increasing the amount of capacitors on
the CPU board with some manual soldering?

At least we should change some more different values / capacitors...

 I have picked up some unpopulated CPU PCBs this morning and I
can
 try to measure resistances. If they are in the mOhm range (which
 is to be expected) the precision might not be good.  
 Okay!  
 Here are the first results:
 * Palmas A33 (SMPS3 feedback) *is* cut open in the new boards

Good!

 * VDD-MPU and GND resistance between Palmas and OMAP is <0.3 Ohm
   (my Multimeters are not more precise)

Not sure what the maximum allowed value is. :)