Am Tue, 8 Nov 2016 22:33:52 +0100 hat Matthijs van Duin matthijsvanduin@gmail.com geschrieben:

Hi Matthijs,

...

...

Right now we think a power supply issue can mostly be ruled out.

That sounds a bit premature. Rather, my conclusion would be that if it is a power supply issue, it is specifically an issue with VDD_MPU.

Hm. VDD_MPU is being handled by the PALMAS automatically, as far as I know. It measures the voltage directly at the OMAP5 and adjusts it, so that should be fine.

Or did I confuse that with something else?

Hm, but maybe it's worth a try increasing the voltage a bit and see if that improves things.

That's something to check them. As far as I know, Nikolaus thoroughly rechecked all traces that have to do with the power supply on the CPU board and made sure they follow exactly TIs specs earlier this year, because my first idea also were power supply issues.

But it might be a good idea to recheck them, in case something has been overlooked. AFAIK Nikolaus added even more capacitors for this CPU Board revision, but I think they were only for VSYS, not between the PALMAS and OMAP5.

I've asked Nikolaus a couple of days ago to make a package of the current schematics, etc., as the one I have do not reflect all the changes made since the last prototype run (AFAIK).

Hopefully, he'll find the time soon and can post the schematics here at the mailing list so we can all check and compare it with TIs recommendations.

On November 9, 2016 5:33:49 AM GMT+01:00, Matthijs van Duin matthijsvanduin@gmail.com wrote:

On 9 November 2016 at 01:33, Michael Mrozek EvilDragon@openpandora.org wrote:

...

Hm. VDD_MPU is being handled by the PALMAS automatically, as far as I

know.

...

It measures the voltage directly at the OMAP5 and adjusts it, so that should be fine.

It's not that simple.

The remote sensing of VDD_MPU allows palmas to compensate mean voltage drop, but it is not going to deal with transients. Even if it could measure and compensate fast enough (it can't), the loop inductance of the power distribution net would probably prevent it from succeeding. (Such an aggressive control loop would also be prone to instability.)

Dealing with transients is still the job of decoupling caps as usual. TI gives quite a bit of guidance on the power distribution nets in general (sections 7.2.1-7.2.4, 7.3 of swps051k) and decoupling caps in particular (e.g. section 7.3.2-7.3.4).

Matthijs

Am Wed, 9 Nov 2016 13:41:24 +0100 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:

Hi,

...

That's something to check them. As far as I know, Nikolaus thoroughly rechecked all traces that have to do with the power supply on the CPU board and made sure they follow exactly TIs specs earlier this year, because my first idea also were power supply issues.

Well, the specs from TI are everything except "exact". So following them exactly is not possible...

Hm, in the document Matthijs mentioned, I often fine "must be", etc.

...

I've asked Nikolaus a couple of days ago to make a package of the current schematics, etc., as the one I have do not reflect all the changes made since the last prototype run (AFAIK).

Yes, I am in the middle of making them drawn a little nicer so that they are easier to read.

Great, thanks!

...

The remote sensing of VDD_MPU allows palmas to compensate mean voltage drop, but it is not going to deal with transients. Even if it could measure and compensate fast enough (it can't), the loop inductance of the power distribution net would probably prevent it from succeeding. (Such an aggressive control loop would also be prone to instability.)

Indeed.

So that's probably where we need to take a closer look at.

...

Dealing with transients is still the job of decoupling caps as usual. TI gives quite a bit of guidance on the power distribution nets in general (sections 7.2.1-7.2.4, 7.3 of swps051k)

I didn't find that one but swps050i.

A quick google showed up this one: http://software-dl.ti.com/secure/techdocs/swps051k/swps051k?__gda__=14795634...

You just need to login with your TI account and tell them you don't want to nuclear bomb Trump.

...

and decoupling caps in particular (e.g. section 7.3.2-7.3.4).

Yes, it is guidance but it is not very helpful in detail. Quite a lot of sentences are "should be" or "as good as possible" or "example". Or "calculate with your design tools". Unfortunately we do not have the big budget we would need to have such tools (e.g. calculate resistance or inductance from PCB layout data).

Not sure if there's a difference, but swps051k quite often has "must" in their text, not should be or similar.

And we have quite strong space constraints so that not everything can be copied from the EVM (e.g. placing decoupling capacitors on the backside under the OMAP5) or guide lines (there isn't much space for adding more VIAs or moving traces to optimal positions).

The backside is completely unpopulated right now, but we have 0,5mm space between it and the case. Should be enough for capacitors, so why, for example, couldn't we put decoupling capacitors there? (This is just an example though, because according to the Datasheet:

"2. Place Dcaps on “same-side” as component within their power plane outline to minimize “decoupling loop inductance”. Target distance to power pin should be less than ~500mils depending upon PCB layout characteristics (plane's layer assignment and solid nature). Use PI modeling CAD tool to verify minimum inductance for top vs bottom-side placement."

So it's better to have the DCaps on the same side of the OMAP5 anyways.

Except for:

"Place Dcaps on “opposite-side” as component within their power plane outline if “same-side” is not feasible or if distance to power pin is greater than ~500mils for top-side location."

How many mils are we away from the power pin?

Looking at 7.2.2 of the document, it also seems VERY important to have multiple vias (not one for multiple capacitors), etc.

If I understand correctly, what we need to check for are the IR drop and capacitance-inductance loop requirements... I have no idea how these can be checked though (I'm not really experienced here). Is that something you simulate from the PCB design files?

And something we should also check: "Use minimum 10mil trace width for all voltage and gnd planes connections (i.e. Dcap pads, component power pins, etc.)."

"Use a “Power/Gnd pad/pin to via” ratio of 1:1 whenever possible. Do not exceed 2:1 ratio for small number of vias within restricted PCB areas (i.e. underneath BGA components)."

So it contains a lot of experience and is a big compromise - but certainly not an optimal one.

I also think we're not that far off! Maybe just a change in the decoupling capacitors could fix things...

"7.3.2 Voltage Decoupling" of the file is interesting here.

The recommend various decoupling capacitors, so we might have some ways to play around here. And capacitors can be changed without changing the board design.

"For power IC which can support more than 10uF close to processor, a bulk capacitor of at least 22uF is strongly recommended for VDD_MPU power domains."

Do we have that? :)

I have picked up some unpopulated CPU PCBs this morning and I can try to measure resistances. If they are in the mOhm range (which is to be expected) the precision might not be good.

Okay!

Am Wed, 9 Nov 2016 18:43:16 +0100 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:

Hi,

...

Hm, in the document Matthijs mentioned, I often fine "must be", etc.

Yes sometimes. But I see weakening words, e.g.:

• Whenever possible • Minimizing the separation distance • as close as possible it is recommended to consider • must be as wide as possible

In no such case it is clear when it is "good enough".

Well, yes, the guide first is an overall strategy guide for a design based around the power. Later it has be more clear examples.

...

...

...

Dealing with transients is still the job of decoupling caps as usual. TI gives quite a bit of guidance on the power distribution nets in general (sections 7.2.1-7.2.4, 7.3 of swps051k)

I didn't find that one but swps050i.

A quick google showed up this one: http://software-dl.ti.com/secure/techdocs/swps051k/swps051k?__gda__=14795634... You just need to login with your TI account and tell them you don't want to nuclear bomb Trump.

Says "Access Denied". Maybe they know more about me than I do...

Well, probably because direct linking is not allowed ;) But as mentioned entering swps051k in google gave me a direct link to the TI site where you can download the document. But I've attached it :)

I had studied that as well as some Palmas Layout guide. But this is

...

2 years ago so that I don't remember all details and especially I

do not remember the reasons for all decisions.

That's understandable.

...

Not sure if there's a difference, but swps051k quite often has "must" in their text, not should be or similar.

"must be $value" is different from "must be as good as possible".

They have some values later in the guide, AFAIR, and it's mostly "must be at least" or similar.

So this is something objective we can check.

...

...

And we have quite strong space constraints so that not everything can be copied from the EVM (e.g. placing decoupling capacitors on the backside under the OMAP5) or guide lines (there isn't much space for adding more VIAs or moving traces to optimal positions).

The backside is completely unpopulated right now, but we have 0,5mm space between it and the case. Should be enough for capacitors, so why, for example, couldn't we put decoupling capacitors there?

Because

1. we didn't know before :)
2. we might at some time want to make the pyra thinner without

limitations by the CPU board 3. we need to redesign the whole board

Well, the case was being built around the PCB design, so during the design phase, you could've easily added some things there.

The PCB could've also been made more wide (there's space between the CPU board and the MicroSD/SIM Card reader). But we can't start to redesign now.

...

(This is just an example though, because according to the Datasheet: "2. Place Dcaps on “same-side” as component within their power plane outline to minimize “decoupling loop inductance”. Target distance to power pin should be less than ~500mils depending upon PCB

500mil = 12.7mm

Thanks. I didn't find anything about that on the net :D

This is one of these contradictions where we might have to trade off. With a chip dimension of 17x17 mm this more or less means "on the same side as the BGA balls and as close as possible to the edge of the chip"). It looks as we have that.

Good :)

...

How many mils are we away from the power pin?

There is no single power pin...

Well, I guess they the distance of every capacitor to the correct power pin. I guess the capacitors are not connected to multiple power pins, are they?

...

If I understand correctly, what we need to check for are the IR drop and capacitance-inductance loop requirements... I have no idea how these can be checked though (I'm not really experienced here). Is that something you simulate from the PCB design files?

If you spend me >10 k€ for a high end tool like OrCAD or PADS and a modern Windows workstation, we can easily simulate this.

And there is no company that offers that service? For our TV services, we often rent expensive tools (like 100k EUR measurement devices) if we need them for a job.

For me it's totally normal booking external companies for services I need... or do you plan to buy a population machine as well so we don't need GC? ;)

"It is highly recommended to run board frequency domain analysis on the vdd_mpu, vdd_mm, and vdd_core power supplies to check if proper board decoupling is implemented and confirm that the respective target impedances are met. For more information on target impedance, see , External Capacitors." (Page 272)

"highly recommended" sounds like "it won't work otherwise" to me. Does someone know a company that could check that and give us results?

...

And something we should also check: "Use minimum 10mil trace width for all voltage and gnd planes connections (i.e. Dcap pads, component power pins, etc.)."

10 mil = 250 µm

The minimum width used for all GND and power wires is 0.2 mm - but there is additional GND filling.

Okay, that should be good then.

I.e. big GND/power-strips like this for VDD-MPU: One quite weak thing I see now is that there is only a single VIA to connect this big area running in some inner layer to the surface. The problem is the space in this area is also heavily used for signal wiring:

Ouch... I really do NOT hope this is the issue!

The sentence "Do not share vias among multiple capacitors." on page 274 sounds like that could cause problems.

On that page they also show a nice "Bad design" and "Good design" example, where the good design simply has more vias added. They don't even cut any traces, they just add vias.

Well it could be possible to squeeze in one or two more VIAs in there. And we could use one of the 2 layers we had added for getting the layout done to add another connection strip between Palmas and OMAP. This would reduce resistance definitively to less than 50% of the current value.

50%! This is quite a lot! Is there anything we can to to test that manually? (soldering a wire somewhere?)

That such things are not optimal here is the result of finishing the CPU design in a hurry last year to get the layout done for the first attempt. And a second hurry this year as well. We likely have suboptimal trace layout because there was not enough time (and manpower) to think about and peer-review such details. And to be fair to ourselves - we didn't know for sure that it could be a potential source of a problem.

Well, yes and no. If you remember, the first thing we suspected after the production when that happened were weaknessess in the power supply (that was in February).

I asked you to triple check everything that has to do with the power supply of the CPU board so we can be as sure as possible, it's not a hardware issue.

And yes, while we always need to hurry, we do not need to RUSH things. That's my motto: Hurry but don't rush. So if you told me you need two weeks more time to recheck the power supply circuit, then I would've said "Okay, do that".

Just do that next time!

Sadly, I'm not competent enough to do all that, I wouldn't even have known where I'd have to look for such design guides, otherwise I would've checked that earlier myself.

Improvements here would need a change in board design. I am not sure if this is within financial and time budgets.

I can't possible produce 3000 Pyras which do not work as advertised. Especially as it might be possible they don't even work reliable in all situations with 1GHz.

Fixing this issue is mandatory, that's what I said in February as well.

Hopefully, we can find another workaround in software or without changing the layout...

And it could be a solution but as well it could remain unsolved if it is not the problem.

True, but the more I read about it and think about it, the more I think this is the problem. Well, not particularly the design (maybe just the capacitors), but something with the power here.

Working with 1 Core at 1,5GHz, working with 2 cores at 1 GHz, but crashing with 2 cores when going higher?

That doesn't sound like a kernel issue but like a power supply issue to me.

...

"Use a “Power/Gnd pad/pin to via” ratio of 1:1 whenever possible. Do not exceed 2:1 ratio for small number of vias within restricted PCB areas (i.e. underneath BGA components)."

Maybe someone could help to translate this...

Doesn't sound too complicated to understand to me, unless I understood it wrong:

When you're using a via, the trace from the Power or ground pin should have the same length as the trace from the via (on the other side) to the capacitor pad.

So e.g. if the distance from the power pin to the via is 100mil, it should also be 100mil from the via to the capacitor.

It should not exceed 2:1 (so 200mil and 100mil, in our example).

It can also be seen on page 274 (with examples).

...

The recommend various decoupling capacitors, so we might have some ways to play around here. And capacitors can be changed without changing the board design.

Yes, indeed.

So this is basically our last straw trying to fix it here... :/

...

"For power IC which can support more than 10uF close to processor, a bulk capacitor of at least 22uF is strongly recommended for VDD_MPU power domains." Do we have that? :)

I have checked

1. we did have 3x10µF at the Palmas side (measured as 22µF)
2. now we have 3x10µF at the Palmas side and 10µF at the OMAP side

(measured as 34µF) Seems to be 20% below. Or my capacitance measurements is imprecise. So we have already increased the capacitors by ~50% with no significant success @ 1.5GHz.

Well, it seems to me the amount and size of capacitors DOES matter. In the PDF, 7.3.4 Dynamic PDN Analysis (Page 287)

In their example, they use: 5x 100nF, 3x470nF, 4x1µF, 1x2.2µF, 1x22µF

So they are suggesting 14 capacitors while you are using 4...

There has to be a difference between using one or more capacitors!

I guess these are some clues (all still page 287): "Maximum Loop Inductance per Capacitor (nH) - Values do not include ESL of the capacitor, and the ESL of the capacitor should not exceed 0.5nH"

And "Fpcb (Frequency of Interest) is defined to be a power rail’s max frequency after which adding a reasonable number of decoupling capacitors no longer significantly reduces the power rail impedance below the desired impedance target (Zt2). This is due to the dominance of the PCB’s parasitic planar spreading and internal package inductances."

So it's not only the total value of the capacitors that matters. It seems it's important to use multiple capacitors and make sure the maximum loop inductance is being kept. And that maximum loop inductance per capacitor...

Some more clues could be "PDN Impedance characteristics": "The PDN (Power Delivery Network) impedance characteristics are defined versus the device activity (that runs at different frequency) based on , Recommended Operating Conditions."

This tells us that different frequencies can have an effect of the impedance, if I understand that correctly.

There's an interesting sentence on page 283: "– At high frequencies current follows the path of least inductance. – At low frequencies current follows the path of least resistance."

Could that be our effect, running at more than 1GHz?

There is more explanation on page 276/277:

"Decoupling capacitors must be mounted with minimum impact to inductance. A real capacitor has characteristics not only of capacitance but also inductance and resistance. Figure 7-10 shows the parasitic model of a real capacitor. A real capacitor must be treated as an RLC circuit with effective series resistance (ESR) and effective series inductance (ESL)."

Maybe the capacitors we're using have too high impact on the inductance and that's why higher frequencies fail?

Maybe our capacitor value is TOO HIGH? Or we are using too few capacitors.

If we're lucky, we might be able to fix that without extra vias...

Is it possible that we can test increasing the amount of capacitors on the CPU board with some manual soldering?

At least we should change some more different values / capacitors...

...

...

I have picked up some unpopulated CPU PCBs this morning and I can try to measure resistances. If they are in the mOhm range (which is to be expected) the precision might not be good.

Okay!

Here are the first results:

• Palmas A33 (SMPS3 feedback) *is* cut open in the new boards

Good!

• VDD-MPU and GND resistance between Palmas and OMAP is <0.3 Ohm (my Multimeters are not more precise)

Not sure what the maximum allowed value is. :)

Hi,

On Wed, Nov 9, 2016 at 9:20 PM, Michael Mrozek EvilDragon@openpandora.org wrote:

Am Wed, 9 Nov 2016 18:43:16 +0100 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:

...

...

Okay!

Here are the first results:

• Palmas A33 (SMPS3 feedback) *is* cut open in the new boards

Good!

For me it's a disappointment as I thought that change could have been missed by the confusion of the board production being stopped.

I've noticed one thing in swps051k: the 8.1.2.3 "Device Naming Convention" chapter states that there can be a part number with "Device Speed" "0" which means "Nominal speed grade (Max MPU OPP = OPP_NOM)" instead of "High speed grade (Max MPU OPP = OPP_HIGH)". OPP_NOM is 1GHz and OPP_HIGH is 1.5GHz, this effectively means some OMAP5s don't support 1.5GHz.

Which part number do we have?

Grazvydas

Am Wed, 9 Nov 2016 21:37:52 +0100 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:

Hi,

...

...

Good!

For me it's a disappointment as I thought that change could have been missed by the confusion of the board production being stopped.

Me too.

I had hoped it, but I didn't expect it... things have never been straightforward or easy with the Pandora or Pyra :(

Hm. Very good question! Here is a photo of the chip we have got in the newest production run:> Maybe we can read the speed from it?

Yes. According to that information it's Revision 2 of the OMAP5432, full speed, standard production run from a tray.

No prototype, even the latest revision and not limited to 1GHz.

BTW, here you can see the battery of decoupling capacitors around the OMAP. There are much less, but you also see that there are signal traces directly near them so adding more of them becomes difficult.

Hmm, from the picture it looks like there's still space on the right side of the OMAP (where there are ground planes). Also, they look like there could be smaller capacitors available as well... so we could add more.

It also looks like we could manually solder a few capacitors together and add them there... so we could do some testing here.

And doing it exactly as recommended (individual VIAs for every capacitor) is impossible. The 3 copper layers below the surface layer are for connecting the DRAM chips and there are also constraints how to route them. So we mix two set of design rules in this area.

Do we need Vias? I thought they would only be needed if the capacitors are on the other side of the PCB. Or do they need VIAs to connect to the ground plane on the other side?

Are there THAT many traces for the DRAM chips so that no vias are possible anywhere there?

The OMAP5430 would have made life much easier in this aspect - but buying RAM chips for it would have been a nightmare.

True...

Am 09.11.2016 um 20:20 schrieb Michael Mrozek EvilDragon@openpandora.org:

Am Wed, 9 Nov 2016 18:43:16 +0100 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:

Hi,

...

...

Hm, in the document Matthijs mentioned, I often fine "must be", etc.

Yes sometimes. But I see weakening words, e.g.:

• Whenever possible • Minimizing the separation distance • as close as possible it is recommended to consider • must be as wide as possible

In no such case it is clear when it is "good enough".

Well, yes, the guide first is an overall strategy guide for a design based around the power. Later it has be more clear examples.

...

...

...

...

Dealing with transients is still the job of decoupling caps as usual. TI gives quite a bit of guidance on the power distribution nets in general (sections 7.2.1-7.2.4, 7.3 of swps051k)

I didn't find that one but swps050i.

A quick google showed up this one: http://software-dl.ti.com/secure/techdocs/swps051k/swps051k?__gda__=14795634... You just need to login with your TI account and tell them you don't want to nuclear bomb Trump.

Says "Access Denied". Maybe they know more about me than I do...

Well, probably because direct linking is not allowed ;) But as mentioned entering swps051k in google gave me a direct link to the TI site where you can download the document. But I've attached it :)

Thanks!

It is the same as the swps050 - except that it is newer and the NDA notice removed.

...

I had studied that as well as some Palmas Layout guide. But this is

...

2 years ago so that I don't remember all details and especially I

do not remember the reasons for all decisions.

That's understandable.

...

...

Not sure if there's a difference, but swps051k quite often has "must" in their text, not should be or similar.

"must be $value" is different from "must be as good as possible".

They have some values later in the guide, AFAIR, and it's mostly "must be at least" or similar.

So this is something objective we can check.

...

...

...

And we have quite strong space constraints so that not everything can be copied from the EVM (e.g. placing decoupling capacitors on the backside under the OMAP5) or guide lines (there isn't much space for adding more VIAs or moving traces to optimal positions).

The backside is completely unpopulated right now, but we have 0,5mm space between it and the case. Should be enough for capacitors, so why, for example, couldn't we put decoupling capacitors there?

Because

1. we didn't know before :)
2. we might at some time want to make the pyra thinner without

limitations by the CPU board 3. we need to redesign the whole board

Well, the case was being built around the PCB design, so during the design phase, you could've easily added some things there.

The PCB could've also been made more wide (there's space between the CPU board and the MicroSD/SIM Card reader).

That space would not have solved the problem. The space must be available around the OMAP and not at some edges of the PCB.

But we can't start to redesign now.

...

...

(This is just an example though, because according to the Datasheet: "2. Place Dcaps on “same-side” as component within their power plane outline to minimize “decoupling loop inductance”. Target distance to power pin should be less than ~500mils depending upon PCB

500mil = 12.7mm

Thanks. I didn't find anything about that on the net :D

...

This is one of these contradictions where we might have to trade off. With a chip dimension of 17x17 mm this more or less means "on the same side as the BGA balls and as close as possible to the edge of the chip"). It looks as we have that.

Good :)

...

...

How many mils are we away from the power pin?

There is no single power pin...

Well, I guess they the distance of every capacitor to the correct power pin. I guess the capacitors are not connected to multiple power pins, are they?

There are ca. 20 power pins near each other.

...

...

If I understand correctly, what we need to check for are the IR drop and capacitance-inductance loop requirements... I have no idea how these can be checked though (I'm not really experienced here). Is that something you simulate from the PCB design files?

If you spend me >10 k€ for a high end tool like OrCAD or PADS and a modern Windows workstation, we can easily simulate this.

And there is no company that offers that service?

They offer it as part of an overall engineering service.

It is like self-repairing the car and asking the garage to open one screw. They try everything to convince you that they should do the full repair.

For our TV services, we often rent expensive tools (like 100k EUR measurement devices) if we need them for a job.

For me it's totally normal booking external companies for services I need... or do you plan to buy a population machine as well so we don't need GC? ;)

"It is highly recommended to run board frequency domain analysis on the vdd_mpu, vdd_mm, and vdd_core power supplies to check if proper board decoupling is implemented and confirm that the respective target impedances are met. For more information on target impedance, see , External Capacitors." (Page 272)

"highly recommended" sounds like "it won't work otherwise" to me. Does someone know a company that could check that and give us results?

...

...

And something we should also check: "Use minimum 10mil trace width for all voltage and gnd planes connections (i.e. Dcap pads, component power pins, etc.)."

10 mil = 250 µm

The minimum width used for all GND and power wires is 0.2 mm - but there is additional GND filling.

Okay, that should be good then.

...

I.e. big GND/power-strips like this for VDD-MPU: One quite weak thing I see now is that there is only a single VIA to connect this big area running in some inner layer to the surface. The problem is the space in this area is also heavily used for signal wiring:

Ouch... I really do NOT hope this is the issue!

The sentence "Do not share vias among multiple capacitors." on page 274 sounds like that could cause problems.

On that page they also show a nice "Bad design" and "Good design" example, where the good design simply has more vias added. They don't even cut any traces, they just add vias.

...

Well it could be possible to squeeze in one or two more VIAs in there. And we could use one of the 2 layers we had added for getting the layout done to add another connection strip between Palmas and OMAP. This would reduce resistance definitively to less than 50% of the current value.

50%! This is quite a lot! Is there anything we can to to test that manually? (soldering a wire somewhere?)

Well, on the new board I can try to make another big connection between the Palmas and the capacitor.

...

That such things are not optimal here is the result of finishing the CPU design in a hurry last year to get the layout done for the first attempt. And a second hurry this year as well. We likely have suboptimal trace layout because there was not enough time (and manpower) to think about and peer-review such details. And to be fair to ourselves - we didn't know for sure that it could be a potential source of a problem.

Well, yes and no. If you remember, the first thing we suspected after the production when that happened were weaknessess in the power supply (that was in February).

I asked you to triple check everything that has to do with the power supply of the CPU board so we can be as sure as possible, it's not a hardware issue.

And yes, while we always need to hurry, we do not need to RUSH things. That's my motto: Hurry but don't rush. So if you told me you need two weeks more time to recheck the power supply circuit, then I would've said "Okay, do that".

Well, you said you urgently need the boards in June when I said I need more time. Without explicitly thinking about such power analysis. Then I submitted the files as they were - noting that there is some risk in it.

Just do that next time!

Sadly, I'm not competent enough to do all that, I wouldn't even have known where I'd have to look for such design guides, otherwise I would've checked that earlier myself.

...

Improvements here would need a change in board design. I am not sure if this is within financial and time budgets.

I can't possible produce 3000 Pyras which do not work as advertised. Especially as it might be possible they don't even work reliable in all situations with 1GHz.

Fixing this issue is mandatory, that's what I said in February as well.

To really fix it we must know the reason. And what we are discussing currently is still a guess.

Hopefully, we can find another workaround in software or without changing the layout...

...

And it could be a solution but as well it could remain unsolved if it is not the problem.

True, but the more I read about it and think about it,

well, you can't find out by pure thinking :)

the more I think this is the problem. Well, not particularly the design (maybe just the capacitors), but something with the power here.

Working with 1 Core at 1,5GHz, working with 2 cores at 1 GHz, but crashing with 2 cores when going higher?

That doesn't sound like a kernel issue but like a power supply issue to me.

I don't know how cache coherency and clock switching and SMP really work in the OMAP. So there are a lot of potential reasons.

...

...

"Use a “Power/Gnd pad/pin to via” ratio of 1:1 whenever possible. Do not exceed 2:1 ratio for small number of vias within restricted PCB areas (i.e. underneath BGA components)."

Maybe someone could help to translate this...

Doesn't sound too complicated to understand to me, unless I understood it wrong:

When you're using a via, the trace from the Power or ground pin should have the same length as the trace from the via (on the other side) to the capacitor pad.

So e.g. if the distance from the power pin to the via is 100mil, it should also be 100mil from the via to the capacitor.

It should not exceed 2:1 (so 200mil and 100mil, in our example).

Ah ok. That sounds like a reasonable explanation.

It can also be seen on page 274 (with examples).

...

...

The recommend various decoupling capacitors, so we might have some ways to play around here. And capacitors can be changed without changing the board design.

Yes, indeed.

So this is basically our last straw trying to fix it here... :/

...

...

"For power IC which can support more than 10uF close to processor, a bulk capacitor of at least 22uF is strongly recommended for VDD_MPU power domains." Do we have that? :)

I have checked

1. we did have 3x10µF at the Palmas side (measured as 22µF)
2. now we have 3x10µF at the Palmas side and 10µF at the OMAP side

(measured as 34µF) Seems to be 20% below. Or my capacitance measurements is imprecise. So we have already increased the capacitors by ~50% with no significant success @ 1.5GHz.

Well, it seems to me the amount and size of capacitors DOES matter. In the PDF, 7.3.4 Dynamic PDN Analysis (Page 287)

In their example, they use: 5x 100nF, 3x470nF, 4x1µF, 1x2.2µF, 1x22µF

So they are suggesting 14 capacitors while you are using 4...

We don't have space for 14... So that is one of the compromises that have to be done because we can't fulfill all recommendations perfectly.

It might be that we have choosen the wrong one.

There has to be a difference between using one or more capacitors!

I guess these are some clues (all still page 287): "Maximum Loop Inductance per Capacitor (nH) - Values do not include ESL of the capacitor, and the ESL of the capacitor should not exceed 0.5nH"

And "Fpcb (Frequency of Interest) is defined to be a power rail’s max frequency after which adding a reasonable number of decoupling capacitors no longer significantly reduces the power rail impedance below the desired impedance target (Zt2). This is due to the dominance of the PCB’s parasitic planar spreading and internal package inductances."

So it's not only the total value of the capacitors that matters. It seems it's important to use multiple capacitors and make sure the maximum loop inductance is being kept. And that maximum loop inductance per capacitor...

Yes, that is the basic theory. But all theory doesn't help if there is not enough space around or to connect power supply and signals.

Some more clues could be "PDN Impedance characteristics": "The PDN (Power Delivery Network) impedance characteristics are defined versus the device activity (that runs at different frequency) based on , Recommended Operating Conditions."

This tells us that different frequencies can have an effect of the impedance, if I understand that correctly.

There's an interesting sentence on page 283: "– At high frequencies current follows the path of least inductance. – At low frequencies current follows the path of least resistance."

Could that be our effect, running at more than 1GHz?

There is more explanation on page 276/277:

"Decoupling capacitors must be mounted with minimum impact to inductance. A real capacitor has characteristics not only of capacitance but also inductance and resistance. Figure 7-10 shows the parasitic model of a real capacitor. A real capacitor must be treated as an RLC circuit with effective series resistance (ESR) and effective series inductance (ESL)."

Maybe the capacitors we're using have too high impact on the inductance and that's why higher frequencies fail?

Well, we have not tried different capacitors.

Maybe our capacitor value is TOO HIGH? Or we are using too few capacitors.

If we're lucky, we might be able to fix that without extra vias...

Is it possible that we can test increasing the amount of capacitors on the CPU board with some manual soldering?

At least we should change some more different values / capacitors...

...

...

...

I have picked up some unpopulated CPU PCBs this morning and I can try to measure resistances. If they are in the mOhm range (which is to be expected) the precision might not be good.

Okay!

Here are the first results:

• Palmas A33 (SMPS3 feedback) *is* cut open in the new boards

Good!

...

• VDD-MPU and GND resistance between Palmas and OMAP is <0.3 Ohm

(my Multimeters are not more precise)

Not sure what the maximum allowed value is. :)

Well, we can estimate. If the core draws 5A peak current I would say the 1.25V should not change by more than 0.05V. This means < 10 mOhm.

A rough estimate for standard copper traces on such PCBs is (I hope they did not make them thinner):

1cm wire 100µm wide means 50 mOhm.

So 500µm wide and 1cm long means 10mOhm. We might have 2cm...

But all this is rough guess.

BR, Nikolaus

Am 10.11.2016 um 04:20 schrieb Michael Mrozek EvilDragon@openpandora.org:

Am Wed, 9 Nov 2016 21:25:39 +0100 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:

Hi,

...

...

Well, the case was being built around the PCB design, so during the design phase, you could've easily added some things there. The PCB could've also been made more wide (there's space between the CPU board and the MicroSD/SIM Card reader).

That space would not have solved the problem. The space must be available around the OMAP and not at some edges of the PCB.

Wouldn't it have been possible to move the RAM away a bit further so we could have used more capacitors?

No they are squeezed between the B2B connectors and those are defined by the main board. And the main board is defined by the position of the Nubs.

Well, won't help now, as that's too much of a redesign anyways.

...

...

Well, I guess they the distance of every capacitor to the correct power pin. I guess the capacitors are not connected to multiple power pins, are they?

There are ca. 20 power pins near each other.

Yes, but you don't connect one capacitor to all 20 power pins, or do you?

It is only possible to connect the capacitor to all of them. The capacitor is almost the same dimension as the 20 pins. Well, the area is 3 times of the capacitor. But no chance to connect them in a useful way separately.

...

...

...

If you spend me >10 k€ for a high end tool like OrCAD or PADS and a modern Windows workstation, we can easily simulate this.

And there is no company that offers that service?

They offer it as part of an overall engineering service. It is like self-repairing the car and asking the garage to open one screw. They try everything to convince you that they should do the full repair.

That sounds pretty weird to me. It sounds like a typical service different PCBs designers would use.

Not something specifically bound to a design. More like something like XRay for an analysis. It's not their job to fix it, they only need to run the simulation and send the results.

That's more like an EMV test in my opinion...

The difference is that EMV tests or Xray are needed and useful for almost all projects.

The analyses we are talking about is only needed for really high-speed CPU designs. For example my OMAP3 designs didn't need them. How many developers in the world are doing such designs but don't own their own tool that has this capability? Probably just a handful (like us?) world-wide.

Apple, Samsung, Lenovo, ... they all simply buy the tool instead of asking for such a service.

And most (99%) other designers simply do not do such CPU designs. They buy standard modules which others have designed. And those others have the tools.

...

...

50%! This is quite a lot! Is there anything we can to to test that manually? (soldering a wire somewhere?)

Well, on the new board I can try to make another big connection between the Palmas and the capacitor.

Okay... if that has a similar effect. Reading the design guide made me think that every single thing can affect anything...

Yes. And we have so many "single things" which can potentially affect everything...

...

...

And yes, while we always need to hurry, we do not need to RUSH things. That's my motto: Hurry but don't rush. So if you told me you need two weeks more time to recheck the power supply circuit, then I would've said "Okay, do that".

Well, you said you urgently need the boards in June when I said I need more time. Without explicitly thinking about such power analysis. Then I submitted the files as they were - noting that there is some risk in it.

The CPU boards have been submitted in August, AFAIR, shortly before GamesCom and your holiday, not in June.

Yes, I wanted to have them in June so we could build units for GamesCom. But as you mentioned you couldn't do that in time, we decided to fix the boards we already had for GamesCom so you could take the time you need to finish the CPU Board design. It's more important to fully check them than to rush them out.

I honestly do not remember you noting that you did say anything about a risk because you didn't have time to recheck the power supply.

I did not specifically mention power supply but recheck everything.

I even remember that I was surprised when notaz found the SMPS3 trace in the schematics, as I assumed you triple-checked the power supply before submitting the CPU board.

I just cross-checked the schematics and removed the additional connection of the Palmas. Nothing else.

A basic principle of review is that you do peer-review in addition to self-review.

Because if you review the same thing triple time yourself you may three times not notice the bug.

Fortunately we are doing it now.

However, I also thought that such things can happen, as the design is more than just complex.

Sadly, I have no real clue about the full hardware design, so I can only trust your decisions. And if you decide that the files are ready to submit, then I hope that all needed things have been done.

Yes, I know there's always a remaining risk, but if you feel uneasy about submitting files, because you'd like to spend a bit more time to check something, feel free to tell me and ask me for a decision.

I feel uneasy that I have to check my work... Because there is the blind spot. You sometimes see your own mistakes, but sometimes you can't.

If you would've said "I haven't had the time to look properly at the full power supply issue yet, I'd like to have 2 more weeks to recheck it" then I would've answered: "Okay, do it!"

And yes, we're in a hurry. But 2 weeks more for checking is better than finding out it doesn't work, THEN needing 2 weeks for checking and afterwards 6 more weeks for another prototype run (with even more costs).

So please, if you ever feel uneasy about that: Let me know and decide whether it's worth the risk and submitting right away or whether some more time should've spent.

It won't help us now, and I don't even know if you felt uneasy (it sounds like it though), but please do remember that for the future.

I think we should discuss this privately.

...

...

Fixing this issue is mandatory, that's what I said in February as well.

To really fix it we must know the reason. And what we are discussing currently is still a guess.

I think it's already a bit more than a guess.

Software issue?

Not likely in my opinion, as it works with 1,5GHz and 1 CPU or up to 1GHz with 2 CPUs.

If there was a bug in the kernel switching to 1,5GHz, it would most likely also affect the EVM AND our own CPU boards when running as single-core.

That doesn't mean I assume there is no software solution available. Maybe changing the voltage in the board file can help. But that would be a workaround, to make an unoptimized hardware work.

I still think (like I thought in February) that there's something wrong with the power supply. Most likely the MPU, as the full system itself has no power issues.

And looking at the design guide it looks like the OMAP5 is extremely picky when it comes to the power supply.

Yes, we don't know if that's the cause. But I can't really think of anything else. Everything we tried so far didn't have any effect, and even the TI guy on the forums asked right away whether we're sure our MPU voltages are correct.

I sure hope we can find a fix with some software magic and minimum hardware changes (capacitors) so we can get it to work with our current PCB designs somehow... but we don't know that yet.

Hopefully, the Palmas I2C and OMAP5 JTAG will give us more clues as well.

...

...

True, but the more I read about it and think about it,

well, you can't find out by pure thinking :)

True. That's why I suggested trying higher and lower voltages for 1,25GHz as a test and trying different capacitor values (not just bigger ones but also smaller ones) to see if that changes anything.

Maybe we need at least 6 or 7 capacitors (compared to the 14 TI suggests) instead of 4, with lower values.

I can't do much more than thinking though. I have no clue how to change the voltages and compile the board files on order to test it with my SD Card.

And I have no clue which capacitor connects to where and which ones are the ones where I could try replacing them with more or smaller ones. (that's why I asked before :))

Who is assigned to do it?

...

...

That doesn't sound like a kernel issue but like a power supply issue to me.

I don't know how cache coherency and clock switching and SMP really work in the OMAP. So there are a lot of potential reasons.

I still don't think this is something where the kernel needs to be changed. It might lead to a workaround for not-optimally created boards, but the main issue would still be the hardware.

And it's especially suspicious that it works with one core.

...

...

Doesn't sound too complicated to understand to me, unless I understood it wrong: When you're using a via, the trace from the Power or ground pin should have the same length as the trace from the via (on the other side) to the capacitor pad. So e.g. if the distance from the power pin to the via is 100mil, it should also be 100mil from the via to the capacitor. It should not exceed 2:1 (so 200mil and 100mil, in our example).

Ah ok. That sounds like a reasonable explanation.

... or the way Matthijs explained it. Probably more correct than what I understood :)

...

...

Well, it seems to me the amount and size of capacitors DOES matter. In the PDF, 7.3.4 Dynamic PDN Analysis (Page 287) In their example, they use: 5x 100nF, 3x470nF, 4x1µF, 1x2.2µF, 1x22µF So they are suggesting 14 capacitors while you are using 4...

We don't have space for 14... So that is one of the compromises that have to be done because we can't fulfill all recommendations perfectly.

But 4 compared to 14 sounds like a big compromise to me :)

If you sum up the capacity you will find a total of 30.11µF. Not much more than a single 22µF.

What's with the empty area to the right of the OMAP (can be seen on the picture you sent)?

Can't we add some capacitors there?

Electrically yes if they are simply connected in parallel to the existing ones. But that is not the area where VDD-MPU is. So we need VIAs to connect them to VDD-MPU.

But: there is the NUB magnetic sensor chip on the mainboard so that there is not the same height for components as everywhere else.

It is a 3D puzzle...

...

It might be that we have choosen the wrong one.

Yes, hopefully. That would be my dream fix! Try a different capacitor and have it working out of the box!

...

...

So it's not only the total value of the capacitors that matters. It seems it's important to use multiple capacitors and make sure the maximum loop inductance is being kept. And that maximum loop inductance per capacitor...

Yes, that is the basic theory. But all theory doesn't help if there is not enough space around or to connect power supply and signals.

Both the EVM and the IGEPv5 have somehow managed to do it. Do they need that much more space?-

Looking at the EVM, it also has the RAM and MMC very close, just the PALMAS is a bit further away.

The layout is more like a square than a rectangle - so I guess that's what makes the difference for us?

Yes. We did have to move the OMAP to the PCB edge.

But just a couple of additional capacitors should not take a lot of space.

It is more than space to consider.

Just looking at the keyboard area of the Pandora makes me dizzy... MWeston has stacked tons of capacitors there :D

Or the backside of the EVM.

...

...

Maybe the capacitors we're using have too high impact on the inductance and that's why higher frequencies fail? >

Well, we have not tried different capacitors.

Which is something we really should do :)

...

...

...

• VDD-MPU and GND resistance between Palmas and OMAP is <0.3 Ohmm

(my Multimeters are not more precise)

Not sure what the maximum allowed value is. :)

Well, we can estimate. If the core draws 5A peak current I would say the 1.25V should not change by more than 0.05V. This means < 10 mOhm.

...

A rough estimate for standard copper traces on such PCBs is (I hope they did not make them thinner):

1cm wire 100µm wide means 50 mOhm.

So 500µm wide and 1cm long means 10mOhm. We might have 2cm...

But all this is rough guess.

So we might have 20mOhm here? And we need < 10mOhm?

Or did I understand something wrong?

(Yes, I know it's just an estimation)

It is an estimation of what we have, not what we certainly need.

BR, Nikolaus

small update:

Am 10.11.2016 um 14:50 schrieb H. Nikolaus Schaller hns@goldelico.com:

Hi,

...

...

...

...

...

• VDD-MPU and GND resistance between Palmas and OMAP is <0.3 Ohmm

(my Multimeters are not more precise)

Not sure what the maximum allowed value is. :)

Well, we can estimate. If the core draws 5A peak current I would say the 1.25V should not change by more than 0.05V. This means < 10 mOhm.

...

A rough estimate for standard copper traces on such PCBs is (I hope they did not make them thinner):

1cm wire 100µm wide means 50 mOhm.

So 500µm wide and 1cm long means 10mOhm. We might have 2cm...

But all this is rough guess.

So we might have 20mOhm here? And we need < 10mOhm?

Or did I understand something wrong?

(Yes, I know it's just an estimation)

It is an estimation of what we have, not what we certainly need.

Now I have finished some real measurements.

What I did was to connect a power supply with current limitation control (set to 1 Ampere) right to the tips of a millivoltmeter.

And use the tips to short circuit some copper traces.

A reference resistor of 10 mOhm did show ~13mV while connecting the tips directly showed ~3mV. This means 10 mV difference at 1A and is indeed 10 mOhms.

Next, I measured between several contact pads on the naked CPU board (already converted into Ohms):

Palmas-GND - twl6040-GND ca. 4 mOhm Palmas GND - OMAP-GND ca. 4 mOhm

(SMPS measured at the inductor output):

Palmas-SMPS1 - OMAP-VDD-MPU ca. 12 mOhm Palmas-SMPS3 - OMAP-VDD-MPU ca. 9 mOhm

So this looks like being almost exactly the rough estimate I had written previously.

Interestingly I did have approx. 7.5 mOhm between Palmas-SMPS1 and Palmas-SMPS3 although they are quite close to each other. They are connected by 2 VIAs and one short copper trace. So the resistance of the VIAs is higher (ca. 2-3 mOhm) than I would have expected.

This means adding more VIAs here does not harm, could but also might not solve any problem we have. At least it can not introduce new problems.

Also interesting was the measured resistance between two almost adjacent BGA pads of the OMAP. There was 4mOhms as well. But dependent on how much I pressed on the PCB.

So my needles have some influence on the results and the real resistance might be lower. On the other hand the BGA balls also have some resistance so this might properly model the measurements inside the OMAP.

A real problem has shown up for the connection of the new (wasn't in the first CPU boards) capacitor for VDD-MMU where you proposed to play with the values. This one shows approx. 33 mOhms. Together with 10 µF it makes a limit frequency in the low MHz range.

In other words: this capacitor has no decoupling effect at all. This might explain why we have no improvement @ 1.5GHz - if the capacitor theory is right.

Conclusions

=> series resistance of GND from Palmas to OMAP seems to be low enough => series resistance of VDD-MPU from Palmas to OMAP seems to be low enough => e.g. 3 A difference makes 3-4 mV difference of VDD-MPU and 1.2mV on GND => this is smaller than how precise we can set the OPP voltages => the resistance is IMHO in green range => anyways, it would be good if the VIAs would have lower resistance => put on to-do list for mass production revision => the additional capacitor could have been connected at a much lower resistance because it has a too small trace width - although there had been space in the innermost layers. It was a very late addition done in a hurry, obviously not thinking about it enough. => changing this value will not show any effect at high speeds :(

I have tried to bridge the weak wire to the capacitor with an external wire.

Unfortunately there is as expected no visible difference (well, this statistics isn't very precise due to small sample size). But as before it did come to login: in 2 out of 10 attempts and stuck in 8 of 10 (@1.25 GHz OPP enabled).

Then I made another observation which makes me believe a little more in a SW issue again:

If the kernel freezes, it is always between 6 and 10 seconds kernel time (@1.25 GHz).

Never before and never after this.

If the kernel runs successfully for 10 seconds, the boot process completes and after ~30 seconds we have a login: prompt.

This 6-10 seconds windows is for 30 boot attempts where a total of 7 did succeed.

Now I really wonder what the kernel doing in this time frame which can make it freeze here but not outside this window?

Some interrupt initialization? Waiting during a spinlock for some wrong address to have a specific state?

In addition we know that clock initialization and switching from U-Boot speed (1GHz) happens at ca. 3 seconds. That is where 1.5 GHz freezes.

Anyone with other ideas?

Or explanations for this effect (immediate crash @3 seconds @1.5GHz, freeze never between 3 and 6 seconds or after 10 seconds @1.25GHz)?

BR, Nikolaus

PS: is it possible to run U-Boot @ 1.5GHz?

Here is another random idea:

perhaps because of different capacitor setup (compared to uEVM) the voltage needs more time to ramp up. Something like attached patch could be used to find out. It would also be interesting to find out if it crashes on regulator_set_voltage() or clk_set_rate(), which could be found by increasing sleep to a very high value and adding some prints. If the former, it could mean something bad happens during the voltage change.

Gražvydas

Am 10.11.2016 um 18:47 schrieb H. Nikolaus Schaller hns@goldelico.com:

small update:

...

Am 10.11.2016 um 14:50 schrieb H. Nikolaus Schaller hns@goldelico.com:

Hi,

...

...

...

...

> * VDD-MPU and GND resistance between Palmas and OMAP is <0.3 Ohmm > (my Multimeters are not more precise) Not sure what the maximum allowed value is. :)

Well, we can estimate. If the core draws 5A peak current I would say the 1.25V should not change by more than 0.05V. This means < 10 mOhm.

...

A rough estimate for standard copper traces on such PCBs is (I hope they did not make them thinner):

1cm wire 100µm wide means 50 mOhm.

So 500µm wide and 1cm long means 10mOhm. We might have 2cm...

But all this is rough guess.

So we might have 20mOhm here? And we need < 10mOhm?

Or did I understand something wrong?

(Yes, I know it's just an estimation)

It is an estimation of what we have, not what we certainly need.

Now I have finished some real measurements.

What I did was to connect a power supply with current limitation control (set to 1 Ampere) right to the tips of a millivoltmeter.

And use the tips to short circuit some copper traces.

A reference resistor of 10 mOhm did show ~13mV while connecting the tips directly showed ~3mV. This means 10 mV difference at 1A and is indeed 10 mOhms.

Next, I measured between several contact pads on the naked CPU board (already converted into Ohms):

Palmas-GND - twl6040-GND ca. 4 mOhm Palmas GND - OMAP-GND ca. 4 mOhm

(SMPS measured at the inductor output):

Palmas-SMPS1 - OMAP-VDD-MPU ca. 12 mOhm Palmas-SMPS3 - OMAP-VDD-MPU ca. 9 mOhm

So this looks like being almost exactly the rough estimate I had written previously.

Interestingly I did have approx. 7.5 mOhm between Palmas-SMPS1 and Palmas-SMPS3 although they are quite close to each other. They are connected by 2 VIAs and one short copper trace. So the resistance of the VIAs is higher (ca. 2-3 mOhm) than I would have expected.

This means adding more VIAs here does not harm, could but also might not solve any problem we have. At least it can not introduce new problems.

Also interesting was the measured resistance between two almost adjacent BGA pads of the OMAP. There was 4mOhms as well. But dependent on how much I pressed on the PCB.

So my needles have some influence on the results and the real resistance might be lower. On the other hand the BGA balls also have some resistance so this might properly model the measurements inside the OMAP.

A real problem has shown up for the connection of the new (wasn't in the first CPU boards) capacitor for VDD-MMU where you proposed to play with the values. This one shows approx. 33 mOhms. Together with 10 µF it makes a limit frequency in the low MHz range.

In other words: this capacitor has no decoupling effect at all. This might explain why we have no improvement @ 1.5GHz - if the capacitor theory is right.

Conclusions

=> series resistance of GND from Palmas to OMAP seems to be low enough => series resistance of VDD-MPU from Palmas to OMAP seems to be low enough

Here from 7.3.1

• Max Effective Resistance (Reff) budget can range from 4 – 50mΩ for key Device power rails not including ground returns

So we appear to be on the good side regarding resistance.

...

=> e.g. 3 A difference makes 3-4 mV difference of VDD-MPU and 1.2mV on GND => this is smaller than how precise we can set the OPP voltages => the resistance is IMHO in green range => anyways, it would be good if the VIAs would have lower resistance => put on to-do list for mass production revision => the additional capacitor could have been connected at a much lower resistance because it has a too small trace width - although there had been space in the innermost layers. It was a very late addition done in a hurry, obviously not thinking about it enough. => changing this value will not show any effect at high speeds :(

I have tried to bridge the weak wire to the capacitor with an external wire.

Unfortunately there is as expected no visible difference (well, this statistics isn't very precise due to small sample size). But as before it did come to login: in 2 out of 10 attempts and stuck in 8 of 10 (@1.25 GHz OPP enabled).

Then I made another observation which makes me believe a little more in a SW issue again:

If the kernel freezes, it is always between 6 and 10 seconds kernel time (@1.25 GHz).

Never before and never after this.

If the kernel runs successfully for 10 seconds, the boot process completes and after ~30 seconds we have a login: prompt.

This 6-10 seconds windows is for 30 boot attempts where a total of 7 did succeed.

Now I really wonder what the kernel doing in this time frame which can make it freeze here but not outside this window?

Some interrupt initialization? Waiting during a spinlock for some wrong address to have a specific state?

In addition we know that clock initialization and switching from U-Boot speed (1GHz) happens at ca. 3 seconds. That is where 1.5 GHz freezes.

Anyone with other ideas?

Or explanations for this effect (immediate crash @3 seconds @1.5GHz, freeze never between 3 and 6 seconds or after 10 seconds @1.25GHz)?

BR, Nikolaus

PS: is it possible to run U-Boot @ 1.5GHz?

---

Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel

Am Thu, 10 Nov 2016 18:47:56 +0100 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:

Hi,

small update to my reply yesterday:

I recorded a quick video showing the power consumption of a boot-up process of the Pyra. The AC adaptor doesn't update very often (about once per second), but I think it's good enough.

It was still running the standard 1GHz frequency. I'll test different ones as soon as I can create my own board files.

Here is the video: https://youtu.be/aJRUnoKVWN0

I switch on the system at 00:04. Power consumption goes up to 0,57A.

At 00:07 / 00:08 it goes up to 0,74A. It probably has a lot higher peaks, as I've seen it go up to 0,87A as well with multiple tries.

That's exactly after the 3 seconds you reported (where 1,5GHz crashes).

Then the power consumption varies a bit (but never gets that high again) and after about 10 seconds, it goes down to 0,48A and mostly stays there during the boot up process.

So yes, it seems power consumption peaks after 3 seconds and then has another few peaks until everything settles down after 12 seconds.

That should explain why the board crashes immediately after 3 seconds at 1,5GHz and doesn't crash anymore when after 10 seconds when running with 1,25GHz.

I guess the kernel maxes the CPU usage when looking for the devices which is too much on our CPU board and it breaks down MPU voltage which leads to the freeze.

At 1,25GHz, we're probably right at the edge - so it boots sometimes. And if it survives the first 10 seconds, then power consumption goes down and there's no reason to crash anymore.

When running single core, power consumption of the OMAP5 is lower as well, that's why 1,5GHz work fine with a single core.

So it seems to be power related.

BUT there's still one thing: Is it the OVERALL power consumption that causes a problem or just peaks? Maybe your capacitor setup is not enough to buffer peaks but would be able to keep the power stable if we are able to reduce the peaks.

Maybe we can make the PALMAS react less aggressive, so that it doesn't increase the voltage suddenly (leading to peaks) but in lower steps?

If the kernel runs successfully for 10 seconds, the boot process completes and after ~30 seconds we have a login: prompt.

This 6-10 seconds windows is for 30 boot attempts where a total of 7 did succeed.

Now I really wonder what the kernel doing in this time frame which can make it freeze here but not outside this window?

Some interrupt initialization? Waiting during a spinlock for some wrong address to have a specific state?

In addition we know that clock initialization and switching from U-Boot speed (1GHz) happens at ca. 3 seconds. That is where 1.5 GHz freezes.

Anyone with other ideas?

Or explanations for this effect (immediate crash @3 seconds @1.5GHz, freeze never between 3 and 6 seconds or after 10 seconds @1.25GHz)?

BR, Nikolaus

PS: is it possible to run U-Boot @ 1.5GHz?

Hi,

while looking at the different voltages, I found another value that spawned my interest:

clock-latency = <300000>;

Does anyone know what that value is and what it could be used for?

I found some discussion on a forum, where someone mentioned it's probably the latency between the clock source pin and the OMAP5 clock pin and therefore depends on board design and trace length.

No idea if that's true, but if yes, could that have an effect in our case?

Hi,

from a quick look at the code, it seems it's only actually used by cpufreq to make decisions when to switch frequencies (opp code only passes the value to cpufreq). For example, if the latency is large, it may make sense to wait for longer CPU usage spike before switching to higher frequency to avoid useless switching back and forth.

Gražvydas

On Fri, Nov 11, 2016 at 10:19 PM, Michael Mrozek EvilDragon@openpandora.org wrote:

Hi,

while looking at the different voltages, I found another value that spawned my interest:

clock-latency = <300000>;

Does anyone know what that value is and what it could be used for?

I found some discussion on a forum, where someone mentioned it's probably the latency between the clock source pin and the OMAP5 clock pin and therefore depends on board design and trace length.

No idea if that's true, but if yes, could that have an effect in our case?

-- Mit freundlichen Grüßen,

Michael Mrozek

---

OpenPandora GmbH Geschäftsführer: Michael Mrozek

Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt

---

eMail: mrozek@openpandora.org _______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel

On Sat, Nov 12, 2016 at 12:04 AM, Michael Mrozek EvilDragon@openpandora.org wrote:

Another idea I had: Can ANY voltage be defined in the board file?

The palmas manual states that 0.5V - 1.65V can be programmed in 0.01V steps. Device tree specifies a bit narrower range of 0.6 - 1.5V, just to be safe I guess.

The OMAP manual says the allowed range is 0.85V - 1.25V. "The voltage at the device ball should never be below the MIN voltage or above the MAX voltage for any amount of time. This requirement includes dynamic voltage events such as AC ripple, voltage transients, voltage dips, etc."

Or could it be that 1,35V / 1,45V is not allowed by the driver and the PALMAS sticks to the default voltage in case it has an illegal voltage?

I don't think so. I think Nikolaus said he has measured the voltage to be really changing, I guess he could confirm that again just in case...

Gražvydas

Am Sat, 12 Nov 2016 00:52:02 +0200 hat Grazvydas Ignotas notasas@gmail.com geschrieben:

Okay, here are some more, very interesting test results I just made!

You can grab all the board files I created here: http://openpandora.org/downloads/testing/?C=M;O=D

And the results are attached.

Now, what's interesting?

Mainly two things:

1. Single vs multiple defined speeds

If I only define a SINGLE speed, I could only get up to 1,18GHz. From 1,19 GHz it gets unstable and with 1,20 GHz it always freezes.

When defining a LOT of speeds, it runs better! For example, with this:

500000 850000 750000 950000 1000000 1060000 1100000 1100000 1150000 1150000 1170000 1170000 1190000 1190000 1200000 1200000 1220000 1220000 1250000 1250000

I can run up to 1,25GHz without any issues. It didn't crash in high-load running for multiple minutes. The voltage used at 1,25GHz wasn't set any higher than with the lower speeds that crashed for me.

Why did this run stable? Because the voltage wasn't increased from 1,06V to 1,25V at once but in smaller steps? That could mean our capacitors aren't able to buffer peaks as good as they should.

Unfortunately, I couldn't get it to run it stable at higher speeds with that trick though, so at some time it simply seems our power supply is not stable enough anymore.

2. Voltages

I'm not 100% sure the voltages work as they should. While it does seem have SOME effect (feels a bit more stable sometimes), it doesn't really seem like it's 100% working.

For example, if I set the voltage to 0,75V (less than the OMAPs allowed minimum), it still runs stable at 1,15GHz. Same with 0,85V at 1GHz.

I expected it to crash right away, with such a low voltage setting for that high frequencies.

So that makes me wonder: Could it be there's some bug somewhere in our board file that leads to a bug where the voltage is not properly applied?

Is there some easy way to check what voltage it really runs?

I will try to add the max. power consumption peaks for some of the interesting values within the next few days.

...

Another idea I had: Can ANY voltage be defined in the board file?

The palmas manual states that 0.5V - 1.65V can be programmed in 0.01V steps. Device tree specifies a bit narrower range of 0.6 - 1.5V, just to be safe I guess.

The OMAP manual says the allowed range is 0.85V - 1.25V. "The voltage at the device ball should never be below the MIN voltage or above the MAX voltage for any amount of time. This requirement includes dynamic voltage events such as AC ripple, voltage transients, voltage dips, etc."

...

Or could it be that 1,35V / 1,45V is not allowed by the driver and the PALMAS sticks to the default voltage in case it has an illegal voltage?

I don't think so. I think Nikolaus said he has measured the voltage to be really changing, I guess he could confirm that again just in case...

Gražvydas

Hi, I have found a new phenomenon which appears to be closely related and needs discussion.

Last Friday, I picked up the remaining newly built CPU-Boards and 12 of them are 4 GB RAM while 18 are 2GB with IM chips.

I tested the 2GB first and except three of them they booted well and without immediately visible problems.

Then I started to test the 4GB boards with Samsung chips.

All of them run U-Boot, load the kernel, but they hang around 3-6 seconds.

But this is with the DT that is limited to 1GHz!

Interestingly I found this in the log:

[ 2.890039] SMPS6: supplied by vsys_cobra [ 2.894966] smps6: Bringing 1350000uV into 1200000-1200000uV

What is SMPS6? It drives the VDD-DDR3L.

So it looks as if U-Boot starts with 1.35V and the Kernel actively reduces it to 1.2V.

If this is too low, we risk that the DRAM fails spuriously.

It does not completely explain why this also depends on the Cortex-A15 clock and OPPs because the EMIF always runs with the same external data rate. This is at least what I understand. There might only be less moments of RAM inactivity.

But it might explain why we did never see this on EVM or IGEP. AFAIK, they use a third type of DRAM chips (Micron).

So we have new questions: a) is 1.2V allowed for the DRAM chips we use b) if not, why does the kernel reduce this voltage

BR, Nikolaus

Am 14.11.2016 um 10:46 schrieb H. Nikolaus Schaller hns@goldelico.com:

Hi, I have found a new phenomenon which appears to be closely related and needs discussion.

Last Friday, I picked up the remaining newly built CPU-Boards and 12 of them are 4 GB RAM while 18 are 2GB with IM chips.

I tested the 2GB first and except three of them they booted well and without immediately visible problems.

Then I started to test the 4GB boards with Samsung chips.

All of them run U-Boot, load the kernel, but they hang around 3-6 seconds.

But this is with the DT that is limited to 1GHz!

Interestingly I found this in the log:

[ 2.890039] SMPS6: supplied by vsys_cobra [ 2.894966] smps6: Bringing 1350000uV into 1200000-1200000uV

What is SMPS6? It drives the VDD-DDR3L.

So it looks as if U-Boot starts with 1.35V and the Kernel actively reduces it to 1.2V.

This mechanism might have been introduced recently and older kernels may have simply used the setting by U-Boot which appears to be 1.35V.

If this is too low, we risk that the DRAM fails spuriously.

It does not completely explain why this also depends on the Cortex-A15 clock and OPPs because the EMIF always runs with the same external data rate. This is at least what I understand. There might only be less moments of RAM inactivity.

Some explanation could be that higher current drawn by the OMAP5 core increases overall noise and running the DRAMs at wrong volts becomes more critical.

But it might explain why we did never see this on EVM or IGEP. AFAIK, they use a third type of DRAM chips (Micron).

So we have new questions: a) is 1.2V allowed for the DRAM chips we use

Answer: NO

for IM chips:

• Power supply : VDD, VDDQ = 1.35V (1.283V to 1.45V) - Backward compatible to VDD, VDDQ = 1.5V ± 0.075V

for Samsung chips:

• JEDEC standard 1.35V(1.28V~1.45V) & 1.5V(1.425V~1.575V) • VDDQ = 1.35V(1.28V~1.45V) & 1.5V(1.425V~1.575V)

Well, we have to check if this is just a spurious message from the kernel or if it is for real.

b) if not, why does the kernel reduce this voltage

Well, because it is instructed so:

http://git.goldelico.com/?p=gta04-kernel.git;a=blob;f=arch/arm/boot/dts/omap...

I will try to set this to 1350000.

BR, Nikolaus

Am Mon, 14 Nov 2016 12:19:35 +0100 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:

Hi,

Good news! As adding multiple frequencies made my board run stable with a 1,25GHz as well (even though it never worked with anything higher than 1,8GHz before), maybe this can help.

If you don't get it fully working today I can do some more experiments tonight as well.

BTW: Is it worth checking other voltages of the PALMAS as well or is the rest just common stuff that should be the same for us as for the EVM?

...

I will try to set this to 1350000.

Result: 4GB CPU now reliably (as far as 3 boot attempts can tell) boots at 1.25GHz. And 2GB CPU as well...

So it looks as if we have found one important factor.

Is not a hardware but a software bug. Although nobody did believe me :)

Anyways, only the next onion shell has been opened since it does freeze after 6-8 seconds @ 1.5GHz...

Since the DRAM is capable of handling 1.5V we have now a new tool to check if the 1.5Ghz problem is also coming from the DRAM.

BR, Nikolaus

Hi,

Ok, setting SMPS6 to 1.5V does not change anything at 1.5Ghz. So like I described there is another problem behind this onion shell. Maybe this time it is really an hardware (capacitor) issue.

I had some thoughts about my results, maybe here are some ideas that might put you onto the right track:

Based on my experience so far, the higher clockspeed a CPU runs, the higher the voltage it needs has to be. To overclock a CPU people often overvolt it.

-> It's not only the overall power supply that's important here, but also the voltage. 5W is in this case not 5W.

Also based on my experience is the behaviour of a CPU when you overclock it: The further you overclock, the more calculation errors it does. That's why it becomes unstable the more you overclock it. That's also showing in my tests: While my board needs about 20 - 30 seconds to freeze running with 1,19GHz and 1,06V, it already shows lots of kernel errors until then, which means the processor goes haywire.

Increasing the speed to 1,2GHz makes it freeze right away.

-> The CPU behaves exactly like it's overclocking (which is is at 1,06V)

Increasing the voltage to 1,25V improves things a bit (it takes 10 - 15 seconds more until it crashes), however, at 1,25V, the OMAP5 should work fine.

-> It looks like the voltage is not high enough

Interesting though is that it works better if I add more inbetween frequencies with inbetween voltages which lets me run it stable with 0,06 GHz more.

-> Besides the voltage, we seem to also have the effect that something goes wrong with sudden voltage changes

The PALMAS should take care of the voltage, and it does so by measuring the voltage at the OMAP5, to compensate any trace resistance values we have.

That doesn't seem to work for higher voltages (1,25V) though.

-> Something is lowering the voltage OR something causes the voltage to be unstable (as the OMAP5 is a bit picky and the voltage should be mostly stable). Our traces have been measured and seem to be okay, so that doesn't seem to be the issue.

Now combining that with what I learned about capacitors (I'm not sure I got everything correct...)

Capacitors also have a resistance which changes based on the frequency (ESL). This is what TI also notes in their documents. Also, capacity changes with the voltage, which could probably be a factor in our case as well.

If I understood correctly, as different capacitors behave differently with these parameters, you use multiple different sized ones in such cases. This should minimize the capacity loss and also the resistance leading to a much more stable voltage. As current usually takes the way of the least resistance, multiple capacitors should improve things here, right?

Another thing that comes to my mind: If the capacity / resistance changes in a capacitor, could it be the PALMAS is constantly changing the voltage when running at 1,25V to compensate that? So basically that the changing resistance / capacity of the capacitor is causing the PALMAS to always trying to correct the voltage?

In that case, we'd have a pretty unstable voltage, which could also explain why it works better in case we don't have a sudden increase from 1,06V to 1,25V but using many voltage steps inbetween.

Another idea I had: AFAIK, the bigger a capacitor is, the longer it takes to adapt to a new voltage. So maybe our capacitor is too big which means that the CPU clocks higher before it's getting the higher stable voltage. That would also explain why voltage steps improve things.

The question for me is:

* What role plays the total capacity (with all capacitors combined)? * What role plays the position of the capacitors? If it's more about having different valued capacitors instead of just one, then manually soldering them in parallel to the existing ones we have on the CPU board might improve the situation? * What role play the single capacity of a capacitor? Do larger ones have a larger increase of resistance / decrease of capacity with higher voltage than smaller ones? * What other differences are there when using two smaller capacitors compared to using one bigger one?

I guess something that's important is to simply try to replace existing capacitors with different ones or piggyback / add additional ones to the side or top.

It might not improve things, but it should at least have SOME effect (even if it makes things worse), right?

Hi, thanks to combining some test approaches we have recently developed together, I have done some new tests with new insights.

What I have played with are these factors: a) high-load with bash-loop vs. NEON-loop (the assembler program from Notaz) b) different VSYS settings in bq24297 (3.5V ... 3.85V), i.e. what is fed into the PALMAS c) different OMAP5 clock range (including/excluding the 1.25GHz OPP) d) try to evaluate at different temperatures (which is difficult since OMAP heats up fast)

All this was done on a single CPU board (C22 a 2GB RAM from the new production. This board has a retrofitted VDD-MPU bypass wire. I need to compare with other CPU boards as well).

This did lead to some interesting results and with them we can fortunately exclude some potential reasons. And it seems as if there is a combination which can trigger a freeze within seconds, so that we don't have to wait for 30 minutes if it occurs or not. This is important for quickly evaluating potential fixes.

Now here are the key results: * changing VSYS between 3.5 and 3.85V has no influence * running bash-loop works fine at 1.2GHz at any VSYS and temperature * VSYS goes down a little with rising temperature but this effect is much lower (less than 50mV) than on the EVM@12V * bash-loop heats up fast but does not freeze * it shows that the thermal throttling makes clock OPP go down to 750/500 MHz, sometimes in multiple steps * since this OPP change also includes VDD-MPU changes (between 1.25V and 0.85V) it shows that the Palmas regulation works fine * what makes a really big difference is running the bash-loop vs. the NEON-loop * high-load freezes almost immediately after switching from 1GHz to 1.25GHz I did not get more than two reports with 1250MHz. And this independent of VSYS or temperature! Suche a deciding factor is what I have been looking for for a while.

The freeze-matrix looks like:

| 1.0 GHz | 1.25GHz | | bash | NEON | NEON | bash | -------------------------+-------+-------+-------+-------+ low VSYS | ok | ok |freeze | ok | high ----------+-------+-------+-------+-------+ Temperature high VSYS | ok | ok |freeze | ok | -------------------------+-------+-------+-------+-------+ low high VSYS | ok | ok |freeze | ok | Temperature ----------+-------+-------+-------+-------+ low VSYS | ok | ok |freeze | ok | -------------------------+-------+-------+-------+-------+

This IMHO excludes: * it is not a temperature dependent effect (that would have been the worst case) * it is not a problem of the Palmas regulator switching between OPPs * it is indeed the Cortex A15 MPU/NEON core and not any peripheral on the OMAP5 SoC A weird reason could still be that the DRAM interface is used differently/faster if we run NEON at >1GHz. But the EMIF DRAM clock is constant if CPU clock changes. * it is unlikely a kernel software problem: the kernel does not explicitly control the combination of user-space NEON + CPU clock frequency * it is not a (direct) effect of the bq24297 VSYS output (although it might still be a short breakdown. I want to run another test which is much easier to do now with a known "freeze" command. Anyways this effect should be influenced by VSYS).

This means that the list of potential reasons is rapidly reducing.

As soon as there is only one remaining, we know where an improvement has the highest chance of fixing it with a single shot. It is more likely now that it is indeed a weakness in decoupling. The question will remain how small or big it is. This would define the amount of PCB change we need.

BR, Nikolaus

Here are some examples how the measurements looks like.

A) 1.25GHz enabled:

root@letux:~# ./high-load -n 100% load stress test for 2 cores Sun Jan 2 07:20:54 UTC 2000 39° 38° 39° 3816mV 1000MHz Sun Jan 2 07:20:54 UTC 2000 39° 38° 39° 3822mV 1000MHz Sun Jan 2 07:20:56 UTC 2000 40° 41° 39° 3808mV 1250MHz Sun Jan 2 07:20:57 UTC 2000 50° 42° 41° 3805mV 1250MHz -- freeze --

root@letux:~# i2cset -f -y 1 0x6b 0x01 0x1f root@letux:~# ./high-load 100% load stress test for 2 cores Sun Jan 2 07:32:57 UTC 2000 54° 56° 52° 3819mV 1000MHz Sun Jan 2 07:32:58 UTC 2000 54° 56° 52° 3819mV 1000MHz Sun Jan 2 07:32:59 UTC 2000 55° 55° 53° 3808mV 1250MHz Sun Jan 2 07:33:00 UTC 2000 65° 57° 57° 3808mV 1250MHz .... Sun Jan 2 07:35:01 UTC 2000 94° 87° 83° 3786mV 1250MHz Sun Jan 2 07:35:03 UTC 2000 96° 89° 85° 3787mV 1250MHz Sun Jan 2 07:35:04 UTC 2000 97° 91° 86° 3788mV 1250MHz Sun Jan 2 07:35:06 UTC 2000 99° 92° 87° 3799mV 1250MHz Sun Jan 2 07:35:07 UTC 2000 99° 93° 88° 3800mV 1250MHz Sun Jan 2 07:35:09 UTC 2000 99° 93° 88° 3813mV 750MHz Sun Jan 2 07:35:10 UTC 2000 100° 93° 89° 3826mV 500MHz Sun Jan 2 07:35:12 UTC 2000 101° 89° 83° 3824mV 750MHz Sun Jan 2 07:35:13 UTC 2000 89° 89° 83° 3804mV 1250MHz Sun Jan 2 07:35:15 UTC 2000 86° 87° 82° 3802mV 1250MHz Sun Jan 2 07:35:16 UTC 2000 94° 87° 84° 3783mV 1250MHz Sun Jan 2 07:35:18 UTC 2000 98° 91° 87° 3781mV 1250MHz Sun Jan 2 07:35:19 UTC 2000 99° 92° 86° 3781mV 1250MHz Sun Jan 2 07:35:21 UTC 2000 100° 93° 88° 3822mV 750MHz Sun Jan 2 07:35:22 UTC 2000 100° 93° 88° 3825mV 500MHz Sun Jan 2 07:35:24 UTC 2000 89° 90° 84° 3824mV 1000MHz Sun Jan 2 07:35:25 UTC 2000 89° 87° 81° 3794mV 1250MHz Sun Jan 2 07:35:27 UTC 2000 87° 87° 81° 3800mV 1250MHz Sun Jan 2 07:35:28 UTC 2000 94° 88° 84° 3780mV 1250MHz Sun Jan 2 07:35:30 UTC 2000 97° 90° 86° 3800mV 1250MHz ^Ckill 2554 2555

root@letux:~# ./high-load -n 100% load stress test for 2 cores Sun Jan 2 07:35:35 UTC 2000 87° 89° 84° 3815mV 1000MHz Sun Jan 2 07:35:35 UTC 2000 87° 89° 84° 3815mV 1000MHz -- freeze --

B) cores limited to 1GHz:

root@letux:~# ./high-load -n 100% load stress test for 2 cores Sun Jan 2 08:22:59 UTC 2000 45° 44° 43° 3825mV 750MHz Sun Jan 2 08:22:59 UTC 2000 45° 44° 43° 3826mV 750MHz Sun Jan 2 08:23:01 UTC 2000 43° 43° 42° 3821mV 1000MHz Sun Jan 2 08:23:02 UTC 2000 49° 44° 43° 3820mV 1000MHz Sun Jan 2 08:23:04 UTC 2000 52° 46° 46° 3822mV 1000MHz Sun Jan 2 08:23:05 UTC 2000 53° 48° 47° 3821mV 1000MHz .... Sun Jan 2 09:03:18 UTC 2000 99° 96° 90° 3807mV 1000MHz Sun Jan 2 09:03:20 UTC 2000 100° 95° 90° 3813mV 1000MHz Sun Jan 2 09:03:21 UTC 2000 100° 95° 90° 3814mV 1000MHz Sun Jan 2 09:03:23 UTC 2000 100° 95° 91° 3814mV 1000MHz Sun Jan 2 09:03:24 UTC 2000 100° 95° 91° 3814mV 1000MHz Sun Jan 2 09:03:26 UTC 2000 101° 96° 91° 3825mV 500MHz Sun Jan 2 09:03:27 UTC 2000 100° 95° 89° 3819mV 750MHz Sun Jan 2 09:03:29 UTC 2000 93° 94° 89° 3814mV 1000MHz Sun Jan 2 09:03:30 UTC 2000 92° 93° 87° 3811mV 1000MHz Sun Jan 2 09:03:32 UTC 2000 96° 93° 88° 3815mV 1000MHz Sun Jan 2 09:03:33 UTC 2000 98° 93° 89° 3814mV 1000MHz Sun Jan 2 09:03:35 UTC 2000 99° 94° 89° 3814mV 1000MHz Sun Jan 2 09:03:36 UTC 2000 99° 95° 89° 3814mV 1000MHz Sun Jan 2 09:03:38 UTC 2000 100° 95° 89° 3814mV 1000MHz Sun Jan 2 09:03:39 UTC 2000 100° 95° 90° 3820mV 1000MHz Sun Jan 2 09:03:41 UTC 2000 100° 96° 90° 3815mV 1000MHz -- no freeze yet --

Hi,

Am 23.11.2016 um 11:45 schrieb Grazvydas Ignotas notasas@gmail.com:

On Tue, Nov 22, 2016 at 10:49 PM, H. Nikolaus Schaller hns@goldelico.com wrote:

...

• it shows that the thermal throttling makes clock OPP go down to 750/500 MHz,

sometimes in multiple steps

Hmm still using those TI unsupported low frequencies? I'd drop them just in case they have some problem, they must have had a reason to remove it from mainline.

Well, yes. And we have good reasons to add them... Initially it was to get the boards booted without freezes.

I have tested 6 of the new CPU boards with 1.0GHz and 1.25GHz only and I sometimes get a freeze even with the bash-loop which I didn't see with 500/750 included.

With 500/750/1000/1250 it seems as if the 4GB RAM boards have a slightly higher risk to freeze, but this is not statistically validated.

Anyways this would not be enough reason to keep the lower OPP even when we have fixed the freezes, because there is another effect:

The chips get hotter. During high-load they typically oscillate between 98° and 108° instead of 90 and 100°. The reason seems to be that even with thermal throttling to 1GHz (and not deeper) there is more average energy dissipated inside the chip and this results in a higher average temperature. Well, higher average performance as well :)

This may become a general problem: we might get it up to 1.7GHz peak clock rate by fixing the freeze issue, but thermal throttling might reduce the average clock rate to let's say 1.2GHz. Then it is the same as running them permanently at 1.2GHz.

Also the latest datasheet (swps051k) lists 1.014V as the lowest allowed voltage, I'd not ever go below that in tests like this as you may end up triggering several unrelated issues and then arrive to wrong conclusions.

I have not seen any problem with lower frequency OPPs so far and they significantly reduce power consumption in low-load situations.

So unless we really find a problem I think we better should keep them.

Anyways, having them reduces the 1.25GHz freeze risk a little for the tests I am running.

...

• since this OPP change also includes VDD-MPU changes (between 1.25V

and 0.85V) it shows that the Palmas regulation works fine

• what makes a really big difference is running the bash-loop vs. the NEON-loop
• high-load freezes almost immediately after switching from 1GHz to 1.25GHz

What about 1.2GHz? ED already announced 1.2GHz as fully working on the forums, but I fear it might actually not work with NEON.

Yes, it is likely that it fails at 1.20GHz as well. Maybe a little later than with 1.25GHz. I will try in the next run.

...

• it is indeed the Cortex A15 MPU/NEON core and not any peripheral on the OMAP5 SoC

A weird reason could still be that the DRAM interface is used differently/faster if we run NEON at >1GHz. But the EMIF DRAM clock is constant if CPU clock changes.

The test doesn't use memory at all, it operates on registers only, except repeatedly reading and writing to the same locations, which should always hit L1 cache.

OTOH now that I think about it, it might actually start doing writes after a task switch because ARM doesn't write-allocate cache. You can comment out lines with ldrd and strd instructions (add @ at the start of lines 24 and 29) and then there will be no chance for it to do RAM accesses, except kernel reloading the instructions after a task switch (after interrupts or whatever). This should rule out any RAM issue.

Ah, good to know! I will try that next and see if we have any differences. But anyways it becomes more and more clear that we only have a CPU issue and no peripheral or RAM.

BR and thanks, Nikolaus

Am Wed, 23 Nov 2016 21:03:51 +0100 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:

Hi,

...

This may become a general problem: we might get it up to 1.7GHz peak clock rate by fixing the freeze issue, but thermal throttling might reduce the average clock rate to let's say 1.2GHz. Then it is the same as running them permanently at 1.2GHz.

Maybe, is this the reason why TI dropped the 1.7GHz specification because nobody can use it 100% of the time without overheating the chip even with active cooling?

Linux-Swat is running his devboards (with a better passive cooling than the original) at 1,7GHz without overheating.

So the "freeze-barrier" is somewhere between 1.0 and 1.1 GHz.

For the CPU only and NEON, yes. Don't forget that using the 3D GPU will need more power, so it might freeze with 1GHz already as well.

Am Thu, 24 Nov 2016 00:45:07 +0200 hat Grazvydas Ignotas notasas@gmail.com geschrieben:

Hi,

is there an easy way to read out the REAL voltage? So I can see that the setting really is doing something?

I still find ED's observation that voltage changes almost have no effect really strange, as if VDD-MPU was connected to wrong OMAP balls. Can it really do 1GHz at just 0.75V like ED says? With NEON too? Does 1.06V vs 1.25V setting have any difference for the high frequencies? Perhaps there is some clue here too...

Gražvydas

On Thu, Nov 24, 2016 at 12:03 AM, Michael Mrozek EvilDragon@openpandora.org wrote:

...

Am Wed, 23 Nov 2016 21:03:51 +0100 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:

Hi,

...

...

This may become a general problem: we might get it up to 1.7GHz peak clock rate by fixing the freeze issue, but thermal throttling might reduce the average clock rate to let's say 1.2GHz. Then it is the same as running them permanently at 1.2GHz.

Maybe, is this the reason why TI dropped the 1.7GHz specification because nobody can use it 100% of the time without overheating the chip even with active cooling?

Linux-Swat is running his devboards (with a better passive cooling than the original) at 1,7GHz without overheating.

...

So the "freeze-barrier" is somewhere between 1.0 and 1.1 GHz.

For the CPU only and NEON, yes. Don't forget that using the 3D GPU will need more power, so it might freeze with 1GHz already as well.

-- Mit freundlichen Grüßen,

Michael Mrozek

---

OpenPandora GmbH Geschäftsführer: Michael Mrozek

Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt

---

eMail: mrozek@openpandora.org

I have done this test by soldering a wire to the PALMAS SMPS123 outputs and connecting an oscilloscope:

We clearly see that the voltages change (I did start in idle and type "apt-get update" and ENTER).

Voltages are quite exactly what I have defined in the OPP DT, i.e. 840/880mV going up to 1100mV.

What you also can see is such voltage steps occur not faster than every 300ms.

There is some noise on the supply line but that can be an artifact of the setup and the oscilloscope precision (8bit).

What I have planned to do is a similar test with 1.25GHz and high-load loops.

Interestingly we can even underclock the OMAP5 down to 250 MHz and it appears to run stable as well.

root@letux:~# cpufreq-info cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to cpufreq@vger.kernel.org, please. analyzing CPU 0: driver: cpufreq-dt CPUs which run at the same hardware frequency: 0 1 CPUs which need to have their frequency coordinated by software: 0 1 maximum transition latency: 350 us. hardware limits: 250 MHz - 1000 MHz available frequency steps: 250 MHz, 500 MHz, 750 MHz, 1000 MHz available cpufreq governors: conservative, userspace, powersave, ondemand, performance current policy: frequency should be within 250 MHz and 1000 MHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 250 MHz (asserted by call to hardware). cpufreq stats: 250 MHz:81.74%, 500 MHz:12.91%, 750 MHz:2.24%, 1000 MHz:3.11% (516) analyzing CPU 1: driver: cpufreq-dt CPUs which run at the same hardware frequency: 0 1 CPUs which need to have their frequency coordinated by software: 0 1 maximum transition latency: 350 us. hardware limits: 250 MHz - 1000 MHz available frequency steps: 250 MHz, 500 MHz, 750 MHz, 1000 MHz available cpufreq governors: conservative, userspace, powersave, ondemand, performance current policy: frequency should be within 250 MHz and 1000 MHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 250 MHz (asserted by call to hardware). cpufreq stats: 250 MHz:81.74%, 500 MHz:12.91%, 750 MHz:2.24%, 1000 MHz:3.11% (516) root@letux:~#

This was done with these OPP:

&cpu0 { operating-points = < /* kHz uV */ 250000 800000 500000 850000 750000 950000 1000000 1050000 // 1250000 1150000 // 1500000 1250000 // 1700000 1250000 >; };

As you can see I plan to run a slightly lowered 1.25GHz OPP voltage to be linear between 500 MHz and 1500 MHz.

Am 24.11.2016 um 01:05 schrieb Grazvydas Ignotas notasas@gmail.com:

Not that I know of, only Nikolaus can measure it directly.

Gražvydas

On Thu, Nov 24, 2016 at 1:42 AM, Michael Mrozek EvilDragon@openpandora.org wrote:

...

Am Thu, 24 Nov 2016 00:45:07 +0200 hat Grazvydas Ignotas notasas@gmail.com geschrieben:

Hi,

is there an easy way to read out the REAL voltage? So I can see that the setting really is doing something?

BR, Nikolaus

Am Fri, 25 Nov 2016 13:35:29 +0100 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:

Can you try setting the voltage to something lower (like 0,75V)?

Because that's what still worked with 1GHz and Highload (and that's REALLY undervolting it), so the question is whether there's maybe some limiter somewhere.

I have done this test by soldering a wire to the PALMAS SMPS123 outputs and connecting an oscilloscope:

We clearly see that the voltages change (I did start in idle and type "apt-get update" and ENTER).

Voltages are quite exactly what I have defined in the OPP DT, i.e. 840/880mV going up to 1100mV.

What you also can see is such voltage steps occur not faster than every 300ms.

There is some noise on the supply line but that can be an artifact of the setup and the oscilloscope precision (8bit).

What I have planned to do is a similar test with 1.25GHz and high-load loops.

Interestingly we can even underclock the OMAP5 down to 250 MHz and it appears to run stable as well.

root@letux:~# cpufreq-info cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to cpufreq@vger.kernel.org, please. analyzing CPU 0: driver: cpufreq-dt CPUs which run at the same hardware frequency: 0 1 CPUs which need to have their frequency coordinated by software: 0 1 maximum transition latency: 350 us. hardware limits: 250 MHz - 1000 MHz available frequency steps: 250 MHz, 500 MHz, 750 MHz, 1000 MHz available cpufreq governors: conservative, userspace, powersave, ondemand, performance current policy: frequency should be within 250 MHz and 1000 MHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 250 MHz (asserted by call to hardware). cpufreq stats: 250 MHz:81.74%, 500 MHz:12.91%, 750 MHz:2.24%, 1000 MHz:3.11% (516) analyzing CPU 1: driver: cpufreq-dt CPUs which run at the same hardware frequency: 0 1 CPUs which need to have their frequency coordinated by software: 0 1 maximum transition latency: 350 us. hardware limits: 250 MHz - 1000 MHz available frequency steps: 250 MHz, 500 MHz, 750 MHz, 1000 MHz available cpufreq governors: conservative, userspace, powersave, ondemand, performance current policy: frequency should be within 250 MHz and 1000 MHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 250 MHz (asserted by call to hardware). cpufreq stats: 250 MHz:81.74%, 500 MHz:12.91%, 750 MHz:2.24%, 1000 MHz:3.11% (516) root@letux:~#

This was done with these OPP:

&cpu0 { operating-points = < /* kHz uV */ 250000 800000 500000 850000 750000 950000 1000000 1050000 // 1250000 1150000 // 1500000 1250000 // 1700000 1250000

...

;

};

As you can see I plan to run a slightly lowered 1.25GHz OPP voltage to be linear between 500 MHz and 1500 MHz.

...

Am 24.11.2016 um 01:05 schrieb Grazvydas Ignotas notasas@gmail.com:

Not that I know of, only Nikolaus can measure it directly.

Gražvydas

On Thu, Nov 24, 2016 at 1:42 AM, Michael Mrozek EvilDragon@openpandora.org wrote:

...

Am Thu, 24 Nov 2016 00:45:07 +0200 hat Grazvydas Ignotas notasas@gmail.com geschrieben:

Hi,

is there an easy way to read out the REAL voltage? So I can see that the setting really is doing something?

BR, Nikolaus

Hi,

Am 25.11.2016 um 14:47 schrieb Urja Rannikko urjaman@gmail.com:

Hi and sorry for interjecting like this, but ...

no problem.

On Fri, Nov 25, 2016 at 3:28 PM, Michael Mrozek EvilDragon@openpandora.org wrote:

...

Am Fri, 25 Nov 2016 13:35:29 +0100 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:

Can you try setting the voltage to something lower (like 0,75V)?

Because that's what still worked with 1GHz and Highload (and that's REALLY undervolting it), so the question is whether there's maybe some limiter somewhere.

This, and ...

...

...

I have done this test by soldering a wire to the PALMAS SMPS123 outputs

Could you try to find some place more focused on what the OMAP5 is "receiving"?

Unfortunately no chance since power distribution is in inner PCB layers and comes to the surface under the OMAP chip. So there is no chance to solder wires there...

Am 25.11.2016 um 14:28 schrieb Michael Mrozek EvilDragon@openpandora.org:

Am Fri, 25 Nov 2016 13:35:29 +0100 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:

Can you try setting the voltage to something lower (like 0,75V)?

Because that's what still worked with 1GHz and Highload (and that's REALLY undervolting it), so the question is whether there's maybe some limiter somewhere.

Ok, setting all voltages to 0.75V didn't boot for me (stuck at 4 seconds) so it is probably too low for 750MHz and above.

What works for me on board C22 is this (which shows quite some room for power consumption optimization since P ~ uV ^ 2):

&cpu0 { operating-points = < /* kHz uV */ 250000 750000 500000 750000 750000 950000 1000000 1050000 // 1250000 1150000 // 1500000 1250000 // 1700000 1250000 >; };

The voltmeter did say 0.746V and it went up to 1.043V. So there is no limiter and voltages are quite precise (if measured properly - and a voltmeter is better than an oscilloscope in this case).

Kernel defines a valid range for the regulator driver between 0.6V and 1.5V, i.e. no limiter as well:

http://lxr.free-electrons.com/source/arch/arm/boot/dts/omap5-board-common.dt...

PS: would be interesting to check something like 150MHz and 0.6V... but that does not help for high frequencies PPS: would be interesting to add some OPP interface from user space and automatically plot the range it works...

...

I have done this test by soldering a wire to the PALMAS SMPS123 outputs and connecting an oscilloscope:

We clearly see that the voltages change (I did start in idle and type "apt-get update" and ENTER).

Voltages are quite exactly what I have defined in the OPP DT, i.e. 840/880mV going up to 1100mV.

What you also can see is such voltage steps occur not faster than every 300ms.

There is some noise on the supply line but that can be an artifact of the setup and the oscilloscope precision (8bit).

What I have planned to do is a similar test with 1.25GHz and high-load loops.

Interestingly we can even underclock the OMAP5 down to 250 MHz and it appears to run stable as well.

root@letux:~# cpufreq-info cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to cpufreq@vger.kernel.org, please. analyzing CPU 0: driver: cpufreq-dt CPUs which run at the same hardware frequency: 0 1 CPUs which need to have their frequency coordinated by software: 0 1 maximum transition latency: 350 us. hardware limits: 250 MHz - 1000 MHz available frequency steps: 250 MHz, 500 MHz, 750 MHz, 1000 MHz available cpufreq governors: conservative, userspace, powersave, ondemand, performance current policy: frequency should be within 250 MHz and 1000 MHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 250 MHz (asserted by call to hardware). cpufreq stats: 250 MHz:81.74%, 500 MHz:12.91%, 750 MHz:2.24%, 1000 MHz:3.11% (516) analyzing CPU 1: driver: cpufreq-dt CPUs which run at the same hardware frequency: 0 1 CPUs which need to have their frequency coordinated by software: 0 1 maximum transition latency: 350 us. hardware limits: 250 MHz - 1000 MHz available frequency steps: 250 MHz, 500 MHz, 750 MHz, 1000 MHz available cpufreq governors: conservative, userspace, powersave, ondemand, performance current policy: frequency should be within 250 MHz and 1000 MHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 250 MHz (asserted by call to hardware). cpufreq stats: 250 MHz:81.74%, 500 MHz:12.91%, 750 MHz:2.24%, 1000 MHz:3.11% (516) root@letux:~#

This was done with these OPP:

&cpu0 { operating-points = < /* kHz uV */ 250000 800000 500000 850000 750000 950000 1000000 1050000 // 1250000 1150000 // 1500000 1250000 // 1700000 1250000

...

;

};

As you can see I plan to run a slightly lowered 1.25GHz OPP voltage to be linear between 500 MHz and 1500 MHz.

...

Am 24.11.2016 um 01:05 schrieb Grazvydas Ignotas notasas@gmail.com:

Not that I know of, only Nikolaus can measure it directly.

Gražvydas

On Thu, Nov 24, 2016 at 1:42 AM, Michael Mrozek EvilDragon@openpandora.org wrote:

...

Am Thu, 24 Nov 2016 00:45:07 +0200 hat Grazvydas Ignotas notasas@gmail.com geschrieben:

Hi,

is there an easy way to read out the REAL voltage? So I can see that the setting really is doing something?

BR, Nikolaus

-- Mit freundlichen Grüßen,

Michael Mrozek

---

OpenPandora GmbH Geschäftsführer: Michael Mrozek

Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt

---

eMail: mrozek@openpandora.org

Am Sat, 26 Nov 2016 23:25:03 +0100 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:

Hi,

it means that something is consuming all your CPU power - which is probably the NEON loop ;)

http://unix.stackexchange.com/questions/252045/rcu-preempt-detected-stalls-o...

What does this "INFO: rcu_preempt self-detected stall on CPU" mean? Does this give a hint what is going wrong? Is something wrong in some SoC interconnect? Or is just one core recognising that the other core did freeze?

BR and thanks, Nikolaus

---

Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel

Am 27.11.2016 um 09:13 schrieb H. Nikolaus Schaller hns@goldelico.com:

...

Am 27.11.2016 um 02:59 schrieb Michael Mrozek EvilDragon@openpandora.org:

Am Sat, 26 Nov 2016 23:25:03 +0100 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:

Hi,

it means that something is consuming all your CPU power - which is probably the NEON loop ;)

http://unix.stackexchange.com/questions/252045/rcu-preempt-detected-stalls-o...

Yes, this would mean that the NEON pipeline stops processing, i.e. is no longer advancing to the next assembler instruction.

But why does it occur only with this voltage level? And how can it happen at all?

I think I have an answer what happens.

There are two A15 cores. And due to slightly different parameters one of the core continues to work while for the other some transistors are too slow for the given frequency and voltage level (they are faster for higher voltage) and some flip-flops stop switching when they should.

This mangles up instruction processing and the second core recognises that the other one is not completing some rcu task (by setting some semaphore bit).

And the LEDs threads and the controlling bash command (crl-C) are frozen as well because these are scheduled to run on the frozen A15 core.

So it is just a very peculiar symptom of a low-voltage/high-frequency partial freeze.

...

...

What does this "INFO: rcu_preempt self-detected stall on CPU" mean? Does this give a hint what is going wrong? Is something wrong in some SoC interconnect? Or is just one core recognising that the other core did freeze?

BR and thanks, Nikolaus

---

Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel

-- Mit freundlichen Grüßen,

Michael Mrozek

---

OpenPandora GmbH Geschäftsführer: Michael Mrozek

Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt

---

eMail: mrozek@openpandora.org

---

Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel

Hi,

Am 27.11.2016 um 15:11 schrieb Grazvydas Ignotas notasas@gmail.com:

On Sun, Nov 27, 2016 at 11:02 AM, H. Nikolaus Schaller hns@goldelico.com wrote:

...

So it is just a very peculiar symptom of a low-voltage/high-frequency partial freeze.

Well it could be even simpler than that - it could be some corruption caused by incorrect results from some CPU processing that has left some semaphores locked or similar.

Ok, this is very plausible.

We have seen that a lot on pandora where games would start to show glitches or misbehave when people overclock, but still carry on and not hang. Such problems have been reported as software bugs to me several times, which was extremely annoying.

Ouch.

That's why I don't like your use of low frequencies that are not approved by TI.

I use them for my board but nothing else... The final OS installed can be different from my setup and has to be discussed.

If it doesn't hang it doesn't mean it works. The low frequencies are usually entered when CPU is idle and doing nothing, so the problems are unlikely to trigger, but then when you do a test that constantly thermal throttles (like the one you recently posted), the CPUs run at 100% at a low unapproved OPP and is quote likely they'll fail there instead of the high frequency, and you then misinterpret the result thinking it failed at the high frequency.

There was no detectable coincidence of failure and thermal throttling to occur or not.

And it disappears by raising the voltage a little (at exactly the same set of low clock frequencies and temperatures).

So the only influencing factor was the voltage at high frequency OPPs and not the presence of low frequencies.

Of course that is just an observation without a warranty from TI.

Sure, saving power and reducing temperatures is great, by we should do that optimization after we know the SoC works in conditions specified by TI. As we don't have low OPP data from TI (like we had for OMAP3), we'll need careful tests that do lots of calculations without stalling the CPUs and verifies the calculation results and run them for a long time on lots of boards forced to those low OPPs to prove they indeed work. This is all challenging

and time consuming...

to do. Sticking random low OPPs

The values I use are not randomly chosen but based on CMOS design assumptions.

like it's done now is irresponsible and is very likely to give wrong test results, making the test useless.

I don't see any wrong test results in this case. I have seen such rcu stalls only happen if the voltage is close to its lower limit and that lower voltage limit is what I wanted to empirically find out for this board. For that kind of test it doesn't matter how the OMAP5 freezes or stumbles. I was just puzzled that it could be partially freezing. So my cautious conclusion is that is not working in this situation as well and my table was marked orange or red.

And in all my tests done so far it does not make any difference to include or exclude the lower frequencies. We had freezes before we even tried lower frequencies and still have them with.

Rather, all the test results show me that we have a problem at high voltages and trying to run higher CPU clock requires to raise voltage to a point where the overvoltage freezes occur. Never with rcu stall or other messages. They simply stop every sign of activity.

My question was to understand if the rcu stall message at low voltages gives us some more hints. Obviously it does not and that is all answers I am looking for.

BTW: I just finished booting the EVM with 1GHz + 1.5GHz @ 1.25V and it works without noticeable problems at overvoltage. So it can run at OPPs completely outside the green range for the Pyra CPU.

And I was able to estimate (by using the B2B connectors as a shunt resistor) total current consumption between:

bash-idle 100% bash-loop 200% neon-loop 270%

So this means the OMAP draws 2.7 times as much Amperes when running neon-loop than in bash-idle (if run with a single frequency OPP list: 1GHz).

I noticed that there is also a small temperature effect (+20% at 100° compared to 50°) but it is small compared to the code or overall OPP influence.

So I totally agree that low frequency/voltage OPPs must be verified to work if we want to reduce power demand.

Since estimates indicate that we could be able to reduce power demand by factor 3 to 7 this is something I think we *must* try to achieve for a portable device intended for daily use.

But not now.

BR and thanks, Nikolaus

Hi,

Am 27.11.2016 um 23:47 schrieb Grazvydas Ignotas notasas@gmail.com:

On Sun, Nov 27, 2016 at 7:08 PM, H. Nikolaus Schaller hns@goldelico.com wrote:

...

Hi,

...

Am 27.11.2016 um 15:11 schrieb Grazvydas Ignotas notasas@gmail.com:

If it doesn't hang it doesn't mean it works. The low frequencies are usually entered when CPU is idle and doing nothing, so the problems are unlikely to trigger, but then when you do a test that constantly thermal throttles (like the one you recently posted), the CPUs run at 100% at a low unapproved OPP and is quote likely they'll fail there instead of the high frequency, and you then misinterpret the result thinking it failed at the high frequency.

There was no detectable coincidence of failure and thermal throttling to occur or not.

And it disappears by raising the voltage a little (at exactly the same set of low clock frequencies and temperatures).

So the only influencing factor was the voltage at high frequency OPPs and not the presence of low frequencies.

Ok I'll just trust you on that...

...

Of course that is just an observation without a warranty from TI.

...

Sure, saving power and reducing temperatures is great, by we should do that optimization after we know the SoC works in conditions specified by TI. As we don't have low OPP data from TI (like we had for OMAP3), we'll need careful tests that do lots of calculations without stalling the CPUs and verifies the calculation results and run them for a long time on lots of boards forced to those low OPPs to prove they indeed work. This is all challenging

and time consuming...

I've updated neon_loop to check it's calculations, if it detects an error it will now abort. I've tried to reduce number of checks to avoid stalling the processor by carefully selecting instructions in such a way that if one calculation fails, the error would propagate to the others and would be detected later. This is a different workload now and I don't know if it stresses the CPU as much as the earlier one, but maybe it'll be able to detect problems before something bad happens to the kernel, and you don't have to wonder if you've hit a kernel bug, or a CPU issue.

I have found a little time to do some tests (frequency range 250MHz ... 1250MHz) to estimate current demand (and stress):

bash-idle 100% neon-loop 310% ... 373% (as temperature rises) neon-loop2 336% ... 390% (as temperature rises)

So the new loop is stressing even more.

And now I got freezes: * before 100° was reached, i.e. processor continuously running at 1250MHz * no message printed * current consumption in freeze state ca. 205%

This all fits to the theory that the switching noise inside the OMAP becomes too big (at high frequencies, high load and high voltage).

BR, Nikolaus

On 23 November 2016 at 11:45, Grazvydas Ignotas notasas@gmail.com wrote:

OTOH now that I think about it, it might actually start doing writes after a task switch because ARM doesn't write-allocate cache.

Eh what? Yes it does. In fact the cortex-A15 does not support a no-write-allocate policy at all.

On 28 November 2016 at 00:45, Grazvydas Ignotas notasas@gmail.com wrote:

write RAM, even on cache hit) but you can still have no-allocate.

No-allocate is not the same thing as no-write-allocate, it means *never* allocate in cache, not even on read. It's a slightly obscure cache policy and only available when using LPAE since the short descriptor format has no encoding for it.

(The LPAE long descriptor format gives you complete freedom over allocation hint, you can even declare memory to be something crazy like write-through write-allocate no-read-allocate. Of course just because you can ask for it doesn't mean you'll get it.)

From what I see in the manual you can't have write-though

Correct, it treats write-through as non-cacheable. Quite annoying.

Overall it only supports: - strongly-ordered - device - normal, non-cacheable - normal, write-back, no-allocate - normal, write-back, read-write-allocate and I think the only reason it supports no-allocate at all is because it's also implicitly used when you "disable" the caches (which only suppresses allocation) as necessary step of powering down a core.

Add to this the fact it essentially uses a linefill for normal non-cacheable reads (which is what tripped up TILER) and it becomes clear they really wanted to minimize the number of different types of memory transactions used, presumably because it's the first processor with ACE support.

Matthijs

Hi,

Am 14.11.2016 um 04:11 schrieb Michael Mrozek EvilDragon@openpandora.org:

Hi,

I've got some more questions.

The board I used for testing the didn't have a display board or LCD cable attached.

Sometimes, I got the Atomic Timeout errors from omapdrm and MIPI initializing but finding no device (even running X Server) while sometimes these errors don't appear and X simply doesn't find a screen.

Any idea what that is and how this happens?

That is likely a race condition, i.e. something assumes that some other thing is already initialized.

I think those atomic timeouts are no longer in effect when we have the non-rotated display boards. Then we can use the proper panel driver module and no longer need the mipi-panel-debug driver (which triggers those Atomic Timeout errors).

I also always see: palmas-usb 48070000.i2c:palmas@48:palmas_usb: failed to get id gpio

Yes, I have this as well.

Could that be related to our power issue or does this simply mean I've only have an AC power adaptor connected to the USB port?

Neither. It is about the USB ID pin (auto-detection of an OTG cable). It has been introduced here:

https://lkml.org/lkml/2015/7/13/319

I am not sure why it fails, but it likely is spurious from deferred probing until it succeeds.

BR, Nikolaus

PS: I had quite some problems booting the Pyra board - until I found out that I still had configured EARLY_PRINTK for OMAP3...

Am 10.11.2016 um 17:23 schrieb Grazvydas Ignotas notasas@gmail.com:

On Thu, Nov 10, 2016 at 5:20 AM, Michael Mrozek EvilDragon@openpandora.org wrote:

...

I can't do much more than thinking though. I have no clue how to change the voltages and compile the board files on order to test it with my SD Card.

I can set up the environment on your server where you can build your own device tree (.dtb) files if you like. Changing the voltages is as easy as editing operating-points entry in arch/arm/boot/dts/omap5.dtsi .

Just if someone reading this list wants to try on local computer, it is (assuming some ARM gcc):

git clone http://git.goldelico.com/gta04-kernel.git cd gta04-kernel git checkout letux-4.9-rc4

vi arch/arm/boot/dts/omap5.dtsi

make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- letux_defconfig uImage LOADADDR=0x80008000 dtbs INSTALL_MOD_PATH=/tmp/kernel-modules-$$ modules modules_install cp arch/arm/boot/uImage modules arch/arm/boot/dts/omap*.dtb /mount/sdcard/boot/

or simplified for DT only

make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- letux_defconfig LOADADDR=0x80008000 dtbs cp arch/arm/boot/dts/omap5*.dtb /mount/sdcard/boot/

Description:

http://projects.goldelico.com/p/gta04-kernel/

Am Thu, 10 Nov 2016 18:23:20 +0200 hat Grazvydas Ignotas notasas@gmail.com geschrieben:

Hi,

Server movement complete, but I haven't checked everything yet (except for mail and webstuff).

You should be able to set it up now and let me know if anything is missing :)

On Thu, Nov 10, 2016 at 5:20 AM, Michael Mrozek EvilDragon@openpandora.org wrote:

...

I can't do much more than thinking though. I have no clue how to change the voltages and compile the board files on order to test it with my SD Card.

I can set up the environment on your server where you can build your own device tree (.dtb) files if you like. Changing the voltages is as easy as editing operating-points entry in arch/arm/boot/dts/omap5.dtsi .

Gražvydas

Am 09.11.2016 um 22:40 schrieb Matthijs van Duin matthijsvanduin@gmail.com:

On 9 November 2016 at 18:43, H. Nikolaus Schaller hns@goldelico.com wrote:

...

"Use a “Power/Gnd pad/pin to via” ratio of 1:1 whenever possible. Do not exceed 2:1 ratio for small number of vias within restricted PCB areas (i.e. underneath BGA components)."

Maybe someone could help to translate this...

I'm quite sure they're talking about the ratio of the number of power/gnd pads to the number of vias used to connect those pads to the power/gnd plane.

Well, this would mean we need approx. 10-15 Vias to connect.

Am Wed, 9 Nov 2016 14:41:20 +0100 hat Matthijs van Duin matthijsvanduin@gmail.com geschrieben:

Hi,

...

• we need a SD shaped hardware adapter to get the JTAG signals to a

probe

This, however, is the tricky part. It would have to be custom made since the (already obscure) NIDnT interface is based on micro-SD. maybe with level-shifters (I don't know)

AFAIK, we still have the possibility to solder a Micro-SD-Slot directly to the CPU board, so that should work.

Otherwise, if custom made, it should be easy to rip an SD->Micro-SD adapter apart and solder a small PCB to the traces.

Am 09.11.2016 um 15:05 schrieb H. Nikolaus Schaller hns@goldelico.com:

...

Am 09.11.2016 um 14:49 schrieb Michael Mrozek EvilDragon@openpandora.org:

Am Wed, 9 Nov 2016 14:41:20 +0100 hat Matthijs van Duin matthijsvanduin@gmail.com geschrieben:

Hi,

...

...

• we need a SD shaped hardware adapter to get the JTAG signals to a

probe

This, however, is the tricky part. It would have to be custom made since the (already obscure) NIDnT interface is based on micro-SD. maybe with level-shifters (I don't know)

AFAIK, we still have the possibility to solder a Micro-SD-Slot directly to the CPU board, so that should work.

Ah, you are right! I have been working on that for a too long time to remember such details :)

There are 8 unused solder pads on the battery oriented side of the CPU board where exactly those shared pins are available we need :)

...

Otherwise, if custom made, it should be easy to rip an SD->Micro-SD adapter apart and solder a small PCB to the traces.

the benefit of building an SD-card shaped JTAG adapter is that it can be applied to any Pyra without opening it.

Something like

http://www.camerahacker.com/Digital/Inside_miniSD_Adapter.shtml

A problem is that these contact pins are not very stable unless sealed inside the adapter.

I would simply make an 1.6mm PCB having the standard SD card shape and contact pads at the right locations. Similar to this:

http://downloads.qi-hardware.com/people/werner/wpan/web/800px-Atben_atusb_pr...

BR, Nikolaus

---

Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel