On Mo, 2017-09-11 at 20:32 -0700, Tony Lindgren wrote:

Well, sadly, I couldn't test it yet (as I couldn't boot a single PCB without battery, see other mailing list post), but I took the time to browse through the documents to find some clues about why that should be working...

Here are some things I found. I'm not sure how much you all know about that already (and I'm definitely not that knowledgable), but maybe there's something in there that gives you a clue what it could be :)

So, DLL is the internal clock of the memory and it MUST be enabled on normal operation (that's what the JEDEC document says).

What's different between DLL on and DLL off?

1. The maximum clock frequency for DLL-off Mode is specified by the parameter tCKDLL_OFF. There is no minimum frequency limit besides the need to satisfy the refresh interval, tREFI. (JESD79-3F, chapter 4.5)

--> What frequency is tCKDLL_OFF in our setup? And what is it when it's switched on?

2. Due to latency counter and timing restrictions, only one value of CAS Latency (CL) in MR0 and CAS Write Latency (CWL) in MR2 are supported. The DLL-off mode is only required to support setting of both CL=6 and CWL=6. (JESD79-3F, chapter 4.5)

--> So CL and CWL are the same and different to the setup in DLL on?

3. DLL-off mode will affect the Read data Clock to Data Strobe relationship (tDQSCK), but not the Data Strobe to Data relationship (tDQSQ, tQH). Special attention is needed to line up Read data to controller time domain. Comparing with DLL-on mode, where tDQSCK starts from the rising clock edge (AL+CL) cycles after the Read command, the DLL-off mode tDQSCK starts (AL+CL - 1) cycles after the read command. Another difference is that tDQSCK may not be small compared to tCK (it might even be larger than tCK) and the difference between tDQSCKmin and tDQSCKmax is significantly larger than in DLL-on mode. tDQSCK(DLL_off) values are vendor specific.

--> So tDQSCK is different when DLL if off.

4. With DLL off, the internal frequency is disabled. What is the frequency we run the memory with when DLL is off?

5. DLL off also disables ODT. Could it be we have issues with ODT? Improper setup? It's also possible to disable ODT - does that make the memory work when DLL is on?

6. One thing that still puzzles me: According to the information I found, the timings need to be setup SPECIFICALLY for the memory chips you use. Even the JEDEC standard mentions sometimes that the specific timings need to be taken from the datasheet of the manufacturer.

So if we HAVEN'T specifically set the timings according to our datasheet but used different ones, it might or might now work. The datasheet of the memory chip has tons of information about timing setup, and I even sent a document out on this mailing list about a week ago which automatically spits out the EMIF Hex Values once all values have been correctly entered. Are we really using the timings specified in our memory chip datasheet or are we using some random ones?

Because: "The functionality described and the timing specifications included in this data sheet are for the DLL Enabled mode of operation."

So the timings apparently are different with DLL is disabled - maybe that's the reason it works for us?

One interesting thing I noticed: It seems DLL can be switched on and off anytime. What happens if we switch DLL on again after the system has fully booted up?

And about the hardware leveling: I still find it a bit weird that it shouldn't work until you let the system cool down. Especially if cooling down always takes around 10 seconds, regardless whether the system has run for 5 seconds or 5 minutes (that's the case right now, if I understood you correctly?)

Are we sure there's NOTHING that simply powers a part of the Pyra for 10 seconds after it has been switched off? According to the OMAP5 document, hardware levelling must be triggered by the software, so what happens if we never trigger hardware levelling?

• Michael Mrozek EvilDragon@openpandora.org [170910 11:58]:

...

On Do, 2017-09-07 at 18:41 +0200, H. Nikolaus Schaller wrote:

Hi,

I'm now on vacation until Thursday and took various 4GB Boards and Mainboards with me.

If you want to me to do any tests and dump some output of the results, please send me some binary files for that.

I can easily mix all PCBs and make a table with various results.

OK here's some vacation fun for you:

http://muru.com/linux/pyra/

Regards,

Tony

On Mi, 2017-09-13 at 08:40 +0200, Matthijs van Duin wrote:

Hi,

Not quite... a DLL (Delay Locked Loop) is a way to create fine- grained timing from a reference clock signal. The DLL inside the memory locks onto the bus clock (ck-nck) provided by the memory controller and is used to make its outputs line up right to the bus clock. Specifically, this affects the alignment (relative to bus clock) of:

1. memory's dq (data) and dqs outputs (data clock) during reads;
2. changes to its termination resistors in response to the ODT signal

(synchronous ODT) and/or during writes (dynamic ODT).

When the memory DLL is on, the alignment of these relative to the bus clock is tightly specified by jedec. When the DLL is off, they no longer are.

For some reason JEDEC also forbids use of ODT whatsoever in DLL-off mode. I have no idea why. I would have expected that ODT in DLL-off mode simply follows the "asynchronous ODT" rules which also apply when the memory is in self-refresh (at which time the DLL is implicitly disabled).

Note btw that although JEDEC specifies various aspects of DLL-off mode, is entirely optional for memory to support DLL-off mode in the first place.

Thanks for the detailed explanation. Hmm. But then it doesn't really sound like we have noise-issues - these should also have an effect when the internal timing is disabled, as both relay on the same reference clock.

...

What frequency is tCKDLL_OFF in our setup?

JEDEC specifies tCK(DLL_OFF)(min) to be 8ns, i.e. the max clock frequency specified for DLL-off mode is 125 MHz.

...

And what is it when it's switched on?

We're running the memory bus at 532 MHz, i.e. tCK = 1.88 ns.

Ah - so the memory is running faster when DLL is enabled?

Running with DLL off at that frequency is therefore waay outside JEDEC-specified territory.

Well, yes and no. JEDEC specifies in the documentation that the timings are different when DLL is off and also specifies that DLL should always been on.

So the timings specified by JEDEC are DLL On timings.

--> So CL and CWL are the same and different to the setup in DLL

...

on?

Tony only toggled the DLL-off bit, he didn't change any timings. Since the memory is still being run at high frequency, using the normal CL and CWL values is the right thing to do.

Well, now I'm confused... I thought with DLL off it's running with 125MHz and with DLL on it's 532MHz?

Or are we actually running with 532MHz and DLL Off?

...

5. DLL off also disables ODT.

There's nothing saying it disables ODT. Rather, JEDEC says you're supposed to refrain from using ODT in DLL-off mode. Using it anyway would be undefined behaviour, but my suspicion would be that it behaves like asynchronous ODT in practice.

...

6. One thing that still puzzles me: According to the information I

found, the timings need to be setup SPECIFICALLY for the memory chips you use. Even the JEDEC standard mentions sometimes that the specific timings need to be taken from the datasheet of the manufacturer.

Where exactly? I'm pretty sure this isn't true, and it would completely defeat the purpose of standardized speed bins.

Well, the TI information I sent two weeks ago says that you need to change the EMIF depending on the exact specifications found in the datasheet of the memory chip you use.

The JEDEC Standard documentation says in chapter 3.4.2:

"The mode register MR0 stores the data for controlling various operating modes of DDR3 SDRAM. It controls burst length, read burst type, CAS latency, test mode, DLL reset, WR and DLL control for precharge Power-Down, which include various vendor specific options to make DDR3 SDRAM useful for various applications."

That also sounds like they have some standards but fine-tuning settings need to be gotten from the vendors datasheet. I'm not sure WHAT they mean though.

I just checked: There ARE standard speed bins in the JEDEC document as well - but they are identical to the speed bins listed in the Samsung memory chip.

However, JEDEC has multiple timings for DDR3-1066 (E, F, G, F(Optional), and so on), which are all different.

The Samsung datasheet only has ONE DDR3-1066, which seems to be DDR3- 1066F.

Are those the timings we're using? Or could it be we're using the ones from DDR3-1066E or G, for example?

...

Are we sure there's NOTHING that simply powers a part of the Pyra for 10 seconds after it has been switched off?

Pretty sure yes.

Hm, but the battery chip ALSO seems to remember some stuff. You can't boot with AC Adaptor only and without battery for the first time, but it seems it works when you have at least booted once with battery.

I will test this further, but that's what aTc mentioned. And I think you can even fully remove the power for a while and it keeps that setting for a few seconds.

Could it be the PALMAS (or we?) do have some supercap included that keeps the settings for a few settings (so you can replace the battery, for example)?

...

According to the OMAP5 document, hardware levelling must be triggered by the software, so what happens if we never trigger hardware levelling?

It would use the reset default values and fail spectacularly. Using software-provided leveling values should be possible though. Here's what I said about this on irc a few days ago:

<@zmatt> notaz/tmlind: btw, in case you want to try manual leveling... I think the procedure is: <@zmatt> 1. set the appropriate values in ext phy control 2-13 ( https://goo.gl/bDV8js ) instead of the seed values being written currently. you should be able to use the values obtained from a successful hw leveling. <@zmatt> 2. skip the hw leveling procedure (obviously) <@zmatt> and the step 3 that I was missing when I tried this a while back: <@zmatt> 3. reset the phy to latch the config registers into the phy. This is done by setting bit 10 of the "iodft_tlgc" register at offset 0x60. You can find it in the dra7 trm, it's omitted from the omap5 trm.

Ah, okay. Hs anyone already tried that?

At 08:40 +0200 on Wednesday 2017-09-13, Matthijs van Duin wrote:

We're running the memory bus at 532 MHz, i.e. tCK = 1.88 ns.

At 06:03 +0200 on Monday 2017-09-18, Matthijs van Duin wrote:

DDR3-1066 only has E (CL=6), F (CL=7), and G (CL=8) as JEDEC-specified bins.

The timings are often also specified as 6-6-6 / 7-7-7 / 8-8-8, which refer to CL, nRCD and nRP.

At 06:03 +0200 on Monday 2017-09-18, Matthijs van Duin wrote:

For our memory that's the DDR3-1066F bin, which means it supports 7-7-7 timings (which we're using).​

Does the memory bus clock need to run so fast, so close to the maximum for the bin?

With CL = 7 the range is (400, 533 1/3] MHz according to the K4B8G1646D datasheet. 532 MHz seems very close to the top edge.

What happens if you back it off 5% from the bin edge and run the clock at 507 MHz or so?

Or, alternatively, kick the clock up to around 561 MHz (if the board / processor will support that) and use the next bin (533 1/3, 666 2/3] MHz with CL = 9? (Although, according to Matthijs's post, that's not JEDEC standardised.)

N.

On Mo, 2017-09-18 at 21:11 +0200, Matthijs van Duin wrote:

Well, sure, it wouldn't be the solution, but might be interesting to see if it works.

I still find it weird though that it probably works with DLL off but full speed :O

...

What happens if you back it off 5% from the bin edge and run the clock at 507 MHz or so?

The memory is binned for CL=7 @ 533 MHz, which means the manufacturer guarantees those timings. I'm also fairly sure that CL is only relevant for timing closure internal to the memory, hence increasing it won't fix issues with the interfacing between memory and omap.

Of course lowering the clock frequency could nevertheless be beneficial since gives all nets (both signal and power) more time to stabilize again after a transition. Doing so however would require lowering the frequency of the core PLL and thereby all clocks that derive from it (which include all interconnects and the GPU), which would be annoying from a performance point of view.

Matthijs _______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel

Am Tue, 19 Sep 2017 11:05:40 +0200 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:

Well, it already tells us there's no general problem with using Samsung memory, our timings should be correct (well, we are sticking to JEDEC, so these should work).

So the main guess would be noise right now - though we have no idea why it works when DLL is set to off.

Maybe the DLL within the memory chip is more responsive to noise than the signal integrity, etc.?

That would be an explanation.

If we can get the RAM to work 100% fine without any issues with DLL off, I'm fine with that for the first 500 CPU boards... but nevertheless, we should look for the reason what's going wrong.

We don't a bad surprise in the future if a different batch of RAM chips won't work anymore even with the DLL off.

First results from powering up the reworked CPU boards:

• Samsung K4B4G works equally well with the same 2GB EMIF setup we

use for the IMG 2GB boards.

• IM 8G16 board does not boot - but there is a short circuit on

VSYS :( I.e. rework has damaged the board and we can't evaluate the memory chips yet.

I'll check if we can get that repaired (swapping Palmas for example could help, or transplanting the memory chips to yet another CPU board).

Unfortunately, this doesn't help us much.

BR, Nikolaus

...

Am 19.09.2017 um 01:19 schrieb Michael Mrozek EvilDragon@openpandora.org:

On Mo, 2017-09-18 at 21:11 +0200, Matthijs van Duin wrote:

Well, sure, it wouldn't be the solution, but might be interesting to see if it works.

I still find it weird though that it probably works with DLL off but full speed :O

...

...

What happens if you back it off 5% from the bin edge and run the clock at 507 MHz or so?

The memory is binned for CL=7 @ 533 MHz, which means the manufacturer guarantees those timings. I'm also fairly sure that CL is only relevant for timing closure internal to the memory, hence increasing it won't fix issues with the interfacing between memory and omap.

Of course lowering the clock frequency could nevertheless be beneficial since gives all nets (both signal and power) more time to stabilize again after a transition. Doing so however would require lowering the frequency of the core PLL and thereby all clocks that derive from it (which include all interconnects and the GPU), which would be annoying from a performance point of view.

Matthijs _______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel

-- Mit freundlichen Grüßen,

Michael Mrozek

---

OpenPandora GmbH Geschäftsführer: Michael Mrozek

Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt

---

eMail: mrozek@openpandora.org _______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel

Am Mittwoch, den 20.09.2017, 11:52 +0300 schrieb Grazvydas Ignotas:

I'm not sure - I think they only tested the patch, not his binaries. I've got multiple boards and binaries and want to test those too.

On Tue, Sep 19, 2017 at 4:47 PM, Michael Mrozek EvilDragon@openpandora.org wrote:

...

Am Tue, 19 Sep 2017 11:05:40 +0200 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:

Well, it already tells us there's no general problem with using Samsung memory, our timings should be correct (well, we are sticking to JEDEC, so these should work).

So the main guess would be noise right now - though we have no idea why it works when DLL is set to off.

From what I understood DLL off doesn't work for Nikolaus (and Matt?), only helps on Tony's board, so DLL-off should be considered non-working also.

Gražvydas

I did test Tony's binaries on four different PCBs.

Always with the same result:

U-Boot SPL 2016.11-00247-g97ece0bff7 (Sep 11 2017 - 20:25:07) OMAP5432-GP ES2.0 SDRAM: identified size not same as expected size identified: 4 expected: 40000000

...

spl:board_init_r()

spl_init() Trying to boot from MMC1 WARNING: DDR RAM is not working! (12340078) Found LC15 V5.1 board_mmc_init called (vers = 51) MLO hard: gpio7 = eMMC soft: gpio82 = 0 (mmc1=eMMC) ctrl: gpio76 = 1 (ctrl=soft) MMC Device 0 not found spl: could not find mmc device. error: -19 SPL: failed to boot from all boot devices ### ERROR ### Please RESET the board ###ä

The number in the brackets behind DDR RAM is not working! is different with each PCB but seems to be consistent for each one.

Is that the issue you had as well, Nikolaus?

One more idea: Tony, as it worked for you with either 1.39V (or was it 1.42V?) OR DLL off, maybe both have a positive effect?

What about a build with 1.39V with DLL off?

Could you please provide a build with that?

Am Mon, 2 Oct 2017 06:45:48 -0700 hat Tony Lindgren tony@atomide.com geschrieben:

Yes, but without updated kernel... so I guess I was really running at 1.35V with DLL off (if the kernel lowered the voltage).

I need to test that.

One thing to notice: I didn't have ANY issues with rebooting. It booted fine every time I tried, no failure, regardless how warm it was.

If U-Boot starts with 1.5V, then hardware levelling is done with 1.5V, right? The kernel switches to 1.35V after that... maybe that's the trick?

I'll try with various boards and combinations!

• Michael Mrozek EvilDragon@openpandora.org [171001 16:25]:

...

Tony, as it worked for you with either 1.39V (or was it 1.42V?) OR DLL off, maybe both have a positive effect?

What about a build with 1.39V with DLL off?

Could you please provide a build with that?

Sounds like you got that from notaz already, see my other reply though for things to test.

Regards,

Tony

* Michael Mrozek EvilDragon@openpandora.org [171003 09:32]:

On Di, 2017-10-03 at 12:27 +0200, Michael Mrozek wrote:

Hi,

indeed, whereas U-Boot sets it to 1.5V, the kernel switches back to 1.35V: [ 2.780669] smps6: Bringing 1500000uV into 1350000-1350000uV

So my Pyra was running over 24 hours with 4GB RAM at 1.35V (until I shut it down).

It was just idling though, so no extensive tests, and it still was running with DLL off (haven't done any other test yet).

But as it was booting up reliably.

Switching the memory voltage after booting can be glitchy and there could be issues coming up from suspend etc. So best to stick to whatever voltage makes the voltage measured at the memory chips be the correct 1.35V.

And I'd leave out the DLL off hack too because we're running things outside the specs. If it's really the voltage drop then it should not be needed. And there might be also some performance issues without the DLL.

Regards,

Tony

* H. Nikolaus Schaller hns@goldelico.com [170907 09:41]:

...

Am 07.09.2017 um 17:41 schrieb Tony Lindgren tony@atomide.com: What's the u-boot commit you're using as base? I can try with that at some point later on my time today.

http://git.goldelico.com/?p=gta04-uboot.git;a=shortlog;h=refs/heads/letux-20...

This one almost always works in U-Boot for me but usually hangs in kernel after 3-7 seconds.

I'll try your u-boot at some point.. Meanwhile, can you please check that you're testing the right updated u-boot?

What was happening to me at some point was I had u-boot in the raw eMMC unused area from install.sh, then changed the partitions to boot MLO from vfat, but ended up still booting the old u-boot in the raw eMMC unused area until I reset that data with dd..

Regards,

Tony

* H. Nikolaus Schaller hns@goldelico.com [170907 09:41]:

...

Am 07.09.2017 um 17:41 schrieb Tony Lindgren tony@atomide.com:

• H. Nikolaus Schaller hns@goldelico.com [170907 04:23]:

...

Hi,

...

Am 07.09.2017 um 13:15 schrieb Michael Mrozek EvilDragon@openpandora.org:

Am Wed, 6 Sep 2017 22:05:38 -0700 hat Tony Lindgren tony@atomide.com geschrieben:

Hi,

nice! Is that with 1,35V as well?

for me it does not work (at least for 1.35V). U-Boot complains about non-existing RAM with this change right from the first power-on at room temperature.

Works for me at 1.35V, except u-boot is having hard time starting when host.

...

...

The interesting thing is about the cool down... I wonder if that works better when we have the cooling applied.

Maybe I should put it into a fridge to get U-Boot starting?

What's the u-boot commit you're using as base? I can try with that at some point later on my time today.

http://git.goldelico.com/?p=gta04-uboot.git;a=shortlog;h=refs/heads/letux-20...

This one almost always works in U-Boot for me but usually hangs in kernel after 3-7 seconds.

That base works for me with the internal dll hack, I uploaded the pyra-4g-u-boot-2017-09-11.tar.gz binary to:

http://muru.com/linux/pyra/

Maybe you did not have the DUAL_RANK option enabled?

Regards,

Tony

Just a quick update:

notaz sent me a U-Boot version that runs the 4GB RAM with DLL off and 1.5V.

That seems to work fine. I was able to boot multiple times without any issues (so no heat or similar).

The unit I tested is now running for 2.5h. It's only idling around so far, but I'll try some more stuff like memtester, etc. I also need to test this with the other boards, and I also need to test 1.5V only (with DLL enabled).

Just the DLL Off version with 1.35V didn't work for me.

So yes, raising the voltage seems to do the trick. As Juliano simulated, we have a pretty high voltage drop (from 1.5V to 1.1V on some balls), and that would certainly explain that.

It really looks like our timings are fine and the noise / voltage drop is the main issue.

So what can we do here?

1. Try to improve the board design as much as possible. It seems we're not THAT far away from getting it to work under normal conditions, so some small changes (additional vias, some capacitors) could be enough.

2. Try to find the minimum voltage setting with DLL off that works with all 4GB RAM PCBs. If 1.4V would work reliable, that would at least be a workaround for the first batch. We should still try to improve the PCB though.