Looking for help: U-Boot RAM Timings

[EvilDragon]

Okay, this is one of the most critical issues I'm having to deal with at the moment:
The RAM Timing / Calibration / Setup in Pyra's U-Boot.

What's the problem?

Multiple CPU boards have the issue that they don't boot at all, only sometimes or most of the times.
The problem is that with these units, one bit of the RAM reports a faulty bit, and as the OMAP5 checks the RAM right after the calibration, it fails to boot.
Our guess is that something with the calibration goes wrong and the RAM is not properly setup.

Dave Shaw made a version that works on MOST Pyras, but not all of them, so I still think it's something related to the timing setup.
https://dev.pyra-handheld.com/daveshah/pyra-uboot/-/branches (pyra-alt-ddr3)

notaz made a nice patch for U-Boot where I can set up the RAM timings in a more comfortable way. But just trying to poke around at these settings probably isn't helpful unless you know what you're doing

So basically, I'm looking for help here.

1. Is there someone who is knowledgeable about RAM timings / calibration, etc. who could help me to figure out what values to try out?
2. The last time I played around with the timings, I didn't really see any change (not even if I set them way out of bounds). So I'm not sure they are even correctly applied / being used. Can someone help me to work on U-Boot and make sure these are applied? Maybe even show the currently setup RAM timings in U-Boot when booting up, which would be helpful for debugging

Thanks in advance

 

[fantomid]

@EvilDragon know some guys from Bootlin in Toulouse
Maybe, I could ask them for help ?

 

[EvilDragon]

That would be great! I can already compile the U-Boot and change the timings, but it would be great having the support of someone who knows what he's doing!

 

[TeDaDeS]

Looks like a really specific issue. Hope anyone with experience can help.

Maybe it's good to post some more details about the hardware here as the issue is probably very specific for the SoC and RAM.
Soc= OMAP5432
RAM= Dual Channel PC DDR3L 532MHz

 

[EvilDragon]

Looks like a really specific issue. Hope anyone with experience can help.

Maybe it's good to post some more details about the hardware here as the issue is probably very specific for the SoC and RAM.
Soc= OMAP5432
RAM= Dual Channel PC DDR3L 532MHz

Of course, but most of the specsheets are NDA. I am allowed to give them out to the user who wants to help, but I cannot make them public.
So once someone wants to take a look at them, I can give the detailled information

 

[hns]

Looks like a really specific issue. Hope anyone with experience can help.

Yes, indeed, it is very specific to the Pyra and the mix of the OMAP5432 and the DDR3L.
Someone more experienced with OMAP5 (or its successors) can certainly help.

According to my last information (but it may have changed) DRAM chips are: Samsung K4B8G1646D-MMK0. It could be helpful or not to know that we initially built some 2GB RAM units (MT41K256M16HA-125) where nobody observed this effect (although it might have been there as well and the units weren't tested well enough).

The SoC module involved is called "EMIF" (external memory interface). The issues are somewhere around

• Output impedance (ZQ) calibration
• Programmable SDRAM timing parameters
• Write/read leveling/calibration and data eye training for DDR3

AFAIR there are public versions of interest:

swpu249x TRM (public version)
swpz037d OMAP543x Errata (public version)

but they still contain an EAR export restriction notice (not clear to me if it is for the chips only or includes this documentation).

IMHO also noteworthy: it is neither a Linux nor an U-Boot issue. Would happen with any code that does really initial processor setup. It just happens that U-Boot is doing it. And ED can change that to any optimal setting. We "just™" need to figure out which one is better and more universal than the current one

 

[hns]

Here is additional background information on the issue: https://pyra-handheld.com/boards/threads/analyzing-4gb-ram.81687/

 

[matzesu]

Werent the Germans good whit U-Boot back then ??

Sorry couldn’t resist ..

 

[Djoga'Ro]

Werent the Germans good whit U-Boot back then ??

Go ahead then, fix it

 

[EvilDragon]

And some more information:

* The OMAP5 does some kind of automatic calibration, but it's unknown what exactly it does. There seems to be a way to disable it and manually do that, so that might be something to look at.
* notaz made a U-Boot for me where I can change the RAM timings. However, after booting, when I check the RAM timings, it seems that these timings are not applied, as they don't change when I change the values.

So there might also be something wrong in how these timings are applied - or something changes these timings back after they had been applied. So that's also something that needs to be checked.

 

[pyrat]

Werent the Germans good whit U-Boot back then ??

Sorry couldn’t resist ..

I'm not sure where I got it from, but I associated U-Boot with the (Bavarian?) company Denx.de .So, yes Germans are still good with U-Boot.

 

[matzesu]

Not really it was more like a Joke as "U-Boot" is the German Short for "Unterseeboot", or Submarine ^^
"Wir leben alle in einem Gelben Uboot" ^^

 

[Git1]

https://de.wikipedia.org/wiki/U-Boot

 

[Confuzzled]

Not really it was more like a Joke as "U-Boot" is the German Short for "Unterseeboot", or Submarine ^^
"Wir leben alle in einem Gelben Uboot" ^^

If you have not read the book, or seen the film 'Das Boot', I'd recommend it.

 

[pimaster]

Just a note to say I am trying.
After reading "15.3.6.2 EMIF Registers" of the OMAP documentation it became clear that I would need to find out what the performance of the memory was. Attached is the best documentation I could find. When I got to the end I was a little worried that 533mhz memory wasn't included but that documentation is defined in data rate, and since it is "double data rate", 1066 was in there.
Annoyingly, CPU doc is configured in clocks. Memory doc is specified in ns.
I was a little excited to see that we only had 7 pico seconds of grace time for some commands (1 tick is 1.8 nano seconds), so I padded that timing out a little. My board still booted, so I was excited that I didn't break anything. But I was still having some boot issues later whilst playing around with other settings (Trying to boot from SD card, but load contents from internal mmc)
I have read in the documentation that there is an impedance training configuration and I was going to look at that after double checking all of the memory timing variables.
EvilDragon is a little suspicious that there might be other processes that change this timing variables (which part of me doubts, because that might cause corrupted read/writes for a couple of cycles) but I'm happy to go looking for confirmation. So a trip into devmem2 I will go.

Motivation got hiccuped this week because I somehow blew the partition table on the pyra (which took a few days to recover) then had to do repairs on my monitor. Also fear a cold coming on.

But it was nice to pick up an extra Dev board from Wally (last week). That was a nice motivation boost.

 

[pimaster]

I wrote a script to interrogate the memory after boot.

#!/bin/bash

focusAdd=$1

function readMem {
        add=$1
        name=$2
        val=`devmem2 $add | grep -Po "(?<=: )0x.*"`
        valI=$(( $val * 1))
        valB=`echo "obase=2;$valI" | bc`
        bits=("${@:3}")
        count=${#bits[@]}
        echo $name @ $add = $val
        if [ -n $focusAdd ] && [ "$focusAdd" == "$add" ]; then
                echo $valB is the bit value of the register.
        fi
        for (( i = 0; $i < ${count}; i += 2 ))
        do
                name=${bits[$i]}
                split=(${bits[$i + 1]//:/ })
                size=${split[0]}
                pos=${split[1]}
                mask=$(( ( 1 << $size ) - 1 << $pos ))
                maskS=`echo "obase=2;$mask" | bc`
                bitV=$(( ($val & $mask) >> $pos ))
                if [ -z $focusAdd ]; then
                        echo -ne "$name: $bitV\t"
                elif [ "$focusAdd" == "$add" ]; then
                        echo `printf %32s $maskS | tr ' ' 0` $name = $bitV
                fi
        done;
        if [ -z $focusAdd ]; then
                echo ""
        fi
}
readMem 0x4c000000 EMIF_rev revision 32:0
readMem 0x4c000004 EMIF_Status be 1:31 dual_clk 1:30 fast_init 1:29 rdlgate 1:6 rdlto 1:5 wrlto 1:4 phy_dll 1:2
readMem 0x4c000008 EMIF_config type 3:29 pos 2:27 ddr3_termR 3:24 ddr2_ddqs 1:23 ddr3_odt 2:21 ddr_disableDll 1:20 driveStre 2:18 cwl 2:16 narrowMode 2:14 cl 4:10 rowsize 3:7 ibank 3:4 ebank 1:3 pageS 3:0
readMem 0x4c00000c EMIF_config2 cs1nvmen 1:30 ebankPos 1:27 rdbnum 2:4 rdbsize 3:0
readMem 0x4c000010 EMIF_refresh initDis 1:31 srt 1:29 asr 1:28 pasr 3:24 refreshRate 16:0
readMem 0x4c000014 EMIF_refreshS rate 16:0
readMem 0x4c000018 EMIF_tim1 tRTW 3:29 tRP 4:25 tRCD 4:21 tWR 4:17 tRAS 5:12 tRC 6:6 tRRD 3:3 tWTR 3:0
readMem 0x4c00001c EMIF_tim1S
readMem 0x4c000020 EMIF_tim2 tXP 3:28 tODT 3:25 tXSNR 9:16 xXSRD 10:6 tRTP 3:3 tCKE 3:0
readMem 0x4c000024 EMIF_tim2S
readMem 0x4c000028 EMIF_tim3 tDDLunl 4:28 tCSTA 4:24 tCKESR 3:21 ZQCS 6:15 tTDQSCKmax 2:13 tRFC 9:4 tRASmax 4:0
readMem 0x4c00002c EMIF_tim3S
readMem 0x4c000038 EMIF_pow pdTim 4:12 dpdEn 1:11 lpMode 3:8 srTim 4:4
readMem 0x4c00003c EMIF_powS
readMem 0x4c000040 EMIF_ddr2D val0 7:0
readMem 0x4c000050 EMIF_ddr2config cs 1:31 refresh 1:30 add 8:0
readMem 0x4c000054 EMIF_ocp_conf sysMax 4:24 mpuMax 4:20 llMax 4:16
readMem 0x4c000058 EMIF_ocp_confV1 sysBusW 2:30 llBusW 2:28 wrFifoD 8:8 cmdFifoD 8:0
readMem 0x4c00005C EMIF_ocp_confV2 counter1 32:0

On my board with some timings I've tweaked, I get

EMIF_rev @ 0x4c000000 = 0x51440501
revision: 1363412225
EMIF_Status @ 0x4c000004 = 0x40000004
be: 0   dual_clk: 1     fast_init: 0    rdlgate: 0      rdlto: 0        wrlto: 0        phy_dll: 1
EMIF_config @ 0x4c000008 = 0x61951BBA
type: 3 pos: 0  ddr3_termR: 1   ddr2_ddqs: 1    ddr3_odt: 0     ddr_disableDll: 1       driveStre: 1    cwl: 1  narrowMode: 0   cl: 6rowsize: 7      ibank: 3        ebank: 1        pageS: 2
EMIF_config2 @ 0x4c00000c = 0x0
cs1nvmen: 0     ebankPos: 0     rdbnum: 0       rdbsize: 0
EMIF_refresh @ 0x4c000010 = 0x1035
initDis: 0      srt: 0  asr: 0  pasr: 0 refreshRate: 4149
EMIF_refreshS @ 0x4c000014 = 0x1035
rate: 4149
EMIF_tim1 @ 0x4c000018 = 0xEEEF46F3
tRTW: 7 tRP: 7  tRCD: 7 tWR: 7  tRAS: 20        tRC: 27 tRRD: 6 tWTR: 3
EMIF_tim1S @ 0x4c00001c = 0xEEEF46F3

EMIF_tim2 @ 0x4c000020 = 0x30C47FDB
tXP: 3  tODT: 0 tXSNR: 196      xXSRD: 511      tRTP: 3 tCKE: 3
EMIF_tim2S @ 0x4c000024 = 0x30C47FDB

EMIF_tim3 @ 0x4c000028 = 0x29F8BE8
tDDLunl: 0      tCSTA: 2        tCKESR: 4       ZQCS: 63        tTDQSCKmax: 0   tRFC: 190       tRASmax: 8
EMIF_tim3S @ 0x4c00002c = 0x29F8BE8

EMIF_pow @ 0x4c000038 = 0xF2F0
pdTim: 15       dpdEn: 0        lpMode: 2       srTim: 15
EMIF_powS @ 0x4c00003c = 0xF0F0

EMIF_ddr2D @ 0x4c000040 = 0x0
val0: 0
EMIF_ddr2config @ 0x4c000050 = 0x0
cs: 0   refresh: 0      add: 0
EMIF_ocp_conf @ 0x4c000054 = 0xA500000
sysMax: 10      mpuMax: 5       llMax: 0
EMIF_ocp_confV1 @ 0x4c000058 = 0x9000190A
sysBusW: 2      llBusW: 1       wrFifoD: 25     cmdFifoD: 10
EMIF_ocp_confV2 @ 0x4c00005C = 0x42727
counter1: 272167

Now the interesting ones that have changed most recently are the timing1/2/3 registers. We haven't been printing those during the uboot process so I added the debugging:

        printf("EMIF_SDRAM_TIMING  %08x %08x\n", readl((volatile uint32_t*) 0x4C000018), readl((volatile uint32_t*) 0x4D000018));
        printf("EMIF_SDRAM_TIMING2 %08x %08x\n", readl((volatile uint32_t*) 0x4C000020), readl((volatile uint32_t*) 0x4D000020));
        printf("EMIF_SDRAM_TIMING3 %08x %08x\n", readl((volatile uint32_t*) 0x4C000028), readl((volatile uint32_t*) 0x4D000028));

to void print_status(void) { in arch/arm/cpu/armv7/omap-common/emif-common.c

I thought I was onto something interesting but it turns out I had a precedence issue. Added some more detailed debugging for a memory address. If you call ./omapEmifDump.sh 0x4c000018, you get:

EMIF_tim1 @ 0x4c000018 = 0xEEEF46F3
11101110111011110100011011110011 is the bit value of the register.
11100000000000000000000000000000 tRTW = 7
00011110000000000000000000000000 tRP = 7
00000001111000000000000000000000 tRCD = 7
00000000000111100000000000000000 tWR = 7
00000000000000011111000000000000 tRAS = 20
00000000000000000000111111000000 tRC = 27
00000000000000000000000000111000 tRRD = 6
00000000000000000000000000000111 tWTR = 3

which is the same as above, but shows the bit masking. Might come in handy if we get suspicious.

This was to test a theory that some process was changing these registers after uboot. I haven't seen any evidence of that yet. I've run out of energy to add more registers at this point. May continue tomorrow if weather isn't too brutal.
This script may still help EvilDragon identify what timings are being set on which boards and we can pick the most compatible.

 

[Magic Sam]

Hi all,

@pimaster : thank you for your work on this issue, really appreciated.

I ran your script on my Pyra and got slightly different results.

Are they worth posting here ?

Cheers, Magic Sam

 

[pimaster]

I ran your script on my Pyra and got slightly different results.
Are they worth posting here ?

Um. Not yet. I loosened the timing of my board hoping for an easy fix and may not have reverted it. It was mostly a script for EvilDragon so he can see what settings are on what boards and then see if the "worst" (probably meaning slowest) would work for all.
And if anyone was feeling adventurous into solving the problem with me, it might be some insight.

 

[Risca]

Has anyone checked if this errata bug is relevant for memory freezes?

i878 MPU Lockup With Concurrent DMM and EMIF Accesses
CRITICALITY High
DESCRIPTION The MPU has two primary paths to DDR and system address space via the MPU
Memory Adapter (MPU_MA).
The Low Latency path is the predominant path for DDR accesses and provides
direct/low latency/interleaved access to the two EMIFs.
The L3 Interconnect path (via MPU_AXI2OCP bridge) is most typically used for access
to non-DDR address space, but is also used for access to DMM and EMIF control
registers and to Tiled regions of DDR address space.
Issue is seen to come when there is a heavy memory access through the MPU L3 path,
if the MPU is concurrently issuing write transactions via the Low Latency path to DDR
and via the L3 Interconnect to the DMM/EMIF/Tiler address space then the transactions
can hang and the MPU and DMM/DDR become unresponsive. A device reset is required
in order to recover from this condition

 

[pimaster]

Given that there are memory issues on boot, before Tiler should become an issue I don't think this is an issue yet.
I'm not sure if DMM is already in use at uboot time. Thought it was more for Linux to use, but a quick google suggests that it could be set up in uboot.
When I've experienced crashes inside Linux, I've generally been doing nothing. I'll be looking something up or compiling on a different machine and when I return the device is unresponsive.