Hi all,
Here are two fixes for issues I found testing with the new pyra CPU boards.
These are against commit c8fb1e629c77 ("U-Boot: OMAP: fix variables names in default boot environment"), I did not want to change anything from last year while debugging this. No other patches are needed any longer, so no need to disable the DDR internal DLL. And there is no need to set the memory voltage to 1.5V, so please also check your kernel dts file to remove that.
I've tested with new revision pyra C02, C03, C05 and C21. Also my older revision pyra C50 now works.
Please test and report back if you need to increase the T_RFC value in the second patch for some pyra board.
Regards,
Tony
Tony Lindgren (2): LC15: Fix flakey ddr3 detection with ref_ctrl and ref_ctrl_final LC15: Increase T_RFC to fix 4GB kernel memory errors
board/goldelico/letux-cortex15/lc15.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
Since commit 802bb57a584d ("ARM: DRA7: EMIF: Update SDRAM_REF_CTRL register value") we now do this in two phases.
We are not currently configuring ref_ctrl and this seems to cause bogus memory errors early in u-boot that often happen if the board has been powered off for a while:
SDRAM: identified size not same as expected size identified: 0 expected: 40000000 ... WARNING: DDR RAM is not working! (12000000) ... ### ERROR ### Please RESET the board ###
Note that this does not solve the kernel related memory issues, but gets us to the point where u-boot starts reliably.
Signed-off-by: Tony Lindgren tony@atomide.com --- board/goldelico/letux-cortex15/lc15.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/board/goldelico/letux-cortex15/lc15.c b/board/goldelico/letux-cortex15/lc15.c --- a/board/goldelico/letux-cortex15/lc15.c +++ b/board/goldelico/letux-cortex15/lc15.c @@ -40,7 +40,8 @@ const struct emif_regs emif_regs_ddr3_532_mhz_2cs_es2 = { .sdram_config_init = 0x61851B3A, .sdram_config = 0x61851B3A, .sdram_config2 = 0x0, - .ref_ctrl = 0x00001035, + .ref_ctrl = 0x000040F1, + .ref_ctrl_final = 0x00001035, .sdram_tim1 = 0xCCCF36B3, .sdram_tim2 = 0x308F7FDA, .sdram_tim3 = 0x027F88A8,
With u-boot related memory errors out of the way fixed with patch "LC15: Fix flakey ddr3 detection with ref_ctrl and ref_ctrl_final", we can now narrow down the kernel related errors easily.
Playing with timing changes I narrowed down the kernel hangs to register sdram_tim3 where we need to increase the time for T_RFC. This is for "Minimum number of DDR clock cycles from Refresh or Load Mode to Refresh or Activate, minus one" according to omap5432 TRM.
Maxing out the value for bits 12:4 made the new 4GB pyra's C02 and C03 boot reliably. And then looking at the timings again, the T_RFC value should be 187 - 1 (0xBA) and not 139 - 1 (0x8A). Note the Upper case B vs 8 :)
Value 187 - 1 (0xBA) matches Samsung K4B8G1646D-MYK0 datasheet "[ Table 34 ] Timing used for IDD and IDDQ Measured - Loop Patterns" for DDR3-1066 8Gb memory. We are currently using 4Gb value 139 - 1 (0x8A).
It is possible that this was caused by B vs 8 typo initially and then was not noticed because of the u-boot memory errors and getting confused with the memory datasheet for 4Gb vs 4GB while we have four 8Gb memory chips.
However I also noticed that value 187 - 1 (0xBA) still won't work reliably with new board C05. And new C21 and my older revision board C50 still did not work at all. However, using value of 191 - 1 (0xBE) makes all of them boot reliably for me. So let's use value 191 - 1 (0xBE) for now, and adjust it if needed based on testing with more pyras.
Note that we still can have pyra hang running two instances of memtester 1500M for about 10 minutes on bare CPU boards with no heat sink connected. The boards now boot reliably after that though, so presumably the pyra heat sink will solve this issue.
Signed-off-by: Tony Lindgren tony@atomide.com --- board/goldelico/letux-cortex15/lc15.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/board/goldelico/letux-cortex15/lc15.c b/board/goldelico/letux-cortex15/lc15.c --- a/board/goldelico/letux-cortex15/lc15.c +++ b/board/goldelico/letux-cortex15/lc15.c @@ -44,7 +44,7 @@ const struct emif_regs emif_regs_ddr3_532_mhz_2cs_es2 = { .ref_ctrl_final = 0x00001035, .sdram_tim1 = 0xCCCF36B3, .sdram_tim2 = 0x308F7FDA, - .sdram_tim3 = 0x027F88A8, + .sdram_tim3 = 0x027F8BE8, .read_idle_ctrl = 0x00050000, .zq_config = 0xD007190B, .temp_alert_config = 0x00000000,
Hi Tony,
Am 29.05.2018 um 18:10 schrieb Tony Lindgren tony@atomide.com:
With u-boot related memory errors out of the way fixed with patch "LC15: Fix flakey ddr3 detection with ref_ctrl and ref_ctrl_final", we can now narrow down the kernel related errors easily.
Playing with timing changes I narrowed down the kernel hangs to register sdram_tim3 where we need to increase the time for T_RFC. This is for "Minimum number of DDR clock cycles from Refresh or Load Mode to Refresh or Activate, minus one" according to omap5432 TRM.
Maxing out the value for bits 12:4 made the new 4GB pyra's C02 and C03 boot reliably. And then looking at the timings again, the T_RFC value should be 187 - 1 (0xBA) and not 139 - 1 (0x8A). Note the Upper case B vs 8 :)
Value 187 - 1 (0xBA) matches Samsung K4B8G1646D-MYK0 datasheet "[ Table 34 ] Timing used for IDD and IDDQ Measured - Loop Patterns" for DDR3-1066 8Gb memory. We are currently using 4Gb value 139 - 1 (0x8A).
It is possible that this was caused by B vs 8 typo initially and then was not noticed because of the u-boot memory errors and getting confused with the memory datasheet for 4Gb vs 4GB while we have four 8Gb memory chips.
However I also noticed that value 187 - 1 (0xBA) still won't work reliably with new board C05. And new C21 and my older revision board C50 still did not work at all. However, using value of 191 - 1 (0xBE) makes all of them boot reliably for me. So let's use value 191 - 1 (0xBE) for now, and adjust it if needed based on testing with more pyras.
Wow! Cool!
This will help EvilDragon to cough less fire [after another project setback from this morning] :)
I'll try these patches ASAP.
BR and thanks, Nikolaus
Note that we still can have pyra hang running two instances of memtester 1500M for about 10 minutes on bare CPU boards with no heat sink connected. The boards now boot reliably after that though, so presumably the pyra heat sink will solve this issue.
Signed-off-by: Tony Lindgren tony@atomide.com
board/goldelico/letux-cortex15/lc15.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/board/goldelico/letux-cortex15/lc15.c b/board/goldelico/letux-cortex15/lc15.c --- a/board/goldelico/letux-cortex15/lc15.c +++ b/board/goldelico/letux-cortex15/lc15.c @@ -44,7 +44,7 @@ const struct emif_regs emif_regs_ddr3_532_mhz_2cs_es2 = { .ref_ctrl_final = 0x00001035, .sdram_tim1 = 0xCCCF36B3, .sdram_tim2 = 0x308F7FDA,
- .sdram_tim3 = 0x027F88A8,
- .sdram_tim3 = 0x027F8BE8, .read_idle_ctrl = 0x00050000, .zq_config = 0xD007190B, .temp_alert_config = 0x00000000,
-- 2.17.0
* H. Nikolaus Schaller hns@goldelico.com [180529 16:21]:
This will help EvilDragon to cough less fire [after another project setback from this morning] :)
Hmm I'm not aware of this one yet..
I'll try these patches ASAP.
Looks like the jedec spec JESD79-3F-1.pdf "Table 61 — Refresh parameters by device density" for 8Gb mentions 350, so I guess the worst case could be up to 350 - 1 (0x15d). But let's try with the current values first. I'm not exactly what all T_RFC value affects, seems like every time we enter and exit self refresh at least.
Regards,
Tony
Am 29.05.2018 um 18:54 schrieb Tony Lindgren tony@atomide.com:
- H. Nikolaus Schaller hns@goldelico.com [180529 16:21]:
This will help EvilDragon to cough less fire [after another project setback from this morning] :)
Hmm I'm not aware of this one yet..
Well, it is not communicated yet (and neither SW nor HW related). ED will certainly do after calming down...
I'll try these patches ASAP.
Looks like the jedec spec JESD79-3F-1.pdf "Table 61 — Refresh parameters by device density" for 8Gb mentions 350, so I guess the worst case could be up to 350 - 1 (0x15d). But let's try with the current values first. I'm not exactly what all T_RFC value affects, seems like every time we enter and exit self refresh at least.
Regards,
Tony
BR and thanks for this help (this task of fixing 4GB is well beyond my capabilities), Nikolaus
Hi,
On Tue, May 29, 2018 at 7:10 PM, Tony Lindgren tony@atomide.com wrote:
Note that we still can have pyra hang running two instances of memtester 1500M for about 10 minutes on bare CPU boards with no heat sink connected. The boards now boot reliably after that though, so presumably the pyra heat sink will solve this issue.
I've tried these patches on my 4g board from last year (also with double memtesters) and although it did not crash, memtester detected errors after running for about an hour. My board is "lucky" that it boots even with old code, but it used to crash in around 20 minutes or so. I haven't seen it crash with 1.37V with the old code, although it might just have been luck. I'll do some more testing later (don't want to leave it unattended while I'm at work).
Gražvydas
* Grazvydas Ignotas notasas@gmail.com [180530 08:41]:
Hi,
On Tue, May 29, 2018 at 7:10 PM, Tony Lindgren tony@atomide.com wrote:
Note that we still can have pyra hang running two instances of memtester 1500M for about 10 minutes on bare CPU boards with no heat sink connected. The boards now boot reliably after that though, so presumably the pyra heat sink will solve this issue.
I've tried these patches on my 4g board from last year (also with double memtesters) and although it did not crash, memtester detected errors after running for about an hour. My board is "lucky" that it boots even with old code, but it used to crash in around 20 minutes or so. I haven't seen it crash with 1.37V with the old code, although it might just have been luck. I'll do some more testing later (don't want to leave it unattended while I'm at work).
Yeah worth testing for sure.
Do you have a heat sink attached? I recall the last time I played with this I was able to run two instances of memtester 1500M for longer periods of time only if I added a proper heatsink and a fan.
Regards,
Tony
On Wed, May 30, 2018 at 5:42 PM, Tony Lindgren tony@atomide.com wrote:
- Grazvydas Ignotas notasas@gmail.com [180530 08:41]:
Hi,
On Tue, May 29, 2018 at 7:10 PM, Tony Lindgren tony@atomide.com wrote:
Note that we still can have pyra hang running two instances of memtester 1500M for about 10 minutes on bare CPU boards with no heat sink connected. The boards now boot reliably after that though, so presumably the pyra heat sink will solve this issue.
I've tried these patches on my 4g board from last year (also with double memtesters) and although it did not crash, memtester detected errors after running for about an hour. My board is "lucky" that it boots even with old code, but it used to crash in around 20 minutes or so. I haven't seen it crash with 1.37V with the old code, although it might just have been luck. I'll do some more testing later (don't want to leave it unattended while I'm at work).
Yeah worth testing for sure.
I've tested 2 more scenarios: new timings, 1.37V: memtester errors after ~1h old timings, 1.37: memtester errors after 4h+ hours
Could just be randomness or heat related, second test was done in the evening under slightly lower room temperature.
Do you have a heat sink attached? I recall the last time I played with this I was able to run two instances of memtester 1500M for longer periods of time only if I added a proper heatsink and a fan.
No heatsink, it's a cased unit. How do you cool the OMAP when it's inside the "sandwich"? Are you simply cooling the "empty" side of CPU board?
Gražvydas
* Grazvydas Ignotas notasas@gmail.com [180531 08:38]:
On Wed, May 30, 2018 at 5:42 PM, Tony Lindgren tony@atomide.com wrote:
- Grazvydas Ignotas notasas@gmail.com [180530 08:41]:
Hi,
On Tue, May 29, 2018 at 7:10 PM, Tony Lindgren tony@atomide.com wrote:
Note that we still can have pyra hang running two instances of memtester 1500M for about 10 minutes on bare CPU boards with no heat sink connected. The boards now boot reliably after that though, so presumably the pyra heat sink will solve this issue.
I've tried these patches on my 4g board from last year (also with double memtesters) and although it did not crash, memtester detected errors after running for about an hour. My board is "lucky" that it boots even with old code, but it used to crash in around 20 minutes or so. I haven't seen it crash with 1.37V with the old code, although it might just have been luck. I'll do some more testing later (don't want to leave it unattended while I'm at work).
Yeah worth testing for sure.
I've tested 2 more scenarios: new timings, 1.37V: memtester errors after ~1h old timings, 1.37: memtester errors after 4h+ hours
Could just be randomness or heat related, second test was done in the evening under slightly lower room temperature.
Hmm OK. What CPU speed are you running it at?
Do you have a heat sink attached? I recall the last time I played with this I was able to run two instances of memtester 1500M for longer periods of time only if I added a proper heatsink and a fan.
No heatsink, it's a cased unit. How do you cool the OMAP when it's inside the "sandwich"? Are you simply cooling the "empty" side of CPU board?
Looking at your memtester times sounds like the case is working well as a heat sink :)
I had some PC heat sink tape around that I put on the empty top side of the CPU board, then dropped four tiny heatsinks and two tiny PC 12V fans on that :)
I can still get memtester errors after about 10 minutes, but I noticed I now have the fans hooked to 5V power supply. I need to test with fans running at 10V or something higher as I think that's what I used the last time to be able to run two instances of memtester 1500M for longer periods of time.
Regards,
Tony
* Tony Lindgren tony@atomide.com [180531 14:00]:
I can still get memtester errors after about 10 minutes, but I noticed I now have the fans hooked to 5V power supply. I need to test with fans running at 10V or something higher as I think that's what I used the last time to be able to run two instances of memtester 1500M for longer periods of time.
Yeah seems to work reliably based on about an hour of testing with fans running at higher voltage. I'm seeing these temperatures:
/sys/class/thermal/thermal_zone0/temp 64026 /sys/class/thermal/thermal_zone1/temp 60677 /sys/class/thermal/thermal_zone2/temp 60600
Will try doubling the refresh rate next without fans.
Regards,
Tony
* Tony Lindgren tony@atomide.com [180531 20:28]:
Yeah seems to work reliably based on about an hour of testing with fans running at higher voltage. I'm seeing these temperatures:
/sys/class/thermal/thermal_zone0/temp 64026 /sys/class/thermal/thermal_zone1/temp 60677 /sys/class/thermal/thermal_zone2/temp 60600
Will try doubling the refresh rate next without fans.
No luck playing with EMIF_SDRAM_REFRESH_CONTROL without fans. I start seeing memory errors at temperatures around 70.
If found a page discussing these register values here if somebody wants to experiment with it:
https://e2e.ti.com/support/arm/sitara_arm/f/791/t/667835?AM5718-EMIF-Tools-h...
Regards,
Tony
On Thu, May 31, 2018 at 4:58 PM, Tony Lindgren tony@atomide.com wrote:
- Grazvydas Ignotas notasas@gmail.com [180531 08:38]:
On Wed, May 30, 2018 at 5:42 PM, Tony Lindgren tony@atomide.com wrote:
- Grazvydas Ignotas notasas@gmail.com [180530 08:41]:
Hi,
On Tue, May 29, 2018 at 7:10 PM, Tony Lindgren tony@atomide.com wrote:
Note that we still can have pyra hang running two instances of memtester 1500M for about 10 minutes on bare CPU boards with no heat sink connected. The boards now boot reliably after that though, so presumably the pyra heat sink will solve this issue.
I've tried these patches on my 4g board from last year (also with double memtesters) and although it did not crash, memtester detected errors after running for about an hour. My board is "lucky" that it boots even with old code, but it used to crash in around 20 minutes or so. I haven't seen it crash with 1.37V with the old code, although it might just have been luck. I'll do some more testing later (don't want to leave it unattended while I'm at work).
Yeah worth testing for sure.
I've tested 2 more scenarios: new timings, 1.37V: memtester errors after ~1h old timings, 1.37: memtester errors after 4h+ hours
Could just be randomness or heat related, second test was done in the evening under slightly lower room temperature.
Hmm OK. What CPU speed are you running it at?
Oh right, forgot about that. Turns out I had all Nikolaus' made up non-TI-approved OPPs in so got rid of them first, but without the 500MHz OPP my unit does thermal shutdown, even if I only leave the 1GHz OPP in. My unit doesn't boot without a battery (too weak supply?), so I've pointed a PC case fan at the cased unit and it maintains 90C now while running at 1GHz only. With that it ran stable for 4h, I'll run it like that for more time later after work.
Gražvydas
Am Freitag, den 01.06.2018, 11:33 +0300 schrieb Grazvydas Ignotas:
Hi,
My unit doesn't boot without a battery (too weak supply?),
It's probably the following issue: PALMAS sets up the USB port to not allow more than 500mA in U-Boot. That's not enough for loading the kernel, so it crashes and resets.
The kernel increases the allowed current on USB ports, so once the kernel has booted once, it should boot without a battery as well.
So: Boot up with battery in, remove the battery. Then it should work without a battery just fine, even if you power it down and restart it (as long as you don't remove the power).
Setting up this and other parameters for the PALMAS with U-Boot is something we still need to do. It's annoying you can't boot without a battery, currently.
Hi,
Am 05.06.2018 um 12:12 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Freitag, den 01.06.2018, 11:33 +0300 schrieb Grazvydas Ignotas:
Hi,
My unit doesn't boot without a battery (too weak supply?),
It's probably the following issue: PALMAS sets up the USB port to not allow more than 500mA in U-Boot.
it is the bq24297 which initially limits the current.
That's not enough for loading the kernel, so it crashes and resets.
The kernel increases the allowed current on USB ports, so once the kernel has booted once, it should boot without a battery as well.
So: Boot up with battery in, remove the battery. Then it should work without a battery just fine, even if you power it down and restart it (as long as you don't remove the power).
Setting up this and other parameters for the PALMAS with U-Boot is something we still need to do.
AFAIR we have code in U-Boot to do that - but not in MLO - so it may still come too late.
It's annoying you can't boot without a battery, currently.
Indeed. Although this is mainly a problem for boards without plastics case.
BR, Nikolaus
Hi,
On Tue, Jun 5, 2018 at 1:24 PM, H. Nikolaus Schaller hns@goldelico.com wrote:
Hi,
Am 05.06.2018 um 12:12 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Freitag, den 01.06.2018, 11:33 +0300 schrieb Grazvydas Ignotas:
Hi,
My unit doesn't boot without a battery (too weak supply?),
It's probably the following issue: PALMAS sets up the USB port to not allow more than 500mA in U-Boot.
it is the bq24297 which initially limits the current.
That's not enough for loading the kernel, so it crashes and resets.
The kernel increases the allowed current on USB ports, so once the kernel has booted once, it should boot without a battery as well.
So: Boot up with battery in, remove the battery. Then it should work without a battery just fine, even if you power it down and restart it (as long as you don't remove the power).
Setting up this and other parameters for the PALMAS with U-Boot is something we still need to do.
AFAIR we have code in U-Boot to do that - but not in MLO - so it may still come too late.
It's annoying you can't boot without a battery, currently.
Indeed. Although this is mainly a problem for boards without plastics case.
Well for me it happens quite late during the boot, in the middle of systemd starting services, so the kernel has had a chance to run for several seconds. It could be it's loading the charger driver at that time, although I haven't confirmed this. Also it only gets that far if the 2A charger is connected to USB3 port, if it's on the debug port, the system dies before the kernel is loaded (this one is probably because of missing charger configuration in MLO?).
Gražvydas
Hi,
Am 06.06.2018 um 10:39 schrieb Grazvydas Ignotas notasas@gmail.com:
Hi,
On Tue, Jun 5, 2018 at 1:24 PM, H. Nikolaus Schaller hns@goldelico.com wrote:
Hi,
Am 05.06.2018 um 12:12 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Freitag, den 01.06.2018, 11:33 +0300 schrieb Grazvydas Ignotas:
Hi,
My unit doesn't boot without a battery (too weak supply?),
It's probably the following issue: PALMAS sets up the USB port to not allow more than 500mA in U-Boot.
it is the bq24297 which initially limits the current.
That's not enough for loading the kernel, so it crashes and resets.
The kernel increases the allowed current on USB ports, so once the kernel has booted once, it should boot without a battery as well.
So: Boot up with battery in, remove the battery. Then it should work without a battery just fine, even if you power it down and restart it (as long as you don't remove the power).
Setting up this and other parameters for the PALMAS with U-Boot is something we still need to do.
AFAIR we have code in U-Boot to do that - but not in MLO - so it may still come too late.
It's annoying you can't boot without a battery, currently.
Indeed. Although this is mainly a problem for boards without plastics case.
Well for me it happens quite late during the boot, in the middle of systemd starting services, so the kernel has had a chance to run for several seconds. It could be it's loading the charger driver at that time, although I haven't confirmed this.
Yes, that could be. If I remember correctly, the bq24297 driver is loaded around that time. It might it turns down/limit power for a short moment (maybe we must write registers in a different sequence).
Or it could be that the full system has a power consumption peak at that time and the bq24297 isn't the problem.
Also it only gets that far if the 2A charger is connected to USB3 port, if it's on the debug port, the system dies before the kernel is loaded (this one is probably because of missing charger configuration in MLO?).
Gražvydas
BR, Nikolaus
Hi all, finally I found some time to do tests.
Firstly, I tried two boards built end 2016: C41 with Samsung 4GB and C45 reworked to IntelligentMemory 4GB. I tried them both with different main boards (M28 resp. M16, both V5.1.3 built 20170119).
They seemed to boot a little more stable, but did either hang in Kernel or even after Starting Kernel... Once I did even see the ### RESET ### notice.
Sometimes they did boot up to login: but there were "Illegal Instruction" and "Segfault" reports in the boot log (maybe from udevd because some kernel modules were missing). I could log in and type a handful of commands, but then the kernel got stuck.
Next I tried the same SD card with one randomly picked new CPU board: 20180502 C24. I also used the M16 main board.
And the result is a stable working 4GB system! Equally to what I am used to from the 2GB CPUs. And this with 1.5GHz enabled.
I haven't run memtester or run the temperature to the maximum. Or other high-load tests, but setting up USB and apt-get upgrade did work fine and continue to run for at least 2 hours now.
Apt-get installing 400 more packages also went well, which is a load comparable to compile something.
There was one stuck situation, when the battery became discharged below 3.2V (the charger function is broken on M16). But replacing the battery with a charged one made it work again. This might have something to do with my M16 setup - it does not use the Pyra battery but a tiny GTA04 battery connected through quite weak cables. This makes power supply a more critical thing than a real Pyra will have.
So I think that both factors, having done a hardware redesign and Tony's new patches, have improved this setup significantly.
BR, Nikolaus
* H. Nikolaus Schaller hns@goldelico.com [180610 15:50]:
Hi all, finally I found some time to do tests.
Firstly, I tried two boards built end 2016: C41 with Samsung 4GB and C45 reworked to IntelligentMemory 4GB. I tried them both with different main boards (M28 resp. M16, both V5.1.3 built 20170119).
They seemed to boot a little more stable, but did either hang in Kernel or even after Starting Kernel... Once I did even see the ### RESET ### notice.
I guess this makes sense as IntelligentMemory needs it's own timings different from Samsung timings :)
So I think that both factors, having done a hardware redesign and Tony's new patches, have improved this setup significantly.
OK good to hear. The thing to do would be to test all the new CPU boards to see if they boot.
Regards,
Tony
-
Grazvydas Ignotas -
H. Nikolaus Schaller -
Michael Mrozek -
Tony Lindgren