As reported by Michael Mrozek, Pyra 4GB seems to hang quite easily within about 10 to 15 minutes when left idle, and it seems to be related to DDR3 self-refresh.
Setting DYN_ODT bits [22:21] enables ODT. Presumably there is one bit per rank or chip select. According to the TRM, we want to keep both bits cleared. And we want to configure DDR_TERM bits [26:24] to 3 which according to the TRM makes it RZQ/6. Trying to boot with DDR_TERM set to 2 won't work.
Additionally we also want to clear EMIF1_PHY_RD_LOCAL_ODT bits [6:5] in CONTROL_EMIF1_SDRAM_CONFIG_EXT to disable ODT further. And the same for CONTROL_EMIF2_SDRAM_CONFIG_EXT register. And we can just drop the unused define for SDRAM_CONFIG_EXT_RD_LVL_11_SAMPLES_ES2.
Note that the wiring for ODT0 and ODT1 should also be checked in the hardware. It seems that ODT0 must only be connected to the banks in the first rank only, and ODT1 must only be connected to the banks in the second rank.
I came across this information while search for DYN_ODT info, see a related KeyStone thread at [0] below. It says:
"ODT0 must be routed to all SDRAMs in the first rank. In dual-rank designs, ODT1 must be routed to all SDRAMS in the second rank. In designs with only one rank, ODT1 will be left unconnected."
It seems we now have ODT0 and ODT1 both connected to all banks on Pyra? I and I wonder if that might be the reason for TRM saying not supported for DYN_ODT?
Anyways, with ODT disabled, these signals should not do anything and fixing the hardware can probably wait.
[0] https://e2e.ti.com/support/processors/f/791/t/450668?DDR3-Dynamic-ODT-on-C66...
Reported-by: Michael Mrozek EvilDragon@openpandora.org Signed-off-by: Tony Lindgren tony@atomide.com ---
Only lightly tested so far, but my Pyra 4GB has survived now for more than 15 minutes :) Please test on multiple devices.. No need for the "disable internal DLL hack" with this any longer.
--- arch/arm/cpu/armv7/omap5/hw_data.c | 4 ++-- arch/arm/include/asm/emif.h | 3 ++- board/goldelico/letux-cortex15/lc15.c | 2 +- 3 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/arch/arm/cpu/armv7/omap5/hw_data.c b/arch/arm/cpu/armv7/omap5/hw_data.c --- a/arch/arm/cpu/armv7/omap5/hw_data.c +++ b/arch/arm/cpu/armv7/omap5/hw_data.c @@ -661,8 +661,8 @@ const struct ctrl_ioregs ioregs_omap5432_es2 = { .ctrl_ddrio_0 = DDR_IO_0_VREF_CELLS_DDR3_VALUE_ES2, .ctrl_ddrio_1 = DDR_IO_1_VREF_CELLS_DDR3_VALUE_ES2, .ctrl_ddrio_2 = DDR_IO_2_VREF_CELLS_DDR3_VALUE_ES2, - .ctrl_emif_sdram_config_ext = SDRAM_CONFIG_EXT_RD_LVL_11_SAMPLES, - .ctrl_emif_sdram_config_ext_final = SDRAM_CONFIG_EXT_RD_LVL_4_SAMPLES, + .ctrl_emif_sdram_config_ext = SDRAM_CONFIG_EXT_NO_ODT_11_SAMPLES, + .ctrl_emif_sdram_config_ext_final = SDRAM_CONFIG_EXT_NO_ODT_4_SAMPLES, };
const struct ctrl_ioregs ioregs_dra7xx_es1 = { diff --git a/arch/arm/include/asm/emif.h b/arch/arm/include/asm/emif.h --- a/arch/arm/include/asm/emif.h +++ b/arch/arm/include/asm/emif.h @@ -538,7 +538,8 @@
#define SDRAM_CONFIG_EXT_RD_LVL_11_SAMPLES 0x0000C1A7 #define SDRAM_CONFIG_EXT_RD_LVL_4_SAMPLES 0x000001A7 -#define SDRAM_CONFIG_EXT_RD_LVL_11_SAMPLES_ES2 0x0000C1C7 +#define SDRAM_CONFIG_EXT_NO_ODT_11_SAMPLES 0x0000C187 +#define SDRAM_CONFIG_EXT_NO_ODT_4_SAMPLES 0x00000187
/* DMM */ #define DMM_BASE 0x4E000040 diff --git a/board/goldelico/letux-cortex15/lc15.c b/board/goldelico/letux-cortex15/lc15.c --- a/board/goldelico/letux-cortex15/lc15.c +++ b/board/goldelico/letux-cortex15/lc15.c @@ -42,7 +42,7 @@ const struct emif_regs emif_regs_ddr3_532_mhz_2cs_es2 = { .sdram_config2 = 0x0, .ref_ctrl = 0x000040F1, .ref_ctrl_final = 0x00001035, - .sdram_tim1 = 0xCCCF36B3, + .sdram_tim1 = 0xCB8F36B3, .sdram_tim2 = 0x308F7FDA, .sdram_tim3 = 0x027F8BE8, .read_idle_ctrl = 0x00050000,
* Tony Lindgren tony@atomide.com [181010 22:15]:
Only lightly tested so far, but my Pyra 4GB has survived now for more than 15 minutes :) Please test on multiple devices.. No need for the "disable internal DLL hack" with this any longer.
Hmm I guess that claim was too early. I saw an oops after 20 some minutes, and was using a different CPU board than I usually use. So not sure if this makes any difference actually.
Regards,
Tony
On Thu, Oct 11, 2018 at 1:53 AM Tony Lindgren tony@atomide.com wrote:
- Tony Lindgren tony@atomide.com [181010 22:15]:
Only lightly tested so far, but my Pyra 4GB has survived now for more than 15 minutes :) Please test on multiple devices.. No need for the "disable internal DLL hack" with this any longer.
Hmm I guess that claim was too early. I saw an oops after 20 some minutes, and was using a different CPU board than I usually use. So not sure if this makes any difference actually.
I don't think it does, not a positive effect at least.
A helpful thing for testing this problem is to reduce REG_SR_TIM to make self-refresh much more frequent (I use 9). With that 4GB is rarely able to finish boot, while 2GB board is still fine, so you can instantly know if something helps or not. I think I've found some issues (will clean up and send patches later), but nothing really helps this refresh problem really :(
And yes it's almost certain it's a self-refresh issue, if you disable it (change LP_MODE_SELF_REFRESH to LP_MODE_DISABLE in emif.h) the problem also goes away.
Gražvydas
* Grazvydas Ignotas notasas@gmail.com [181010 23:59]:
On Thu, Oct 11, 2018 at 1:53 AM Tony Lindgren tony@atomide.com wrote:
- Tony Lindgren tony@atomide.com [181010 22:15]:
Only lightly tested so far, but my Pyra 4GB has survived now for more than 15 minutes :) Please test on multiple devices.. No need for the "disable internal DLL hack" with this any longer.
Hmm I guess that claim was too early. I saw an oops after 20 some minutes, and was using a different CPU board than I usually use. So not sure if this makes any difference actually.
I don't think it does, not a positive effect at least.
Yeah :( But maybe try it with the following additional changes too for the drive strength and termination:
- .sdram_tim1 = 0xCB8F36B3, + .sdram_tim1 = 0xCA8736B3,
A helpful thing for testing this problem is to reduce REG_SR_TIM to make self-refresh much more frequent (I use 9). With that 4GB is rarely able to finish boot, while 2GB board is still fine, so you can instantly know if something helps or not. I think I've found some issues (will clean up and send patches later), but nothing really helps this refresh problem really :(
Yeah I've been using REG_SR_TIM 0xc, did not get anything booting with lower values there.
And yes it's almost certain it's a self-refresh issue, if you disable it (change LP_MODE_SELF_REFRESH to LP_MODE_DISABLE in emif.h) the problem also goes away.
Yes I can confirm that too.
Regards,
Tony
* Tony Lindgren tony@atomide.com [181011 00:15]:
- Grazvydas Ignotas notasas@gmail.com [181010 23:59]:
On Thu, Oct 11, 2018 at 1:53 AM Tony Lindgren tony@atomide.com wrote:
- Tony Lindgren tony@atomide.com [181010 22:15]:
Only lightly tested so far, but my Pyra 4GB has survived now for more than 15 minutes :) Please test on multiple devices.. No need for the "disable internal DLL hack" with this any longer.
Hmm I guess that claim was too early. I saw an oops after 20 some minutes, and was using a different CPU board than I usually use. So not sure if this makes any difference actually.
I don't think it does, not a positive effect at least.
Yeah :( But maybe try it with the following additional changes too for the drive strength and termination:
.sdram_tim1 = 0xCB8F36B3,
.sdram_tim1 = 0xCA8736B3,
Uhh, I've been patching the wrong register.. Time to take a break I guess :) I think we already have the sdram_config regs OK so the only change in my patch is to clear the ODT for SDRAM_CONFIG_EXT.
Later,
Tony
On Mi, 2018-10-10 at 17:15 -0700, Tony Lindgren wrote:
Hi,
interesting find. I thought self-refresh would mostly work automatic when there's no external clock to go into a power-saving mode - but it still depends on some timings.
Checking the datasheet of the RAM, there are five timings to be setup.
Two have min-values and three have max-values given, so no fixed timings... so there are some values we can test.
Additionally, there is a timing "tXSDLL" which seems to have something to do with the DLL. Description: "Exit Self Refresh to commands requiring a locked DLL"
So my guess is it sometimes hangs when it tries to exit self-refresh and we have DLL enabled?
Those timings are on page 52 of the RAM datasheet, in case you want to look for them.
And yes it's almost certain it's a self-refresh issue, if you disable it (change LP_MODE_SELF_REFRESH to LP_MODE_DISABLE in emif.h) the problem also goes away.
Yes I can confirm that too.
Regards,
Tony _______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
On Thu, Oct 11, 2018 at 6:08 AM Michael Mrozek EvilDragon@openpandora.org wrote:
On Mi, 2018-10-10 at 17:15 -0700, Tony Lindgren wrote:
Hi,
interesting find. I thought self-refresh would mostly work automatic when there's no external clock to go into a power-saving mode - but it still depends on some timings.
Checking the datasheet of the RAM, there are five timings to be setup.
Two have min-values and three have max-values given, so no fixed timings... so there are some values we can test.
Additionally, there is a timing "tXSDLL" which seems to have something to do with the DLL. Description: "Exit Self Refresh to commands requiring a locked DLL"
So my guess is it sometimes hangs when it tries to exit self-refresh and we have DLL enabled?
Those timings are on page 52 of the RAM datasheet, in case you want to look for them.
I've gone through all of them, and the ones that are programmable are correctly programmed already. There were minor mistakes (patches sent) but fixing them has no effect on the hang issue. Things I tried: - higher cl values - more relaxed / higher other timings that are programmable - dumping registers from Linux and confirming everything is programmed as it should be - overclocking to 533.7MHz (not sure if exact 533 can be achieved because of how omap's dpll works) - other things I forgot
Among things not tried are strange registers like CONTROL_DDR3CH2_0 CONTROL_EMIF1_SDRAM_CONFIG_EXT - it's hard to understand what they do and how they could help with our issue.
The only thing that helps besides DLL disable is disabling self-refresh. I think the RAMs are incompatible with how OMAP5 does power saving, that's all there is to it. We'll have to choose one of those workarounds. Probably disabling self-refresh makes the most sense since unlike the DLL option it's not against any specs or requirements.
Gražvydas
On Fr, 2018-10-12 at 02:41 +0300, Grazvydas Ignotas wrote:
Hi,
wow, thanks a lot for that!
Too bad it doesn't help much - but at least we can get the boards reliable to work.
It seems like a problem with some TI-processors. Here is a VERY similar sounding issue: https://e2e.ti.com/support/processors/f/791/t/588807?Linux-AM3352-DDR3-self-...
Micron has an interesting guide about self-refresh modes: https://www.micron.com/products/datasheets/%7B2ADCEEB1-307F-4B37-99C9-7302CA...
Not sure if our memory behaves the same, but it has some information about the requirements to enter self-refresh mode. Not sure if these requirements / procedure is being done by the OMAP5 alone or whether this is controlled by the kernel though.
Do you know if disabling self-refresh has any negative side effects? Would it increase power consumption a lot or probably not really noticeable?
I've gone through all of them, and the ones that are programmable are correctly programmed already. There were minor mistakes (patches sent) but fixing them has no effect on the hang issue. Things I tried:
- higher cl values
- more relaxed / higher other timings that are programmable
- dumping registers from Linux and confirming everything is
programmed as it should be
- overclocking to 533.7MHz (not sure if exact 533 can be achieved
because of how omap's dpll works)
- other things I forgot
Among things not tried are strange registers like CONTROL_DDR3CH2_0 CONTROL_EMIF1_SDRAM_CONFIG_EXT - it's hard to understand what they do and how they could help with our issue.
The only thing that helps besides DLL disable is disabling self-refresh. I think the RAMs are incompatible with how OMAP5 does power saving, that's all there is to it. We'll have to choose one of those workarounds. Probably disabling self-refresh makes the most sense since unlike the DLL option it's not against any specs or requirements.
Gražvydas
On 10/12/18 4:23 AM, Michael Mrozek wrote:
On Fr, 2018-10-12 at 02:41 +0300, Grazvydas Ignotas wrote:
Hi,
wow, thanks a lot for that!
I've made a new version of pyra-install with that new u-boot.
http://packages.pyra-handheld.com/images/stretch/pyra-install.tgz
* Michael Mrozek EvilDragon@openpandora.org [181012 02:23]:
On Fr, 2018-10-12 at 02:41 +0300, Grazvydas Ignotas wrote:
Do you know if disabling self-refresh has any negative side effects? Would it increase power consumption a lot or probably not really noticeable?
It will increase the idle power consumption quite a bit AFAIK, this is a sign of something being broken for sure.
I've gone through all of them, and the ones that are programmable are correctly programmed already. There were minor mistakes (patches sent) but fixing them has no effect on the hang issue. Things I tried:
- higher cl values
- more relaxed / higher other timings that are programmable
- dumping registers from Linux and confirming everything is
programmed as it should be
- overclocking to 533.7MHz (not sure if exact 533 can be achieved
because of how omap's dpll works)
- other things I forgot
Among things not tried are strange registers like CONTROL_DDR3CH2_0 CONTROL_EMIF1_SDRAM_CONFIG_EXT - it's hard to understand what they do and how they could help with our issue.
The only thing that helps besides DLL disable is disabling self-refresh. I think the RAMs are incompatible with how OMAP5 does power saving, that's all there is to it. We'll have to choose one of those workarounds. Probably disabling self-refresh makes the most sense since unlike the DLL option it's not against any specs or requirements.
These are my findings as well, I was not able to find anything that would make things better or worse regarding the self refresh issue other than disable self refresh or disable internal dll like Grazvydas listed. Timing value changes for tRFC and tXSDLL don't seem to make a difference.
My gut feeling is it's some kind of signaling issue occasionally causing the relocking the of the ddr internal dll to fail. The only difference compared to other omap5 and dra7 devices is the use of two chip selects on Pyra 4GB.
But for developer devices and testing, yeah disabling memory self-refresh is the way to go.
Regards,
Tony
Am Fri, 12 Oct 2018 06:15:15 -0700 hat Tony Lindgren tony@atomide.com geschrieben:
Hi.
Do you know if disabling self-refresh has any negative side effects? Would it increase power consumption a lot or probably not really noticeable?
It will increase the idle power consumption quite a bit AFAIK, this is a sign of something being broken for sure.
Well, hardware- or softwarewise broken? Or can't be made out yet.
The only thing that helps besides DLL disable is disabling self-refresh. I think the RAMs are incompatible with how OMAP5 does power saving, that's all there is to it. We'll have to choose one of those workarounds. Probably disabling self-refresh makes the most sense since unlike the DLL option it's not against any specs or requirements.
These are my findings as well, I was not able to find anything that would make things better or worse regarding the self refresh issue other than disable self refresh or disable internal dll like Grazvydas listed. Timing value changes for tRFC and tXSDLL don't seem to make a difference.
Damn.
My gut feeling is it's some kind of signaling issue occasionally causing the relocking the of the ddr internal dll to fail. The only difference compared to other omap5 and dra7 devices is the use of two chip selects on Pyra 4GB.
Hm, though we've used the IBIS files to simulate everything - and the current CPU boards passed, so hardware-wise, it SHOULD be okay.
And what's even weirder: The more is going on, the more noise you have in the normal signals, but that doesn't seem to affect the RAM. What could be bad enough to affect the relocking of the DDR internal DLL?
Are there differences between switching to / from self-refreshing to normal state on dual-die RAM chips which maybe isn't properly included in the kernel or similar?
Hm... does the IGEP use dual-die RAM chips?
But for developer devices and testing, yeah disabling memory self-refresh is the way to go.
Yes, it helps so I can work on the tester script.
However, it's the last issue preventing me to give the "Go" for mass production.
So the question is, how to solve that.
On Fr, 2018-10-12 at 06:15 -0700, Tony Lindgren wrote:
I found an interesting document from SK Hynix regarding DDR3L memory. https://www.skhynix.com/static/filedata/fileDownload.do?seq=212
This one describes the way to enter Self-Refresh mode. The interesting thing:
Their procedure to enter Self-Refreh mode includes disabling DLL and enabling DLL again when you exit self-refresh mode.
So I wonder... is that a standard? Or does every memory vendor have their own requirements / procedures to enter / leave self-refresh?
- Michael Mrozek EvilDragon@openpandora.org [181012 02:23]:
On Fr, 2018-10-12 at 02:41 +0300, Grazvydas Ignotas wrote:
Do you know if disabling self-refresh has any negative side effects? Would it increase power consumption a lot or probably not really noticeable?
It will increase the idle power consumption quite a bit AFAIK, this is a sign of something being broken for sure.
I've gone through all of them, and the ones that are programmable are correctly programmed already. There were minor mistakes (patches sent) but fixing them has no effect on the hang issue. Things I tried:
- higher cl values
- more relaxed / higher other timings that are programmable
- dumping registers from Linux and confirming everything is
programmed as it should be
- overclocking to 533.7MHz (not sure if exact 533 can be achieved
because of how omap's dpll works)
- other things I forgot
Among things not tried are strange registers like CONTROL_DDR3CH2_0 CONTROL_EMIF1_SDRAM_CONFIG_EXT - it's hard to understand what they do and how they could help with our issue.
The only thing that helps besides DLL disable is disabling self-refresh. I think the RAMs are incompatible with how OMAP5 does power saving, that's all there is to it. We'll have to choose one of those workarounds. Probably disabling self-refresh makes the most sense since unlike the DLL option it's not against any specs or requirements.
These are my findings as well, I was not able to find anything that would make things better or worse regarding the self refresh issue other than disable self refresh or disable internal dll like Grazvydas listed. Timing value changes for tRFC and tXSDLL don't seem to make a difference.
My gut feeling is it's some kind of signaling issue occasionally causing the relocking the of the ddr internal dll to fail. The only difference compared to other omap5 and dra7 devices is the use of two chip selects on Pyra 4GB.
But for developer devices and testing, yeah disabling memory self-refresh is the way to go.
Regards,
Tony
Trying to google my way around the Self-Refresh / DLL freezes... it seems like it's a pretty common issue, as multiple users had similar issues (hanging / freezing when getting out of Self-Refresh mode), for various systems (i.MX, AMs, etc.)
One had the issue that only froze when running the memory at 400MHz, with 533MHz, it worked fine.
Another one had the issue that he used an OPP-Setting that wasn't supported by the memory - so it often freezes.
There are some things we might try to look into as well.
If I understood correctly, then Self-Refresh is being entered and exited by the kernel. So both the call and the procedure is being controlled by software.
But our chip also supports the (optional) feature "Auto Self-Refresh". According to the JEDEC document:
"DDR3 SDRAM provides an Auto Self-Refresh mode (ASR) for application ease. ASR mode is enabled by setting MR2 bit A6 = 1b and MR2 bit A7 = 0b . The DRAM will manage Self-Refresh entry in either the Normal or Extended (optional) Temperature Ranges. In this mode, the DRAM will also manage Self-Refresh power consumption when the DRAM operating temperature changes, lower at low temperatures and higher at high temperatures. If the ASR option is not supported by the DRAM, MR2 bit A6 must be set to 0b. If the ASR mode is not enabled (MR2 bit.A6 = 0b), the SRT bit (MR2 A7) must be manually programmed with the operating temperature range required during Self-Refresh operation."
So - are we talking about AUTO self-refresh or self-refresh which causes our issue?
Since Auto Self-Refresh is optional, maybe it's disabled in our setup?
If Auto Self-Refresh does the same as Self-Refresh (but it's being handled by the RAM itself), maybe that woukd work instead?
On Fri, Oct 12, 2018 at 6:36 PM Michael Mrozek EvilDragon@openpandora.org wrote:
Well, hardware- or softwarewise broken?
Probably hardware, there is very little software control for how it does it's thing (see below).
If I understood correctly, then Self-Refresh is being entered and exited by the kernel. So both the call and the procedure is being controlled by software.
Not on OMAP5's EMIF, here it's all done by EMIF hardware. You can only control the on/off switch for the feature, time how long it has to idle (no RAM access by anything) before it starts the procedure and some limited timing parameters.
But our chip also supports the (optional) feature "Auto Self-Refresh".
How do you know that? It's not mentioned in feature list of our RAM's datasheet.
So - are we talking about AUTO self-refresh or self-refresh which causes our issue?
The "manual" one, which is handled by OMAP5 hardware.
Since Auto Self-Refresh is optional, maybe it's disabled in our setup?
Yes
If Auto Self-Refresh does the same as Self-Refresh (but it's being handled by the RAM itself), maybe that woukd work instead?
Yes there is a bit for that in OMAP5, might try it later.
Gražvydas
On Sa, 2018-10-13 at 02:31 +0300, Grazvydas Ignotas wrote:
Hi,
Well, hardware- or softwarewise broken?
Probably hardware, there is very little software control for how it does it's thing (see below).
Hm, I wonder what though. The simulation is successful with the correct IBIS files... I'll ask the experts at Memphis again, they know the RAM and might have some clues.
I'll also test how much power consumption differs with Self-Refresh off or on as well.
If I understood correctly, then Self-Refresh is being entered and exited by the kernel. So both the call and the procedure is being controlled by software.
Not on OMAP5's EMIF, here it's all done by EMIF hardware. You can only control the on/off switch for the feature, time how long it has to idle (no RAM access by anything) before it starts the procedure and some limited timing parameters.
Okay, then it probably just follows JEDEC standard which should just work.
But our chip also supports the (optional) feature "Auto Self- Refresh".
How do you know that? It's not mentioned in feature list of our RAM's datasheet.
Well, it says on page 36 of the datasheet: " Auto Self-Refresh (ASR): set MR2 A6 = 0B to disable or 1B to enable feature"
It's just a footnote though, no further mentioning, so it's either a leftover from a different chip or they didn't want to mention it anywhere else. You never know with datasheets...
It's the same with our 2GB IM-Memory. They mention how to enable ASR, but that's it.
If Auto Self-Refresh does the same as Self-Refresh (but it's being handled by the RAM itself), maybe that woukd work instead?
Yes there is a bit for that in OMAP5, might try it later.
Thanks - if that works, it would be great.
On Sat, 13 Oct 2018 at 02:11, Michael Mrozek EvilDragon@openpandora.org wrote:
But our chip also supports the (optional) feature "Auto Self- Refresh".
How do you know that? It's not mentioned in feature list of our RAM's datasheet.
Well, it says on page 36 of the datasheet: " Auto Self-Refresh (ASR): set MR2 A6 = 0B to disable or 1B to enable feature"
That bit doesn't mean what you think it does. It means that when in self-refresh, the RAM will automatically increase its refresh rate when it detects it is operating at extended temperatures (85͏°C - 95͏°C).
You can find an overview of all JEDEC-defined MR bits in the MR sheet of this spreadsheet of mine: https://docs.google.com/spreadsheets/d/13UBY1PXElPrd8bldaBXN-_bWHXSzUKWU8Lmy...
Matthijs
On Fr, 2018-10-12 at 06:15 -0700, Tony Lindgren wrote:
Hi,
Would it increase power consumption a lot or probably not really noticeable?
It will increase the idle power consumption quite a bit AFAIK, this is a sign of something being broken for sure.
Okay, I now measured the power consumption and compared a 4GB unit with Self-Refresh disabled, the same unit with Self-Refresh enabled and a 2GB RAM unit (old mainboard and old CPU board).
And... well, either Idle / Self-Refresh is not working properly or there isn't really a difference yet.
Here's what I did:
1. Booted into the 4.19 kernel with only a basic set of modules enabled (so I had LEDs and the backlight of the LCD switched on).
2. Removed the battery after booting (so it was only being powered with USB). The USB cable had a USB measurement device inbetween.
3. Let the unit sit still for 10 minutes, measuring the power consumption.
The results:
2GB RAM unit: Varying between 1.9 - 2.1W (or 0.37 - 0.41A). 4GB RAM unit: Varying between 2.1 - 2.3W (or 0.41 - 0.47A).
So as expected, power consumption with the 4GB RAM unit is a bit higher. With a 22Wh battery as we have, the runtime difference in this case between the 2GB RAM unit and the 4GB would be around an hour (11 hours with the 2GB RAM and 10 hours with the 4GB RAM). Of course, it might change when the RAM is actually used :)
However, I didn't have ANY difference betwen the MLO / U-Boot with Self-Refresh enabled or disabled.
No idea why though...
On Sat, Oct 13, 2018 at 5:44 AM Michael Mrozek EvilDragon@openpandora.org wrote:
On Fr, 2018-10-12 at 06:15 -0700, Tony Lindgren wrote:
Hi,
Would it increase power consumption a lot or probably not really noticeable?
It will increase the idle power consumption quite a bit AFAIK, this is a sign of something being broken for sure.
Okay, I now measured the power consumption and compared a 4GB unit with Self-Refresh disabled, the same unit with Self-Refresh enabled and a 2GB RAM unit (old mainboard and old CPU board).
And... well, either Idle / Self-Refresh is not working properly or there isn't really a difference yet.
How are you disabling self-refresh? Note that my patches aren't disabling it yet, I just asked if we should do it. You also have to blank the screen, otherwise DSS is constantly reading the framebuffer (which sits in RAM) to display it.
Gražvydas
On Sat, 2018-10-13 at 19:29 +0300, Grazvydas Ignotas wrote:
Hi,
Okay, I now measured the power consumption and compared a 4GB unit with Self-Refresh disabled, the same unit with Self-Refresh enabled and a 2GB RAM unit (old mainboard and old CPU board). And... well, either Idle / Self-Refresh is not working properly or there isn't really a difference yet.
How are you disabling self-refresh? Note that my patches aren't disabling it yet, I just asked if we should do it.
Ah, okay! I thought that was in aTc's latest U-Boot. So it's only DLL disabled then, okay!
You also have to blank the screen, otherwise DSS is constantly reading the framebuffer (which sits in RAM) to display it.
Hm, there's one thing I understand then:
If our issue is self-refresh and that will only happen if nothing is on the screen, why does the unit freeze in our tests? Because I always have something showing on the screen, I never blank it.
On Sun, Oct 14, 2018 at 5:01 AM Michael Mrozek EvilDragon@openpandora.org wrote:
On Sat, 2018-10-13 at 19:29 +0300, Grazvydas Ignotas wrote:
Hi,
Okay, I now measured the power consumption and compared a 4GB unit with Self-Refresh disabled, the same unit with Self-Refresh enabled and a 2GB RAM unit (old mainboard and old CPU board). And... well, either Idle / Self-Refresh is not working properly or there isn't really a difference yet.
How are you disabling self-refresh? Note that my patches aren't disabling it yet, I just asked if we should do it.
Ah, okay! I thought that was in aTc's latest U-Boot. So it's only DLL disabled then, okay!
No, there is no workaround there, it should still hang. The patches fix other mistakes, but not the actual issue.
So you want a workaround patch there? Should we disable self-refresh or DLL then?
You also have to blank the screen, otherwise DSS is constantly reading the framebuffer (which sits in RAM) to display it.
Hm, there's one thing I understand then:
If our issue is self-refresh and that will only happen if nothing is on the screen, why does the unit freeze in our tests? Because I always have something showing on the screen, I never blank it.
I never said it only happens with blanking screen.
What I meant to say is that measuring power consumption with screen on probably makes little sense because of DSS reading RAM. Only during vsync/vblank (and maybe hblank too) it will be entering self-refresh, instead of anytime with the screen off, so you'd see little power usage difference. And because you work with video a lot, I wrongly (?) assumed you'd have vsync in mind while reading this.
Gražvydas
On Mo, 2018-10-15 at 00:15 +0300, Grazvydas Ignotas wrote:
Hi,
How are you disabling self-refresh? Note that my patches aren't disabling it yet, I just asked if we should do it.
Ah, okay! I thought that was in aTc's latest U-Boot. So it's only DLL disabled then, okay!
No, there is no workaround there, it should still hang. The patches fix other mistakes, but not the actual issue.
Ah, okay, that's probably the latest then which he sent. I haven't run that for longer than 10 minutes. I thought the new one had the self-refresh off included already.
He created another MLO / U-Boot a couple of days before which had the DLL disabled and the Row-Size patch included.
That one runs stable, but I haven't done any power consumption tests with that one and I don't think he even posted this at the mailing list here :)
So you want a workaround patch there? Should we disable self-refresh or DLL then?
Well, question is: What's better?
According to JEDEC: "During normal operation (DLL on), MR1 (A0 = 0), the DLL is automatically disabled upon entering Self-Refresh and is automatically enabled (including a DLL-Reset) upon exiting Self-Refresh."
So does that mean that self-refresh still works if we disable DLL? I still wonder what disabling DLL actually does? It seems to be only used in "normal operation".
You also have to blank the screen, otherwise DSS is constantly reading the framebuffer (which sits in RAM) to display it.
Hm, there's one thing I understand then: If our issue is self-refresh and that will only happen if nothing is on the screen, why does the unit freeze in our tests? Because I always have something showing on the screen, I never blank it.
I never said it only happens with blanking screen. What I meant to say is that measuring power consumption with screen on probably makes little sense because of DSS reading RAM. Only during vsync/vblank (and maybe hblank too) it will be entering self- refresh, instead of anytime with the screen off, so you'd see little power usage difference. And because you work with video a lot, I wrongly (?) assumed you'd have vsync in mind while reading this.
Wow, it enters self-refresh within that short time frame? :O I didn't expect that, as (according to what details I found), self- refresh only really saves power if it's active for longer periods of time.
Okay, what I would like to try is: Compare the power consumption of:
1. Normal operation mode (DLL on, Self-Refresh on). I know it crashes after a few minutes, but I should be able to test the power consumption.
2. DLL disabled
3. Self-Refresh disabled, DLL enabled
I already have U-Boots for 1 and 2, only 3 is missing.
Can I blank the screen from the serial console? Whast would be the command for that?
Thanks :)
On Mon, 15 Oct 2018 03:58:43 +0200 Michael Mrozek EvilDragon@openpandora.org wrote:
Can I blank the screen from the serial console? Whast would be the command for that?
xset dpms force off
That should blank the screen immediately until I move the mouse, at least it does for me on my linux terminal, although I'm not running a pyra prototype or the pyra os so your mileage may vary. That's an Xorg command though, so if you're booting to the terminal you might have better luck with the setterm command, although in testing it just now I couldn't get it to do anything.
Hi,
here is some interesting note from a community member at the boards:
Normally the DLL (Delay-Locked Loop) should only be disabled to run RAM at very low clock speeds (for power saving). Disabling the dynamic DDR RAM clock speed might also fix the issue.
Are we running at a dynamic DDR RAM clock speed? Or is that static?
That would explain why the RAM crashes when the device is idle and the RAM controller drops the memory clock. It might be possible to turn DLL off before power save and re-enabling it when going back to full clock rate but it could be risky if that causes bus glitches during retraining/resync. DDR RAM is always a pain to get stable.
Well, DLL should be automatically disabled on self-refresh anyways (based on the Jedec doc), but maybe the RAM freezes when DLL is being enabled again...
It still means to me: If we disable DLL, then self-refresh should still work - and unless we don't run into any issues when using the RAM normally with DLL switched off, we shouldn't have any downsides, or should we?
As DLL becomes more important the faster the RAM clock is: 532MHz isn't that fast (DDR3 supports way higher clock rates) which is the reason DLL off works fine...?
Am 15.10.2018 um 18:43 schrieb Michael Mrozek EvilDragon@openpandora.org:
Hi,
here is some interesting note from a community member at the boards:
Normally the DLL (Delay-Locked Loop) should only be disabled to run RAM at very low clock speeds (for power saving). Disabling the dynamic DDR RAM clock speed might also fix the issue.
Are we running at a dynamic DDR RAM clock speed? Or is that static?
AFAIR it is static, but experts should know better.
That would explain why the RAM crashes when the device is idle and the RAM controller drops the memory clock. It might be possible to turn DLL off before power save and re-enabling it when going back to full clock rate but it could be risky if that causes bus glitches during retraining/resync. DDR RAM is always a pain to get stable.
Well, DLL should be automatically disabled on self-refresh anyways (based on the Jedec doc), but maybe the RAM freezes when DLL is being enabled again...
Or has a hickup while trying to lock again.
It still means to me: If we disable DLL, then self-refresh should still work - and unless we don't run into any issues when using the RAM normally with DLL switched off, we shouldn't have any downsides, or should we?
As DLL becomes more important the faster the RAM clock is: 532MHz isn't that fast (DDR3 supports way higher clock rates) which is the reason DLL off works fine...?
I have thought a little about the key purpose of the DLL - it is to compensate for different wiring lengths between clocks and data lines, especially on bigger DDR3 DIMMs for PCs. There, each DRAM can have a different distance from the connector and hence a different delay, especially if there are 8 or 16 chips on one board, there may be some cm. Each m distance on some PCB means roughly 10 ns delay. So a memory board with 5cm length can have 500ps delay between both ends which is a lot for a 1GHz signal (half sine wave). And this is compensated by the DLL.
But our CPU bord has a much nearer connection of at most 2cm and mostly parallel data and clock lines to each chip. So this could be the reason why the DLL doesn't seem to be ultimately necessary.
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org _______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Am Mon, 15 Oct 2018 19:02:48 +0200 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:
Hi,
It still means to me: If we disable DLL, then self-refresh should still work - and unless we don't run into any issues when using the RAM normally with DLL switched off, we shouldn't have any downsides, or should we? As DLL becomes more important the faster the RAM clock is: 532MHz isn't that fast (DDR3 supports way higher clock rates) which is the reason DLL off works fine...?
I have thought a little about the key purpose of the DLL - it is to compensate for different wiring lengths between clocks and data lines, especially on bigger DDR3 DIMMs for PCs. There, each DRAM can have a different distance from the connector and hence a different delay, especially if there are 8 or 16 chips on one board, there may be some cm. Each m distance on some PCB means roughly 10 ns delay. So a memory board with 5cm length can have 500ps delay between both ends which is a lot for a 1GHz signal (half sine wave). And this is compensated by the DLL. But our CPU bord has a much nearer connection of at most 2cm and mostly parallel data and clock lines to each chip. So this could be the reason why the DLL doesn't seem to be ultimately necessary.
Interesting. While this doesn't explain why this fails for our system, it is good to know we might not need it.
Are these memory chips the same that are being used on normal DDR3 memory DIMMS? Because in that case, especially, with the additional connectors of the slot, this seems to be very important.
Hm. One thing: Could it be DLL causes issues when the distance from the RAM chip to the CPU is TOO short?
I know that with some video / audio cables (and fibre glass) there's a MINIMUM cable length as well, otherwise it's overpowered (or has reflections).
I know the SIGNAL lines have some autolevelling and a driver impedance for those short traces.
On Mon, 15 Oct 2018 at 18:43, Michael Mrozek EvilDragon@openpandora.org wrote:
Are we running at a dynamic DDR RAM clock speed? Or is that static?
Static. Changing the RAM clock speed is a rather delicate and involved procedure and not supported by linux currently (afaik).
Well, DLL should be automatically disabled on self-refresh anyways (based on the Jedec doc), but maybe the RAM freezes when DLL is being enabled again...
Given that apparently frequent transitions into and out of self-refresh make the problem worse, and DLL-disabled helps, it would suggest that the RAM may be having problems getting its DLL to lock reliably. Question then would be of course: Why? I think it would imply either issues with the ram clock it receives or issues with its power supply.
If we disable DLL, then self-refresh should still work - and unless we don't run into any issues when using the RAM normally with DLL switched off, we shouldn't have any downsides, or should we?
The main concern is that JEDEC puts an upper limit of 125 MHz on DLL-disabled mode, and overall leaves it very poorly specified. We'd either have to get custom timing specs from the manufacturer or just experiment to find stable parameters and hope those don't turn out to be highly variable between devices.
As DLL becomes more important the faster the RAM clock is: 532MHz isn't that fast (DDR3 supports way higher clock rates) which is the reason DLL off works fine...?
Sounds plausible yes. The aforementioned 125MHz limit was obviously set by JEDEC in days when ram speeds were much lower overall.
Matthijs
On Fr, 2018-11-02 at 02:55 +0100, Matthijs van Duin wrote:
Hi,
Are we running at a dynamic DDR RAM clock speed? Or is that static?
Static. Changing the RAM clock speed is a rather delicate and involved procedure and not supported by linux currently (afaik).
Ah, okay. Does that mean that self-refresh also doesn't lower the clock rate?
Is Android different here?
Well, DLL should be automatically disabled on self-refresh anyways (based on the Jedec doc), but maybe the RAM freezes when DLL is being enabled again...
Given that apparently frequent transitions into and out of self-refresh make the problem worse, and DLL-disabled helps, it would suggest that the RAM may be having problems getting its DLL to lock reliably. Question then would be of course: Why? I think it would imply either issues with the ram clock it receives or issues with its power supply.
I think we can rule out the power supply, as the memory runs fine whatever you do when DLL is off, and even when DLL is on as long as you run a memtester and keep the memory busy.
Unless you mean that the power supply has issues keeping the power constant when the memory usage changes but that would be the PALMAS and that handles it automatically.
Also, 2GB RAM doesn't have the issue, and the power consumption is nearly the same.
If we disable DLL, then self-refresh should still work - and unless we don't run into any issues when using the RAM normally with DLL switched off, we shouldn't have any downsides, or should we?
The main concern is that JEDEC puts an upper limit of 125 MHz on DLL-disabled mode, and overall leaves it very poorly specified. We'd either have to get custom timing specs from the manufacturer or just experiment to find stable parameters and hope those don't turn out to be highly variable between devices.
As far as I understood from the JEDEC documents, DLL doesn't have anything to do with the timing settings - we've already set up the custom timings according to the datasheet.
If I understood correctly, DLL is being used to automatically compensate timing issues caused by longer traces (higher resistance values). A Dual-Die RAM could otherwise receive or transfer the requested value from the wrong chip.
However, if it works fine without DLL, then it seems our traces / resistances are good enough for the timings..?
As DLL becomes more important the faster the RAM clock is: 532MHz isn't that fast (DDR3 supports way higher clock rates) which is the reason DLL off works fine...?
Sounds plausible yes. The aforementioned 125MHz limit was obviously set by JEDEC in days when ram speeds were much lower overall.
Okay. BTW: Where did you find thoese 125MHz limit? I've checked the full JEDEC documentation, but didn't find anything about that (or a limit value for DLL)
On Sat, 3 Nov 2018 at 21:03, Michael Mrozek EvilDragon@openpandora.org wrote:
Ah, okay. Does that mean that self-refresh also doesn't lower the clock rate?
The memory clock is gated in self-refresh.
I think we can rule out the power supply, as the memory runs fine whatever you do when DLL is off, and even when DLL is on as long as you run a memtester and keep the memory busy.
The same argument can be made for the memory clock. It is possible however that DLL locking is more sensitive to sub-optimal power supply stability and/or memory clock irregularities. Anyway, it's just a guess, but the best one I have given the info available to me.
Unless you mean that the power supply has issues keeping the power constant when the memory usage changes but that would be the PALMAS and that handles it automatically.
Palmas is more for long-term stability. Keeping the power supply stable during short-term fluctuations of power consumption is dependent on decoupling caps and pcb layout.
Also, 2GB RAM doesn't have the issue, and the power consumption is nearly the same.
That doesn't really say anything since the average power consumption is not the suspect in the first place.
As far as I understood from the JEDEC documents, DLL doesn't have anything to do with the timing settings - we've already set up the custom timings according to the datasheet.
The datasheet contains nothing but standard JEDEC timings. It could just as easily have mentioned which clock speeds are supported and the range of CAS values for each and referred to JEDEC for the rest.
If I understood correctly, DLL is being used to automatically compensate timing issues caused by longer traces
No, the memory controller's DLLs are used for that. The RAM's DLL is not programmable and cannot be used to compensate for trace length. The RAM needs the DLL to make its data outputs and ODT toggle at the right time (as specified by JEDEC) relative to the memory clock. In DLL-off mode the timing for data read from the ram becomes JEDEC-unspecified ("tDQSCK(DLL_off) values are vendor specific"), and ODT is officially not supported at all in DLL-off mode.
(higher resistance values).
Trace length directly affects timings because of the time it takes for signals to travel along the trace. Resistance values have nothing to do with it.
A Dual-Die RAM could otherwise receive or transfer the requested value from the wrong chip.
Ehh, no.
However, if it works fine without DLL, then it seems our traces / resistances are good enough for the timings..?
Trace lengths are not relevant for this. It means that 1. the RAM seems to be okay with DLL-off operation at the frequency and CAS value where we use it (which jedec doesn't spec at all). 2. with our current settings EMIF seems to be okay with whatever (JEDEC-unspecified) timing the RAM outputs data during reads in DLL-off mode. 3. we either don't use/need ODT, or it behaves well enough in DLL-off mode (even though JEDEC does not consider ODT to be supported in DLL-off mode)
Okay. BTW: Where did you find thoese 125MHz limit?
4.5 DLL-off Mode The maximum clock frequency for DLL-off Mode is specified by the parameter tCK(DLL_OFF).
13.1 Timing Parameters for DDR3-800, DDR3-1067, DDR3-1333, and DDR3-1600 Table 68: tCK(DLL_OFF): min 8 ns
Okay, here are my new power consumption tests. Blanking the screen (with setterm --blank force) doesn't only disable the backlight of the LCD but also of the keyboard :)
4GB Unit, DLL off:
Sitting idle with screen on: 0.46A, 2.2W Sitting idle with blanked screen: 0.24A, 1.2W
4GB Unit, Self-Refresh off:
Sitting idle with screen on: 0.46A, 2.2W Sitting idle with blanked screen: 0.28A, 1.4W
4GB Unit: Sitting idle with screen on: 0.46A, 2.2W Sitting idle with blanked screen: 0.24A, 1.2W
2GB Unit:
Sitting idle with screen on: 0.39A, 1.9W Sitting idle with blanked screen: 0.19A, 0.9W
So, with self-refresh off, it seems to need a bit more battery. dll off doesn't have any change in the battery usage.
One interesting side-note:
I hard one freeze witht the dll-off unit.. could be aomething different, but it could also show that disablin self-refresh might nocht be enough.
So I guess from what we know right now, disabling DLL is the best solution for us.
-- Mit freundlichen Grüßen,
Michael Mrozek
----------------------- OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt ----------------------- eMail: mrozek@openpandora.org
On Tue, Oct 16, 2018 at 5:07 AM Michael Mrozek EvilDragon@openpandora.org wrote:
So I guess from what we know right now, disabling DLL is the best solution for us.
ok pushed a patch for that to pyra-u-boot.git until a better solution is found (if ever).
Gražvydas
Quick addition, as I just did another power usage test:
4GB Unit sitting in U-Boot: 0.38A (1.89W) 4GB Unit running mtest in U-Boot: 0.56A (2.8W)
2GB Unit sitting in U-Boot: 0.37A (1.85W) 2GB Unit running mtest in U-Boot: 0.58A (2.85W)
So it seems in full usage, the 2GB RAM uses a bit more power than the 4GB RAM.
-
aTc -
Grazvydas Ignotas -
H. Nikolaus Schaller -
levi -
Matthijs van Duin -
Michael Mrozek -
Tony Lindgren