I've got 22 Mainboard + CPU board combination (latest 4GB RAM versions) lying around at home and would like to ensure that the hardware is working fine - so I can place the order for the mass production.
However, I'm running into issues which I can't solve myself (as I am no developer and can't even do small changes to a kernel, etc.) I'm good with Linux, but not with developing - so I need some serious help here.
The unit I'm testing right now is one of latest batch.
Our own PyraOS usually freezes the system right after 30 seconds to 1 minute, so not really usable.
The latest "stable" Letux (kernel 4.18.8) with LXDE as base OS (installed using the makesd-tool) does boot fine. On two tries, it freezed after around 10 - 12 minutes sitting around idle. And about 2 - 3 minutes running the high-load script (which ups the CPU to 1,5GHz with some loop).
I will do more testings to see if it's always around the same time.
I've made a small script that logs power supply, uptime and temperature to see if it has something to do with that.
The result (incl. bootlog) is attached. I went to bed while it was running, but you can see in the log that it freezes after 12 minutes. Temperature is fine, voltage also seems to be fine and when using the high-load script, the temperature is higher after a few seconds and it still takes minutes to freeze - so we can assume that the freeze doesn't have anything to do with temperature CPU load.
Interestingly, when I came back, the frozen unit was still on (and showed the frozen desktop screen), but pretty warm - a lot warmer compared to it just idling.
I've added the new temperature and charging state at the end of freeze.txt as well.
Some things we know:
* Both askarus' and aTc's older Pyras (older Mainboard, 2GB RAM CPU board without our optimizations) are stable with kernel 4.15. aTc mentioned he'd been running it off USB for days whereas askarus' Pyra runs 10 hours off battery without freezing and in normal usage (internet, LibreOffice, etc.) Unfortunately, the released kernel 4.15 doesn't support the new mainboards / CPU boards for some reason, so I can't compare them.
* According to hns, even the uEVM freezes randomly with the latest kernel 4.18
* These freezes don't seem to be connected with temperature or CPU load (both Askarus and aTcs Pyras don't have any heat sink applied, whereas the 4GB RAM unit I've tested it on does).
* Sometimes the freeze shows random graphic glitches on the display, sometimes the display simply freezes (memory errors?)
* The freeze might NOT be a full freeze! When using hns' QuantumStep OS (which uses the LEDs as heartbeat), the unit also seemed to have frozen after a couple of minutes, as the LEDs stopped blinking. However, the LCD was still updating the graphics - VERY very slowly (1 character per 30 seconds on the terminal), but it was still working - even though it seemed frozen at first. This doesn't seem to be the case on LXDE, but maybe it's even slower because there's so much going on.
* On the unit I tested, sometimes u-Boot reported that it expected 4000000 of memory size but only found 4. This might be related to the unit - or maybe some timings are still wrong for the 4GB RAM so there's some issue here?
* Running the uevm with little less than 5V increases the instability. As aTc and askarus Pyra work fine with less than 5V, this shouldn't be a problem - but I thought I'd mention it here anyways.
So I'd like to find out the reason for these random freezes are. I've got various PCBs and cases, I can run tests - but as I can't code, I can't play around with kernel parameters or similar, so I'm pretty limited finding out what it could be.
That's where I need your help! Does anyone have any ideas what the issue could be and can provide different kernels, board files, etc. for me so I can test if that changes the behaviour?
It would also be good if someone could provide a minimal image with all non- needed drivers disabled per default incl. instructions how I can load them afterwards one by one (for example, let me know the modules to load and what they're used for).
So basically, starting with minimal features to see when it fails. Maybe further debug notices can be added so we see something on serial out just before the crash...?
As far as I know, Tony and notaz did a memory test run for hours, so for some reason, it didn't freeze on their boards.
Who would be willing to help me providing me files and guiding me through the tests?
Here are some ideas that could be the reason for the freezes:
* SD-Card issues - we're running the Linux off SD Card. Not sure it likes it when for example power breaks down or the driver does something wrong with the SD Card (though that should probably lead to a kernel panic).
* Issues switching between various voltages / frequencies (are all the voltages we're using in the dtb approved by TI?)
* RAM timing issues
* PALMAS power supply / timing issues. If PALMAS does something wrong, the system could easily freeze
The biggest issue is, that for the new PCBs both hardware and software have changed, so it's hard to compare it and rule out hardware / software issues.
Kernel 4.15 and old 2GB RAM hardware or uEVM works fine. Kernel 4.18 doesn't work stable on old OR new hardware and neither on uEVM.
It could be hardware, software - or a mixture of both. Well, software seems to be ONE reason at least, as 4.18 doesn't work on uEVM as well. But it means I can't properly test the new hardware right now - which is critical :)
What are your ideas, how can we approach this?
Thanks a lot in advance!
It might be worth your while logging free memory (in the temperature logging script), could be somethings leaking...
On 20/09/18 12:55, Michael Mrozek wrote:
I've got 22 Mainboard + CPU board combination (latest 4GB RAM versions) lying around at home and would like to ensure that the hardware is working fine - so I can place the order for the mass production.
However, I'm running into issues which I can't solve myself (as I am no developer and can't even do small changes to a kernel, etc.) I'm good with Linux, but not with developing - so I need some serious help here.
The unit I'm testing right now is one of latest batch.
Our own PyraOS usually freezes the system right after 30 seconds to 1 minute, so not really usable.
The latest "stable" Letux (kernel 4.18.8) with LXDE as base OS (installed using the makesd-tool) does boot fine. On two tries, it freezed after around 10 - 12 minutes sitting around idle. And about 2 - 3 minutes running the high-load script (which ups the CPU to 1,5GHz with some loop).
I will do more testings to see if it's always around the same time.
I've made a small script that logs power supply, uptime and temperature to see if it has something to do with that.
The result (incl. bootlog) is attached. I went to bed while it was running, but you can see in the log that it freezes after 12 minutes. Temperature is fine, voltage also seems to be fine and when using the high-load script, the temperature is higher after a few seconds and it still takes minutes to freeze - so we can assume that the freeze doesn't have anything to do with temperature CPU load.
Interestingly, when I came back, the frozen unit was still on (and showed the frozen desktop screen), but pretty warm - a lot warmer compared to it just idling.
I've added the new temperature and charging state at the end of freeze.txt as well.
Some things we know:
- Both askarus' and aTc's older Pyras (older Mainboard, 2GB RAM CPU board
without our optimizations) are stable with kernel 4.15. aTc mentioned he'd been running it off USB for days whereas askarus' Pyra runs 10 hours off battery without freezing and in normal usage (internet, LibreOffice, etc.) Unfortunately, the released kernel 4.15 doesn't support the new mainboards / CPU boards for some reason, so I can't compare them.
According to hns, even the uEVM freezes randomly with the latest kernel 4.18
These freezes don't seem to be connected with temperature or CPU load (both
Askarus and aTcs Pyras don't have any heat sink applied, whereas the 4GB RAM unit I've tested it on does).
- Sometimes the freeze shows random graphic glitches on the display, sometimes
the display simply freezes (memory errors?)
- The freeze might NOT be a full freeze! When using hns' QuantumStep OS (which
uses the LEDs as heartbeat), the unit also seemed to have frozen after a couple of minutes, as the LEDs stopped blinking. However, the LCD was still updating the graphics - VERY very slowly (1 character per 30 seconds on the terminal), but it was still working - even though it seemed frozen at first. This doesn't seem to be the case on LXDE, but maybe it's even slower because there's so much going on.
- On the unit I tested, sometimes u-Boot reported that it expected 4000000 of
memory size but only found 4. This might be related to the unit - or maybe some timings are still wrong for the 4GB RAM so there's some issue here?
- Running the uevm with little less than 5V increases the instability. As aTc
and askarus Pyra work fine with less than 5V, this shouldn't be a problem - but I thought I'd mention it here anyways.
So I'd like to find out the reason for these random freezes are. I've got various PCBs and cases, I can run tests - but as I can't code, I can't play around with kernel parameters or similar, so I'm pretty limited finding out what it could be.
That's where I need your help! Does anyone have any ideas what the issue could be and can provide different kernels, board files, etc. for me so I can test if that changes the behaviour?
It would also be good if someone could provide a minimal image with all non- needed drivers disabled per default incl. instructions how I can load them afterwards one by one (for example, let me know the modules to load and what they're used for).
So basically, starting with minimal features to see when it fails. Maybe further debug notices can be added so we see something on serial out just before the crash...?
As far as I know, Tony and notaz did a memory test run for hours, so for some reason, it didn't freeze on their boards.
Who would be willing to help me providing me files and guiding me through the tests?
Here are some ideas that could be the reason for the freezes:
- SD-Card issues - we're running the Linux off SD Card. Not sure it likes it
when for example power breaks down or the driver does something wrong with the SD Card (though that should probably lead to a kernel panic).
- Issues switching between various voltages / frequencies (are all the voltages
we're using in the dtb approved by TI?)
RAM timing issues
PALMAS power supply / timing issues. If PALMAS does something wrong, the
system could easily freeze
The biggest issue is, that for the new PCBs both hardware and software have changed, so it's hard to compare it and rule out hardware / software issues.
Kernel 4.15 and old 2GB RAM hardware or uEVM works fine. Kernel 4.18 doesn't work stable on old OR new hardware and neither on uEVM.
It could be hardware, software - or a mixture of both. Well, software seems to be ONE reason at least, as 4.18 doesn't work on uEVM as well. But it means I can't properly test the new hardware right now - which is critical :)
What are your ideas, how can we approach this?
Thanks a lot in advance!
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Hi, some additions below.
Am 20.09.2018 um 13:55 schrieb Michael Mrozek EvilDragon@openpandora.org:
I've got 22 Mainboard + CPU board combination (latest 4GB RAM versions) lying around at home and would like to ensure that the hardware is working fine - so I can place the order for the mass production.
However, I'm running into issues which I can't solve myself (as I am no developer and can't even do small changes to a kernel, etc.) I'm good with Linux, but not with developing - so I need some serious help here.
The unit I'm testing right now is one of latest batch.
Our own PyraOS usually freezes the system right after 30 seconds to 1 minute, so not really usable.
The latest "stable" Letux (kernel 4.18.8) with LXDE as base OS (installed using the makesd-tool) does boot fine. On two tries, it freezed after around 10 - 12 minutes sitting around idle. And about 2 - 3 minutes running the high-load script (which ups the CPU to 1,5GHz with some loop).
I will do more testings to see if it's always around the same time.
I've made a small script that logs power supply, uptime and temperature to see if it has something to do with that.
The result (incl. bootlog) is attached. I went to bed while it was running, but you can see in the log that it freezes after 12 minutes. Temperature is fine, voltage also seems to be fine and when using the high-load script, the temperature is higher after a few seconds and it still takes minutes to freeze - so we can assume that the freeze doesn't have anything to do with temperature CPU load.
Interestingly, when I came back, the frozen unit was still on (and showed the frozen desktop screen), but pretty warm - a lot warmer compared to it just idling.
I've added the new temperature and charging state at the end of freeze.txt as well.
Some things we know:
- Both askarus' and aTc's older Pyras (older Mainboard, 2GB RAM CPU board
without our optimizations) are stable with kernel 4.15. aTc mentioned he'd been running it off USB for days whereas askarus' Pyra runs 10 hours off battery without freezing and in normal usage (internet, LibreOffice, etc.) Unfortunately, the released kernel 4.15 doesn't support the new mainboards / CPU boards for some reason, so I can't compare them.
- According to hns, even the uEVM freezes randomly with the latest kernel 4.18
Not only 4.18. I also tried several archived system binaries back to 4.9 and both EVMs did freeze. What I did not do is to combine older u-boot, older kernel and older rootfs binaries.
It is very unclear to me what the factors are. At some days I start it in the morning and have to reboot 3 or 4 times. Then it freezes when accessing SD card or USB. Or just when not looking at it. After reset it runs fine for hours - to freeze again unexpectedly.
- These freezes don't seem to be connected with temperature or CPU load (both
Askarus and aTcs Pyras don't have any heat sink applied, whereas the 4GB RAM unit I've tested it on does).
- Sometimes the freeze shows random graphic glitches on the display, sometimes
the display simply freezes (memory errors?)
- The freeze might NOT be a full freeze! When using hns' QuantumStep OS (which
uses the LEDs as heartbeat), the unit also seemed to have frozen after a couple of minutes, as the LEDs stopped blinking. However, the LCD was still updating the graphics - VERY very slowly (1 character per 30 seconds on the terminal), but it was still working - even though it seemed frozen at first. This doesn't seem to be the case on LXDE, but maybe it's even slower because there's so much going on.
- On the unit I tested, sometimes u-Boot reported that it expected 4000000 of
memory size but only found 4. This might be related to the unit - or maybe some timings are still wrong for the 4GB RAM so there's some issue here?
- Running the uevm with little less than 5V increases the instability. As aTc
and askarus Pyra work fine with less than 5V, this shouldn't be a problem - but I thought I'd mention it here anyways.
So I'd like to find out the reason for these random freezes are. I've got various PCBs and cases, I can run tests - but as I can't code, I can't play around with kernel parameters or similar, so I'm pretty limited finding out what it could be.
That's where I need your help! Does anyone have any ideas what the issue could be and can provide different kernels, board files, etc. for me so I can test if that changes the behaviour?
It would also be good if someone could provide a minimal image with all non- needed drivers disabled per default incl. instructions how I can load them afterwards one by one (for example, let me know the modules to load and what they're used for).
So basically, starting with minimal features to see when it fails. Maybe further debug notices can be added so we see something on serial out just before the crash...?
As far as I know, Tony and notaz did a memory test run for hours, so for some reason, it didn't freeze on their boards.
Who would be willing to help me providing me files and guiding me through the tests?
Here are some ideas that could be the reason for the freezes:
- SD-Card issues - we're running the Linux off SD Card. Not sure it likes it
when for example power breaks down or the driver does something wrong with the SD Card (though that should probably lead to a kernel panic).
- Issues switching between various voltages / frequencies (are all the voltages
we're using in the dtb approved by TI?)
I have one kernel patch that restricts to 1GHz and official OPP voltage only, but it also freezes from time to time.
RAM timing issues
PALMAS power supply / timing issues. If PALMAS does something wrong, the
system could easily freeze
The biggest issue is, that for the new PCBs both hardware and software have changed, so it's hard to compare it and rule out hardware / software issues.
Kernel 4.15 and old 2GB RAM hardware or uEVM works fine. Kernel 4.18 doesn't work stable on old OR new hardware and neither on uEVM.
For me, any old/new kernel works stable on 2GB RAM but not any more on uEVM (it was rock solid up to ca. 6 months ago) or 4GB RAM.
With one exception: when I tried to analyse the "reboot" issue I had pwm_bl and the dmtimer modules disabled. So there was *no* backlight. And in that setup the 2GB Pyra did freeze several times just sitting at the login: promt.
Although that is anecdotical evidence only, these problems might be related. Tero has made a suggestion about the NRESWARM line. I have started analysis but not yet finished.
It could be hardware, software - or a mixture of both. Well, software seems to be ONE reason at least, as 4.18 doesn't work on uEVM as well. But it means I can't properly test the new hardware right now - which is critical :)
What are your ideas, how can we approach this?
Thanks a lot in advance!
-- Greetings,
Michael Mrozek
OpenPandora GmbH CEO: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Germany Tel.: +49 841 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org <freeze.txt>_______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Am Donnerstag, den 20.09.2018, 14:17 +0200 schrieb H. Nikolaus Schaller:
Some addition from aTc:
He ran the latest 4.18.8 kernel with LXDE from your makesd script on his old 2GB unit.
It ran for about 23 minutes and then shut itself off (because the battery was empty).
So it seems it IS running a lot more stable there - at least according to a quick test. He needs to charge the battery first for more tests.
That might pinpoint us back to the 4GB RAM. Are all timings / settings for that one correct in our DTB / U-Boot / Kernel?
@Tony: Could you check that? I guess you compiled your own stuff when testing the 4GB RAM stuff, so maybe we've missed something essential here?
Hi, some additions below.
Am 20.09.2018 um 13:55 schrieb Michael Mrozek EvilDragon@openpandora.org:
I've got 22 Mainboard + CPU board combination (latest 4GB RAM versions) lying around at home and would like to ensure that the hardware is working fine - so I can place the order for the mass production.
However, I'm running into issues which I can't solve myself (as I am no developer and can't even do small changes to a kernel, etc.) I'm good with Linux, but not with developing - so I need some serious help here.
The unit I'm testing right now is one of latest batch.
Our own PyraOS usually freezes the system right after 30 seconds to 1 minute, so not really usable.
The latest "stable" Letux (kernel 4.18.8) with LXDE as base OS (installed using the makesd-tool) does boot fine. On two tries, it freezed after around 10 - 12 minutes sitting around idle. And about 2 - 3 minutes running the high-load script (which ups the CPU to 1,5GHz with some loop).
I will do more testings to see if it's always around the same time.
I've made a small script that logs power supply, uptime and temperature to see if it has something to do with that.
The result (incl. bootlog) is attached. I went to bed while it was running, but you can see in the log that it freezes after 12 minutes. Temperature is fine, voltage also seems to be fine and when using the high-load script, the temperature is higher after a few seconds and it still takes minutes to freeze - so we can assume that the freeze doesn't have anything to do with temperature CPU load.
Interestingly, when I came back, the frozen unit was still on (and showed the frozen desktop screen), but pretty warm - a lot warmer compared to it just idling.
I've added the new temperature and charging state at the end of freeze.txt as well.
Some things we know:
- Both askarus' and aTc's older Pyras (older Mainboard, 2GB RAM CPU board
without our optimizations) are stable with kernel 4.15. aTc mentioned he'd been running it off USB for days whereas askarus' Pyra runs 10 hours off battery without freezing and in normal usage (internet, LibreOffice, etc.) Unfortunately, the released kernel 4.15 doesn't support the new mainboards / CPU boards for some reason, so I can't compare them.
- According to hns, even the uEVM freezes randomly with the latest kernel
4.18
Not only 4.18. I also tried several archived system binaries back to 4.9 and both EVMs did freeze. What I did not do is to combine older u-boot, older kernel and older rootfs binaries.
It is very unclear to me what the factors are. At some days I start it in the morning and have to reboot 3 or 4 times. Then it freezes when accessing SD card or USB. Or just when not looking at it. After reset it runs fine for hours - to freeze again unexpectedly.
- These freezes don't seem to be connected with temperature or CPU load
(both Askarus and aTcs Pyras don't have any heat sink applied, whereas the 4GB RAM unit I've tested it on does).
- Sometimes the freeze shows random graphic glitches on the display,
sometimes the display simply freezes (memory errors?)
- The freeze might NOT be a full freeze! When using hns' QuantumStep OS
(which uses the LEDs as heartbeat), the unit also seemed to have frozen after a couple of minutes, as the LEDs stopped blinking. However, the LCD was still updating the graphics - VERY very slowly (1 character per 30 seconds on the terminal), but it was still working - even though it seemed frozen at first. This doesn't seem to be the case on LXDE, but maybe it's even slower because there's so much going on.
- On the unit I tested, sometimes u-Boot reported that it expected 4000000
of memory size but only found 4. This might be related to the unit - or maybe some timings are still wrong for the 4GB RAM so there's some issue here?
- Running the uevm with little less than 5V increases the instability. As
aTc and askarus Pyra work fine with less than 5V, this shouldn't be a problem - but I thought I'd mention it here anyways.
So I'd like to find out the reason for these random freezes are. I've got various PCBs and cases, I can run tests - but as I can't code, I can't play around with kernel parameters or similar, so I'm pretty limited finding out what it could be.
That's where I need your help! Does anyone have any ideas what the issue could be and can provide different kernels, board files, etc. for me so I can test if that changes the behaviour?
It would also be good if someone could provide a minimal image with all non- needed drivers disabled per default incl. instructions how I can load them afterwards one by one (for example, let me know the modules to load and what they're used for).
So basically, starting with minimal features to see when it fails. Maybe further debug notices can be added so we see something on serial out just before the crash...?
As far as I know, Tony and notaz did a memory test run for hours, so for some reason, it didn't freeze on their boards.
Who would be willing to help me providing me files and guiding me through the tests?
Here are some ideas that could be the reason for the freezes:
- SD-Card issues - we're running the Linux off SD Card. Not sure it likes it
when for example power breaks down or the driver does something wrong with the SD Card (though that should probably lead to a kernel panic).
- Issues switching between various voltages / frequencies (are all the
voltages we're using in the dtb approved by TI?)
I have one kernel patch that restricts to 1GHz and official OPP voltage only, but it also freezes from time to time.
RAM timing issues
PALMAS power supply / timing issues. If PALMAS does something wrong, the
system could easily freeze
The biggest issue is, that for the new PCBs both hardware and software have changed, so it's hard to compare it and rule out hardware / software issues.
Kernel 4.15 and old 2GB RAM hardware or uEVM works fine. Kernel 4.18 doesn't work stable on old OR new hardware and neither on uEVM.
For me, any old/new kernel works stable on 2GB RAM but not any more on uEVM (it was rock solid up to ca. 6 months ago) or 4GB RAM.
With one exception: when I tried to analyse the "reboot" issue I had pwm_bl and the dmtimer modules disabled. So there was *no* backlight. And in that setup the 2GB Pyra did freeze several times just sitting at the login: promt.
Although that is anecdotical evidence only, these problems might be related. Tero has made a suggestion about the NRESWARM line. I have started analysis but not yet finished.
It could be hardware, software - or a mixture of both. Well, software seems to be ONE reason at least, as 4.18 doesn't work on uEVM as well. But it means I can't properly test the new hardware right now - which is critical :)
What are your ideas, how can we approach this?
Thanks a lot in advance!
-- Greetings,
Michael Mrozek
OpenPandora GmbH CEO: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Germany Tel.: +49 841 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org <freeze.txt>_______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
I'm traveling now and ED sent me a new board just before I left, and unfortunately it's unusable because of these random hangs. I wasn't able to correlate these hangs to anything too, it just hangs in like 1-10 minutes. Most often it starts showing garbage suggesting RAM issues. I'm using 4.15 for working tiler, with adjusted dts for the new board. Nubs and sesors are not responding on that board too.
Pyra would have helped a lot during the travel, but unfortunately it's just dead weight I have to carry around.
notaz
On Thu, 20 Sep 2018, 20:55 Michael Mrozek, EvilDragon@openpandora.org wrote:
I've got 22 Mainboard + CPU board combination (latest 4GB RAM versions) lying around at home and would like to ensure that the hardware is working fine
- so I
can place the order for the mass production.
However, I'm running into issues which I can't solve myself (as I am no developer and can't even do small changes to a kernel, etc.) I'm good with Linux, but not with developing - so I need some serious help here.
The unit I'm testing right now is one of latest batch.
Our own PyraOS usually freezes the system right after 30 seconds to 1 minute, so not really usable.
The latest "stable" Letux (kernel 4.18.8) with LXDE as base OS (installed using the makesd-tool) does boot fine. On two tries, it freezed after around 10
- 12
minutes sitting around idle. And about 2 - 3 minutes running the high-load script (which ups the CPU to 1,5GHz with some loop).
I will do more testings to see if it's always around the same time.
I've made a small script that logs power supply, uptime and temperature to see if it has something to do with that.
The result (incl. bootlog) is attached. I went to bed while it was running, but you can see in the log that it freezes after 12 minutes. Temperature is fine, voltage also seems to be fine and when using the high-load script, the temperature is higher after a few seconds and it still takes minutes to freeze - so we can assume that the freeze doesn't have anything to do with temperature CPU load.
Interestingly, when I came back, the frozen unit was still on (and showed the frozen desktop screen), but pretty warm - a lot warmer compared to it just idling.
I've added the new temperature and charging state at the end of freeze.txt as well.
Some things we know:
- Both askarus' and aTc's older Pyras (older Mainboard, 2GB RAM CPU board
without our optimizations) are stable with kernel 4.15. aTc mentioned he'd been running it off USB for days whereas askarus' Pyra runs 10 hours off battery without freezing and in normal usage (internet, LibreOffice, etc.) Unfortunately, the released kernel 4.15 doesn't support the new mainboards / CPU boards for some reason, so I can't compare them.
- According to hns, even the uEVM freezes randomly with the latest kernel
4.18
- These freezes don't seem to be connected with temperature or CPU load
(both Askarus and aTcs Pyras don't have any heat sink applied, whereas the 4GB RAM unit I've tested it on does).
- Sometimes the freeze shows random graphic glitches on the display,
sometimes the display simply freezes (memory errors?)
- The freeze might NOT be a full freeze! When using hns' QuantumStep OS
(which uses the LEDs as heartbeat), the unit also seemed to have frozen after a couple of minutes, as the LEDs stopped blinking. However, the LCD was still updating the graphics - VERY very slowly (1 character per 30 seconds on the terminal), but it was still working - even though it seemed frozen at first. This doesn't seem to be the case on LXDE, but maybe it's even slower because there's so much going on.
- On the unit I tested, sometimes u-Boot reported that it expected 4000000
of memory size but only found 4. This might be related to the unit - or maybe some timings are still wrong for the 4GB RAM so there's some issue here?
- Running the uevm with little less than 5V increases the instability. As
aTc and askarus Pyra work fine with less than 5V, this shouldn't be a problem
- but
I thought I'd mention it here anyways.
So I'd like to find out the reason for these random freezes are. I've got various PCBs and cases, I can run tests - but as I can't code, I can't play around with kernel parameters or similar, so I'm pretty limited finding out what it could be.
That's where I need your help! Does anyone have any ideas what the issue could be and can provide different kernels, board files, etc. for me so I can test if that changes the behaviour?
It would also be good if someone could provide a minimal image with all non- needed drivers disabled per default incl. instructions how I can load them afterwards one by one (for example, let me know the modules to load and what they're used for).
So basically, starting with minimal features to see when it fails. Maybe further debug notices can be added so we see something on serial out just before the crash...?
As far as I know, Tony and notaz did a memory test run for hours, so for some reason, it didn't freeze on their boards.
Who would be willing to help me providing me files and guiding me through the tests?
Here are some ideas that could be the reason for the freezes:
- SD-Card issues - we're running the Linux off SD Card. Not sure it likes
it when for example power breaks down or the driver does something wrong with the SD Card (though that should probably lead to a kernel panic).
- Issues switching between various voltages / frequencies (are all the
voltages we're using in the dtb approved by TI?)
RAM timing issues
PALMAS power supply / timing issues. If PALMAS does something wrong, the
system could easily freeze
The biggest issue is, that for the new PCBs both hardware and software have changed, so it's hard to compare it and rule out hardware / software issues.
Kernel 4.15 and old 2GB RAM hardware or uEVM works fine. Kernel 4.18 doesn't work stable on old OR new hardware and neither on uEVM.
It could be hardware, software - or a mixture of both. Well, software seems to be ONE reason at least, as 4.18 doesn't work on uEVM as well. But it means I can't properly test the new hardware right now - which is critical :)
What are your ideas, how can we approach this?
Thanks a lot in advance!
-- Greetings,
Michael Mrozek
OpenPandora GmbH CEO: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Germany Tel.: +49 841 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org _______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Am Donnerstag, den 20.09.2018, 21:49 +0900 schrieb Grazvydas Ignotas:
Hi,
that's bad news :( AFAIR, you were running memtester on the new 4GB RAM PCB I've sent you earlier as well for a few hours.
Were you using a different kernel / dtb here for the test? Because it's very weird that the PCB survives a memtest for both Tony and you but then has issues when simply idling around.
What's different here?
I'm traveling now and ED sent me a new board just before I left, and unfortunately it's unusable because of these random hangs. I wasn't able to correlate these hangs to anything too, it just hangs in like 1-10 minutes. Most often it starts showing garbage suggesting RAM issues. I'm using 4.15 for working tiler, with adjusted dts for the new board. Nubs and sesors are not responding on that board too.
Pyra would have helped a lot during the travel, but unfortunately it's just dead weight I have to carry around.
notaz
On Thu, 20 Sep 2018, 20:55 Michael Mrozek, EvilDragon@openpandora.org wrote:
I've got 22 Mainboard + CPU board combination (latest 4GB RAM versions) lying around at home and would like to ensure that the hardware is working fine
- so I
can place the order for the mass production.
However, I'm running into issues which I can't solve myself (as I am no developer and can't even do small changes to a kernel, etc.) I'm good with Linux, but not with developing - so I need some serious help here.
The unit I'm testing right now is one of latest batch.
Our own PyraOS usually freezes the system right after 30 seconds to 1 minute, so not really usable.
The latest "stable" Letux (kernel 4.18.8) with LXDE as base OS (installed using the makesd-tool) does boot fine. On two tries, it freezed after around 10
- 12
minutes sitting around idle. And about 2 - 3 minutes running the high-load script (which ups the CPU to 1,5GHz with some loop).
I will do more testings to see if it's always around the same time.
I've made a small script that logs power supply, uptime and temperature to see if it has something to do with that.
The result (incl. bootlog) is attached. I went to bed while it was running, but you can see in the log that it freezes after 12 minutes. Temperature is fine, voltage also seems to be fine and when using the high-load script, the temperature is higher after a few seconds and it still takes minutes to freeze - so we can assume that the freeze doesn't have anything to do with temperature CPU load.
Interestingly, when I came back, the frozen unit was still on (and showed the frozen desktop screen), but pretty warm - a lot warmer compared to it just idling.
I've added the new temperature and charging state at the end of freeze.txt as well.
Some things we know:
- Both askarus' and aTc's older Pyras (older Mainboard, 2GB RAM CPU board
without our optimizations) are stable with kernel 4.15. aTc mentioned he'd been running it off USB for days whereas askarus' Pyra runs 10 hours off battery without freezing and in normal usage (internet, LibreOffice, etc.) Unfortunately, the released kernel 4.15 doesn't support the new mainboards / CPU boards for some reason, so I can't compare them.
- According to hns, even the uEVM freezes randomly with the latest kernel
4.18
- These freezes don't seem to be connected with temperature or CPU load
(both Askarus and aTcs Pyras don't have any heat sink applied, whereas the 4GB RAM unit I've tested it on does).
- Sometimes the freeze shows random graphic glitches on the display,
sometimes the display simply freezes (memory errors?)
- The freeze might NOT be a full freeze! When using hns' QuantumStep OS
(which uses the LEDs as heartbeat), the unit also seemed to have frozen after a couple of minutes, as the LEDs stopped blinking. However, the LCD was still updating the graphics - VERY very slowly (1 character per 30 seconds on the terminal), but it was still working - even though it seemed frozen at first. This doesn't seem to be the case on LXDE, but maybe it's even slower because there's so much going on.
- On the unit I tested, sometimes u-Boot reported that it expected 4000000
of memory size but only found 4. This might be related to the unit - or maybe some timings are still wrong for the 4GB RAM so there's some issue here?
- Running the uevm with little less than 5V increases the instability. As
aTc and askarus Pyra work fine with less than 5V, this shouldn't be a problem
- but
I thought I'd mention it here anyways.
So I'd like to find out the reason for these random freezes are. I've got various PCBs and cases, I can run tests - but as I can't code, I can't play around with kernel parameters or similar, so I'm pretty limited finding out what it could be.
That's where I need your help! Does anyone have any ideas what the issue could be and can provide different kernels, board files, etc. for me so I can test if that changes the behaviour?
It would also be good if someone could provide a minimal image with all non- needed drivers disabled per default incl. instructions how I can load them afterwards one by one (for example, let me know the modules to load and what they're used for).
So basically, starting with minimal features to see when it fails. Maybe further debug notices can be added so we see something on serial out just before the crash...?
As far as I know, Tony and notaz did a memory test run for hours, so for some reason, it didn't freeze on their boards.
Who would be willing to help me providing me files and guiding me through the tests?
Here are some ideas that could be the reason for the freezes:
- SD-Card issues - we're running the Linux off SD Card. Not sure it likes
it when for example power breaks down or the driver does something wrong with the SD Card (though that should probably lead to a kernel panic).
- Issues switching between various voltages / frequencies (are all the
voltages we're using in the dtb approved by TI?)
RAM timing issues
PALMAS power supply / timing issues. If PALMAS does something wrong, the
system could easily freeze
The biggest issue is, that for the new PCBs both hardware and software have changed, so it's hard to compare it and rule out hardware / software issues.
Kernel 4.15 and old 2GB RAM hardware or uEVM works fine. Kernel 4.18 doesn't work stable on old OR new hardware and neither on uEVM.
It could be hardware, software - or a mixture of both. Well, software seems to be ONE reason at least, as 4.18 doesn't work on uEVM as well. But it means I can't properly test the new hardware right now - which is critical :)
What are your ideas, how can we approach this?
Thanks a lot in advance!
-- Greetings,
Michael Mrozek
OpenPandora GmbH CEO: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Germany Tel.: +49 841 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org _______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
It was the older 4gb board (I have 2 now) and different dts, remaining things should be the same I think (same SD). I intentionally tried to minimize changes from my working setup before leaving for the trip. Before reading this thread I just assumed my new board is faulty, but now it's clear more testing is needed. One obvious difference from the older board is the heatsink and some sort of new shielding (?), could it be messing something up like shorting or reflecting high freq signals?
On Thu, 20 Sep 2018, 21:56 Michael Mrozek, EvilDragon@openpandora.org wrote:
Am Donnerstag, den 20.09.2018, 21:49 +0900 schrieb Grazvydas Ignotas:
Hi,
that's bad news :( AFAIR, you were running memtester on the new 4GB RAM PCB I've sent you earlier as well for a few hours.
Were you using a different kernel / dtb here for the test? Because it's very weird that the PCB survives a memtest for both Tony and you but then has issues when simply idling around.
What's different here?
I'm traveling now and ED sent me a new board just before I left, and unfortunately it's unusable because of these random hangs. I wasn't able
to
correlate these hangs to anything too, it just hangs in like 1-10
minutes.
Most often it starts showing garbage suggesting RAM issues. I'm using
4.15
for working tiler, with adjusted dts for the new board. Nubs and sesors
are
not responding on that board too.
Pyra would have helped a lot during the travel, but unfortunately it's
just
dead weight I have to carry around.
notaz
On Thu, 20 Sep 2018, 20:55 Michael Mrozek, EvilDragon@openpandora.org wrote:
I've got 22 Mainboard + CPU board combination (latest 4GB RAM versions) lying around at home and would like to ensure that the hardware is working
fine
- so I
can place the order for the mass production.
However, I'm running into issues which I can't solve myself (as I am no developer and can't even do small changes to a kernel, etc.) I'm good with Linux, but not with developing - so I need some serious
help
here.
The unit I'm testing right now is one of latest batch.
Our own PyraOS usually freezes the system right after 30 seconds to 1 minute, so not really usable.
The latest "stable" Letux (kernel 4.18.8) with LXDE as base OS
(installed
using the makesd-tool) does boot fine. On two tries, it freezed after around
10
- 12
minutes sitting around idle. And about 2 - 3 minutes running the
high-load
script (which ups the CPU to 1,5GHz with some loop).
I will do more testings to see if it's always around the same time.
I've made a small script that logs power supply, uptime and
temperature to
see if it has something to do with that.
The result (incl. bootlog) is attached. I went to bed while it was running, but you can see in the log that it freezes after 12 minutes. Temperature is fine, voltage also seems to be fine
and
when using the high-load script, the temperature is higher after a few
seconds
and it still takes minutes to freeze - so we can assume that the freeze
doesn't
have anything to do with temperature CPU load.
Interestingly, when I came back, the frozen unit was still on (and
showed
the frozen desktop screen), but pretty warm - a lot warmer compared to it
just
idling.
I've added the new temperature and charging state at the end of
freeze.txt
as well.
Some things we know:
- Both askarus' and aTc's older Pyras (older Mainboard, 2GB RAM CPU
board
without our optimizations) are stable with kernel 4.15. aTc mentioned
he'd
been running it off USB for days whereas askarus' Pyra runs 10 hours off
battery
without freezing and in normal usage (internet, LibreOffice, etc.) Unfortunately, the released kernel 4.15 doesn't support the new
mainboards
/ CPU boards for some reason, so I can't compare them.
- According to hns, even the uEVM freezes randomly with the latest
kernel
4.18
- These freezes don't seem to be connected with temperature or CPU load
(both Askarus and aTcs Pyras don't have any heat sink applied, whereas the
4GB
RAM unit I've tested it on does).
- Sometimes the freeze shows random graphic glitches on the display,
sometimes the display simply freezes (memory errors?)
- The freeze might NOT be a full freeze! When using hns' QuantumStep OS
(which uses the LEDs as heartbeat), the unit also seemed to have frozen after
a
couple of minutes, as the LEDs stopped blinking. However, the LCD was still updating the graphics - VERY very slowly (1 character per 30 seconds on the terminal), but it was still working - even though it seemed frozen at first. This doesn't seem to be the case on LXDE, but maybe it's even slower because there's so much going on.
- On the unit I tested, sometimes u-Boot reported that it expected
4000000
of memory size but only found 4. This might be related to the unit - or
maybe
some timings are still wrong for the 4GB RAM so there's some issue here?
- Running the uevm with little less than 5V increases the instability.
As
aTc and askarus Pyra work fine with less than 5V, this shouldn't be a
problem
- but
I thought I'd mention it here anyways.
So I'd like to find out the reason for these random freezes are. I've got various PCBs and cases, I can run tests - but as I can't
code, I
can't play around with kernel parameters or similar, so I'm pretty limited finding out what it could be.
That's where I need your help! Does anyone have any ideas what the issue could be and can provide different kernels, board files, etc. for me so I can test if that changes the behaviour?
It would also be good if someone could provide a minimal image with all non- needed drivers disabled per default incl. instructions how I can load
them
afterwards one by one (for example, let me know the modules to load and what they're used for).
So basically, starting with minimal features to see when it fails. Maybe further debug notices can be added so we see something on serial
out
just before the crash...?
As far as I know, Tony and notaz did a memory test run for hours, so
for
some reason, it didn't freeze on their boards.
Who would be willing to help me providing me files and guiding me
through
the tests?
Here are some ideas that could be the reason for the freezes:
- SD-Card issues - we're running the Linux off SD Card. Not sure it
likes
it when for example power breaks down or the driver does something wrong
with
the SD Card (though that should probably lead to a kernel panic).
- Issues switching between various voltages / frequencies (are all the
voltages we're using in the dtb approved by TI?)
RAM timing issues
PALMAS power supply / timing issues. If PALMAS does something wrong,
the
system could easily freeze
The biggest issue is, that for the new PCBs both hardware and software
have
changed, so it's hard to compare it and rule out hardware / software issues.
Kernel 4.15 and old 2GB RAM hardware or uEVM works fine. Kernel 4.18 doesn't work stable on old OR new hardware and neither on
uEVM.
It could be hardware, software - or a mixture of both. Well, software seems to be ONE reason at least, as 4.18 doesn't work on uEVM as well. But it means I can't properly test the new hardware right now - which
is
critical :)
What are your ideas, how can we approach this?
Thanks a lot in advance!
-- Greetings,
Michael Mrozek
OpenPandora GmbH CEO: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Germany Tel.: +49 841 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org _______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
Am Donnerstag, den 20.09.2018, 22:09 +0900 schrieb Grazvydas Ignotas:
Hi,
now THAT is interesting. The old 4GB RAM board was working - whereas the new one fails? Hmm... I'll test if the cooling makes any difference here.
Do you happen to have the dts for the old 4GB board somewhere I could download? I've still got a few of the old 4GB RAM boards here, so I could check those as well.
I guess you also used the older mainboard for the RAM tester? @hns: Could that be an issue as well? Maybe regarding power supply / stability?
It was the older 4gb board (I have 2 now) and different dts, remaining things should be the same I think (same SD). I intentionally tried to minimize changes from my working setup before leaving for the trip. Before reading this thread I just assumed my new board is faulty, but now it's clear more testing is needed. One obvious difference from the older board is the heatsink and some sort of new shielding (?), could it be messing something up like shorting or reflecting high freq signals?
On Thu, 20 Sep 2018, 21:56 Michael Mrozek, EvilDragon@openpandora.org wrote:
Am Donnerstag, den 20.09.2018, 21:49 +0900 schrieb Grazvydas Ignotas:
Hi,
that's bad news :( AFAIR, you were running memtester on the new 4GB RAM PCB I've sent you earlier as well for a few hours.
Were you using a different kernel / dtb here for the test? Because it's very weird that the PCB survives a memtest for both Tony and you but then has issues when simply idling around.
What's different here?
I'm traveling now and ED sent me a new board just before I left, and unfortunately it's unusable because of these random hangs. I wasn't able
to
correlate these hangs to anything too, it just hangs in like 1-10
minutes.
Most often it starts showing garbage suggesting RAM issues. I'm using
4.15
for working tiler, with adjusted dts for the new board. Nubs and sesors
are
not responding on that board too.
Pyra would have helped a lot during the travel, but unfortunately it's
just
dead weight I have to carry around.
notaz
On Thu, 20 Sep 2018, 20:55 Michael Mrozek, EvilDragon@openpandora.org wrote:
I've got 22 Mainboard + CPU board combination (latest 4GB RAM versions) lying around at home and would like to ensure that the hardware is working
fine
- so I
can place the order for the mass production.
However, I'm running into issues which I can't solve myself (as I am no developer and can't even do small changes to a kernel, etc.) I'm good with Linux, but not with developing - so I need some serious
help
here.
The unit I'm testing right now is one of latest batch.
Our own PyraOS usually freezes the system right after 30 seconds to 1 minute, so not really usable.
The latest "stable" Letux (kernel 4.18.8) with LXDE as base OS
(installed
using the makesd-tool) does boot fine. On two tries, it freezed after around
10
- 12
minutes sitting around idle. And about 2 - 3 minutes running the
high-load
script (which ups the CPU to 1,5GHz with some loop).
I will do more testings to see if it's always around the same time.
I've made a small script that logs power supply, uptime and
temperature to
see if it has something to do with that.
The result (incl. bootlog) is attached. I went to bed while it was running, but you can see in the log that it freezes after 12 minutes. Temperature is fine, voltage also seems to be fine
and
when using the high-load script, the temperature is higher after a few
seconds
and it still takes minutes to freeze - so we can assume that the freeze
doesn't
have anything to do with temperature CPU load.
Interestingly, when I came back, the frozen unit was still on (and
showed
the frozen desktop screen), but pretty warm - a lot warmer compared to it
just
idling.
I've added the new temperature and charging state at the end of
freeze.txt
as well.
Some things we know:
- Both askarus' and aTc's older Pyras (older Mainboard, 2GB RAM CPU
board
without our optimizations) are stable with kernel 4.15. aTc mentioned
he'd
been running it off USB for days whereas askarus' Pyra runs 10 hours off
battery
without freezing and in normal usage (internet, LibreOffice, etc.) Unfortunately, the released kernel 4.15 doesn't support the new
mainboards
/ CPU boards for some reason, so I can't compare them.
- According to hns, even the uEVM freezes randomly with the latest
kernel
4.18
- These freezes don't seem to be connected with temperature or CPU load
(both Askarus and aTcs Pyras don't have any heat sink applied, whereas the
4GB
RAM unit I've tested it on does).
- Sometimes the freeze shows random graphic glitches on the display,
sometimes the display simply freezes (memory errors?)
- The freeze might NOT be a full freeze! When using hns' QuantumStep OS
(which uses the LEDs as heartbeat), the unit also seemed to have frozen after
a
couple of minutes, as the LEDs stopped blinking. However, the LCD was still updating the graphics - VERY very slowly (1 character per 30 seconds on the terminal), but it was still working - even though it seemed frozen at first. This doesn't seem to be the case on LXDE, but maybe it's even slower because there's so much going on.
- On the unit I tested, sometimes u-Boot reported that it expected
4000000
of memory size but only found 4. This might be related to the unit - or
maybe
some timings are still wrong for the 4GB RAM so there's some issue here?
- Running the uevm with little less than 5V increases the instability.
As
aTc and askarus Pyra work fine with less than 5V, this shouldn't be a
problem
- but
I thought I'd mention it here anyways.
So I'd like to find out the reason for these random freezes are. I've got various PCBs and cases, I can run tests - but as I can't
code, I
can't play around with kernel parameters or similar, so I'm pretty limited finding out what it could be.
That's where I need your help! Does anyone have any ideas what the issue could be and can provide different kernels, board files, etc. for me so I can test if that changes the behaviour?
It would also be good if someone could provide a minimal image with all non- needed drivers disabled per default incl. instructions how I can load
them
afterwards one by one (for example, let me know the modules to load and what they're used for).
So basically, starting with minimal features to see when it fails. Maybe further debug notices can be added so we see something on serial
out
just before the crash...?
As far as I know, Tony and notaz did a memory test run for hours, so
for
some reason, it didn't freeze on their boards.
Who would be willing to help me providing me files and guiding me
through
the tests?
Here are some ideas that could be the reason for the freezes:
- SD-Card issues - we're running the Linux off SD Card. Not sure it
likes
it when for example power breaks down or the driver does something wrong
with
the SD Card (though that should probably lead to a kernel panic).
- Issues switching between various voltages / frequencies (are all the
voltages we're using in the dtb approved by TI?)
RAM timing issues
PALMAS power supply / timing issues. If PALMAS does something wrong,
the
system could easily freeze
The biggest issue is, that for the new PCBs both hardware and software
have
changed, so it's hard to compare it and rule out hardware / software issues.
Kernel 4.15 and old 2GB RAM hardware or uEVM works fine. Kernel 4.18 doesn't work stable on old OR new hardware and neither on
uEVM.
It could be hardware, software - or a mixture of both. Well, software seems to be ONE reason at least, as 4.18 doesn't work on uEVM as well. But it means I can't properly test the new hardware right now - which
is
critical :)
What are your ideas, how can we approach this?
Thanks a lot in advance!
-- Greetings,
Michael Mrozek
OpenPandora GmbH CEO: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Germany Tel.: +49 841 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org _______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
Hi,
Am 20.09.2018 um 15:19 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Donnerstag, den 20.09.2018, 22:09 +0900 schrieb Grazvydas Ignotas:
Hi,
now THAT is interesting. The old 4GB RAM board was working - whereas the new one fails? Hmm... I'll test if the cooling makes any difference here.
Do you happen to have the dts for the old 4GB board somewhere I could download? I've still got a few of the old 4GB RAM boards here, so I could check those as well.
I guess you also used the older mainboard for the RAM tester? @hns: Could that be an issue as well? Maybe regarding power supply / stability?
Hm. Well, difficult to say.
neither u-boot nor kernel nor DTS modify any DDR3 (EMIF) timing. This all is defined by the SPL (MLO). But the exact timings might be different for each board.
BR, Nikolaus
It was the older 4gb board (I have 2 now) and different dts, remaining things should be the same I think (same SD). I intentionally tried to minimize changes from my working setup before leaving for the trip. Before reading this thread I just assumed my new board is faulty, but now it's clear more testing is needed. One obvious difference from the older board is the heatsink and some sort of new shielding (?), could it be messing something up like shorting or reflecting high freq signals?
On Thu, 20 Sep 2018, 21:56 Michael Mrozek, EvilDragon@openpandora.org wrote:
Am Donnerstag, den 20.09.2018, 21:49 +0900 schrieb Grazvydas Ignotas:
Hi,
that's bad news :( AFAIR, you were running memtester on the new 4GB RAM PCB I've sent you earlier as well for a few hours.
Were you using a different kernel / dtb here for the test? Because it's very weird that the PCB survives a memtest for both Tony and you but then has issues when simply idling around.
What's different here?
I'm traveling now and ED sent me a new board just before I left, and unfortunately it's unusable because of these random hangs. I wasn't able
to
correlate these hangs to anything too, it just hangs in like 1-10
minutes.
Most often it starts showing garbage suggesting RAM issues. I'm using
4.15
for working tiler, with adjusted dts for the new board. Nubs and sesors
are
not responding on that board too.
Pyra would have helped a lot during the travel, but unfortunately it's
just
dead weight I have to carry around.
notaz
On Thu, 20 Sep 2018, 20:55 Michael Mrozek, EvilDragon@openpandora.org wrote:
I've got 22 Mainboard + CPU board combination (latest 4GB RAM versions) lying around at home and would like to ensure that the hardware is working
fine
- so I
can place the order for the mass production.
However, I'm running into issues which I can't solve myself (as I am no developer and can't even do small changes to a kernel, etc.) I'm good with Linux, but not with developing - so I need some serious
help
here.
The unit I'm testing right now is one of latest batch.
Our own PyraOS usually freezes the system right after 30 seconds to 1 minute, so not really usable.
The latest "stable" Letux (kernel 4.18.8) with LXDE as base OS
(installed
using the makesd-tool) does boot fine. On two tries, it freezed after around
10
- 12
minutes sitting around idle. And about 2 - 3 minutes running the
high-load
script (which ups the CPU to 1,5GHz with some loop).
I will do more testings to see if it's always around the same time.
I've made a small script that logs power supply, uptime and
temperature to
see if it has something to do with that.
The result (incl. bootlog) is attached. I went to bed while it was running, but you can see in the log that it freezes after 12 minutes. Temperature is fine, voltage also seems to be fine
and
when using the high-load script, the temperature is higher after a few
seconds
and it still takes minutes to freeze - so we can assume that the freeze
doesn't
have anything to do with temperature CPU load.
Interestingly, when I came back, the frozen unit was still on (and
showed
the frozen desktop screen), but pretty warm - a lot warmer compared to it
just
idling.
I've added the new temperature and charging state at the end of
freeze.txt
as well.
Some things we know:
- Both askarus' and aTc's older Pyras (older Mainboard, 2GB RAM CPU
board
without our optimizations) are stable with kernel 4.15. aTc mentioned
he'd
been running it off USB for days whereas askarus' Pyra runs 10 hours off
battery
without freezing and in normal usage (internet, LibreOffice, etc.) Unfortunately, the released kernel 4.15 doesn't support the new
mainboards
/ CPU boards for some reason, so I can't compare them.
- According to hns, even the uEVM freezes randomly with the latest
kernel
4.18
- These freezes don't seem to be connected with temperature or CPU load
(both Askarus and aTcs Pyras don't have any heat sink applied, whereas the
4GB
RAM unit I've tested it on does).
- Sometimes the freeze shows random graphic glitches on the display,
sometimes the display simply freezes (memory errors?)
- The freeze might NOT be a full freeze! When using hns' QuantumStep OS
(which uses the LEDs as heartbeat), the unit also seemed to have frozen after
a
couple of minutes, as the LEDs stopped blinking. However, the LCD was still updating the graphics - VERY very slowly (1 character per 30 seconds on the terminal), but it was still working - even though it seemed frozen at first. This doesn't seem to be the case on LXDE, but maybe it's even slower because there's so much going on.
- On the unit I tested, sometimes u-Boot reported that it expected
4000000
of memory size but only found 4. This might be related to the unit - or
maybe
some timings are still wrong for the 4GB RAM so there's some issue here?
- Running the uevm with little less than 5V increases the instability.
As
aTc and askarus Pyra work fine with less than 5V, this shouldn't be a
problem
- but
I thought I'd mention it here anyways.
So I'd like to find out the reason for these random freezes are. I've got various PCBs and cases, I can run tests - but as I can't
code, I
can't play around with kernel parameters or similar, so I'm pretty limited finding out what it could be.
That's where I need your help! Does anyone have any ideas what the issue could be and can provide different kernels, board files, etc. for me so I can test if that changes the behaviour?
It would also be good if someone could provide a minimal image with all non- needed drivers disabled per default incl. instructions how I can load
them
afterwards one by one (for example, let me know the modules to load and what they're used for).
So basically, starting with minimal features to see when it fails. Maybe further debug notices can be added so we see something on serial
out
just before the crash...?
As far as I know, Tony and notaz did a memory test run for hours, so
for
some reason, it didn't freeze on their boards.
Who would be willing to help me providing me files and guiding me
through
the tests?
Here are some ideas that could be the reason for the freezes:
- SD-Card issues - we're running the Linux off SD Card. Not sure it
likes
it when for example power breaks down or the driver does something wrong
with
the SD Card (though that should probably lead to a kernel panic).
- Issues switching between various voltages / frequencies (are all the
voltages we're using in the dtb approved by TI?)
RAM timing issues
PALMAS power supply / timing issues. If PALMAS does something wrong,
the
system could easily freeze
The biggest issue is, that for the new PCBs both hardware and software
have
changed, so it's hard to compare it and rule out hardware / software issues.
Kernel 4.15 and old 2GB RAM hardware or uEVM works fine. Kernel 4.18 doesn't work stable on old OR new hardware and neither on
uEVM.
It could be hardware, software - or a mixture of both. Well, software seems to be ONE reason at least, as 4.18 doesn't work on uEVM as well. But it means I can't properly test the new hardware right now - which
is
critical :)
What are your ideas, how can we approach this?
Thanks a lot in advance!
-- Greetings,
Michael Mrozek
OpenPandora GmbH CEO: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Germany Tel.: +49 841 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org _______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Am Donnerstag, den 20.09.2018, 15:24 +0200 schrieb H. Nikolaus Schaller:
Hi,
neither u-boot nor kernel nor DTS modify any DDR3 (EMIF) timing. This all is defined by the SPL (MLO). But the exact timings might be different for each board.
Hm, so could it be that the timings in the MLO are a bit off for our RAM? That it didn't have anything to do with optimized or unoptimized CPU boards but mostly with timings being a bit on the edge?
Maybe even WRONG for the RAM we use?
I'm a bit confused by all the EMIF stuff. According to JEDEC Specs, these should be the same for all RAM types, but I remember that I found some differences when I compared both datasheets (for the old and new RAM).
So do we need to setup EMIF according to our RAM or not...?
Or: Can we change some EMIF values to raise stability (maybe at the cost of tiny bit performance?)
It was the older 4gb board (I have 2 now) and different dts, remaining
things should be the same I think (same SD). I intentionally tried to minimize changes from my working setup before leaving for the trip. Before reading this thread I just assumed my new board is faulty, but now it's clear more testing is needed. One obvious difference from the older board is the heatsink and some sort of new shielding (?), could it be messing something up like shorting or reflecting high freq signals?
On Thu, 20 Sep 2018, 21:56 Michael Mrozek, EvilDragon@openpandora.org wrote:
Am Donnerstag, den 20.09.2018, 21:49 +0900 schrieb Grazvydas Ignotas:
Hi,
that's bad news :( AFAIR, you were running memtester on the new 4GB RAM PCB I've sent you earlier as well for a few hours.
Were you using a different kernel / dtb here for the test? Because it's very weird that the PCB survives a memtest for both Tony and you but then has issues when simply idling around.
What's different here?
I'm traveling now and ED sent me a new board just before I left, and unfortunately it's unusable because of these random hangs. I wasn't able
to
correlate these hangs to anything too, it just hangs in like 1-10
minutes.
Most often it starts showing garbage suggesting RAM issues. I'm using
4.15
for working tiler, with adjusted dts for the new board. Nubs and sesors
are
not responding on that board too.
Pyra would have helped a lot during the travel, but unfortunately it's
just
dead weight I have to carry around.
notaz
On Thu, 20 Sep 2018, 20:55 Michael Mrozek, <EvilDragon@openpandora.org
wrote:
I've got 22 Mainboard + CPU board combination (latest 4GB RAM versions) lying around at home and would like to ensure that the hardware is working
fine
- so I
can place the order for the mass production.
However, I'm running into issues which I can't solve myself (as I am no developer and can't even do small changes to a kernel, etc.) I'm good with Linux, but not with developing - so I need some serious
help
here.
The unit I'm testing right now is one of latest batch.
Our own PyraOS usually freezes the system right after 30 seconds to 1 minute, so not really usable.
The latest "stable" Letux (kernel 4.18.8) with LXDE as base OS
(installed
using the makesd-tool) does boot fine. On two tries, it freezed after around
10
- 12
minutes sitting around idle. And about 2 - 3 minutes running the
high-load
script (which ups the CPU to 1,5GHz with some loop).
I will do more testings to see if it's always around the same time.
I've made a small script that logs power supply, uptime and
temperature to
see if it has something to do with that.
The result (incl. bootlog) is attached. I went to bed while it was running, but you can see in the log that it freezes after 12 minutes. Temperature is fine, voltage also seems to be fine
and
when using the high-load script, the temperature is higher after a few
seconds
and it still takes minutes to freeze - so we can assume that the freeze
doesn't
have anything to do with temperature CPU load.
Interestingly, when I came back, the frozen unit was still on (and
showed
the frozen desktop screen), but pretty warm - a lot warmer compared to it
just
idling.
I've added the new temperature and charging state at the end of
freeze.txt
as well.
Some things we know:
- Both askarus' and aTc's older Pyras (older Mainboard, 2GB RAM CPU
board
without our optimizations) are stable with kernel 4.15. aTc mentioned
he'd
been running it off USB for days whereas askarus' Pyra runs 10 hours off
battery
without freezing and in normal usage (internet, LibreOffice, etc.) Unfortunately, the released kernel 4.15 doesn't support the new
mainboards
/ CPU boards for some reason, so I can't compare them.
- According to hns, even the uEVM freezes randomly with the latest
kernel
4.18
- These freezes don't seem to be connected with temperature or CPU
load (both Askarus and aTcs Pyras don't have any heat sink applied, whereas the
4GB
RAM unit I've tested it on does).
- Sometimes the freeze shows random graphic glitches on the display,
sometimes the display simply freezes (memory errors?)
- The freeze might NOT be a full freeze! When using hns' QuantumStep
OS (which uses the LEDs as heartbeat), the unit also seemed to have frozen after
a
couple of minutes, as the LEDs stopped blinking. However, the LCD was still updating the graphics - VERY very slowly (1 character per 30 seconds on the terminal), but it was still working - even though it seemed frozen at first. This doesn't seem to be the case on LXDE, but maybe it's even slower because there's so much going on.
- On the unit I tested, sometimes u-Boot reported that it expected
4000000
of memory size but only found 4. This might be related to the unit - or
maybe
some timings are still wrong for the 4GB RAM so there's some issue here?
- Running the uevm with little less than 5V increases the
instability.
As
aTc and askarus Pyra work fine with less than 5V, this shouldn't be a
problem
- but
I thought I'd mention it here anyways.
So I'd like to find out the reason for these random freezes are. I've got various PCBs and cases, I can run tests - but as I can't
code, I
can't play around with kernel parameters or similar, so I'm pretty limited finding out what it could be.
That's where I need your help! Does anyone have any ideas what the issue could be and can provide different kernels, board files, etc. for me so I can test if that changes the behaviour?
It would also be good if someone could provide a minimal image with all non- needed drivers disabled per default incl. instructions how I can load
them
afterwards one by one (for example, let me know the modules to load and what they're used for).
So basically, starting with minimal features to see when it fails. Maybe further debug notices can be added so we see something on serial
out
just before the crash...?
As far as I know, Tony and notaz did a memory test run for hours, so
for
some reason, it didn't freeze on their boards.
Who would be willing to help me providing me files and guiding me
through
the tests?
Here are some ideas that could be the reason for the freezes:
- SD-Card issues - we're running the Linux off SD Card. Not sure it
likes
it when for example power breaks down or the driver does something wrong
with
the SD Card (though that should probably lead to a kernel panic).
- Issues switching between various voltages / frequencies (are all
the voltages we're using in the dtb approved by TI?)
RAM timing issues
PALMAS power supply / timing issues. If PALMAS does something
wrong,
the
system could easily freeze
The biggest issue is, that for the new PCBs both hardware and software
have
changed, so it's hard to compare it and rule out hardware / software issues.
Kernel 4.15 and old 2GB RAM hardware or uEVM works fine. Kernel 4.18 doesn't work stable on old OR new hardware and neither on
uEVM.
It could be hardware, software - or a mixture of both. Well, software seems to be ONE reason at least, as 4.18 doesn't work on uEVM as well. But it means I can't properly test the new hardware right now - which
is
critical :)
What are your ideas, how can we approach this?
Thanks a lot in advance!
-- Greetings,
Michael Mrozek
OpenPandora GmbH CEO: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Germany Tel.: +49 841 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org _______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Am 20.09.2018 um 15:37 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Donnerstag, den 20.09.2018, 15:24 +0200 schrieb H. Nikolaus Schaller:
Hi,
neither u-boot nor kernel nor DTS modify any DDR3 (EMIF) timing. This all is defined by the SPL (MLO). But the exact timings might be different for each board.
Hm, so could it be that the timings in the MLO are a bit off for our RAM? That it didn't have anything to do with optimized or unoptimized CPU boards but mostly with timings being a bit on the edge?
Maybe even WRONG for the RAM we use?
I'm a bit confused by all the EMIF stuff. According to JEDEC Specs, these should be the same for all RAM types, but I remember that I found some differences when I compared both datasheets (for the old and new RAM).
So do we need to setup EMIF according to our RAM or not...?
It looks so. We did have two different RAM brands and they definitively did need different settings.
Or: Can we change some EMIF values to raise stability (maybe at the cost of tiny bit performance?)
As far as I remember, Tony did send some EMIF setup patch that is in notaz and letux uboot now, but noted that it might be reasonable to change some more parameters. Unfortunately I do not really understand more about them than you... So I have never played with them.
It was the older 4gb board (I have 2 now) and different dts, remaining
things should be the same I think (same SD). I intentionally tried to minimize changes from my working setup before leaving for the trip. Before reading this thread I just assumed my new board is faulty, but now it's clear more testing is needed. One obvious difference from the older board is the heatsink and some sort of new shielding (?), could it be messing something up like shorting or reflecting high freq signals?
On Thu, 20 Sep 2018, 21:56 Michael Mrozek, EvilDragon@openpandora.org wrote:
Am Donnerstag, den 20.09.2018, 21:49 +0900 schrieb Grazvydas Ignotas:
Hi,
that's bad news :( AFAIR, you were running memtester on the new 4GB RAM PCB I've sent you earlier as well for a few hours.
Were you using a different kernel / dtb here for the test? Because it's very weird that the PCB survives a memtest for both Tony and you but then has issues when simply idling around.
What's different here?
I'm traveling now and ED sent me a new board just before I left, and unfortunately it's unusable because of these random hangs. I wasn't able
to
correlate these hangs to anything too, it just hangs in like 1-10
minutes.
Most often it starts showing garbage suggesting RAM issues. I'm using
4.15
for working tiler, with adjusted dts for the new board. Nubs and sesors
are
not responding on that board too.
Pyra would have helped a lot during the travel, but unfortunately it's
just
dead weight I have to carry around.
notaz
On Thu, 20 Sep 2018, 20:55 Michael Mrozek, <EvilDragon@openpandora.org > wrote:
> I've got 22 Mainboard + CPU board combination (latest 4GB RAM > versions) > lying > around at home and would like to ensure that the hardware is working
fine
> - so I > can place the order for the mass production. > > However, I'm running into issues which I can't solve myself (as I am > no > developer and can't even do small changes to a kernel, etc.) > I'm good with Linux, but not with developing - so I need some > serious
help
> here. > > The unit I'm testing right now is one of latest batch. > > Our own PyraOS usually freezes the system right after 30 seconds to > 1 > minute, so > not really usable. > > The latest "stable" Letux (kernel 4.18.8) with LXDE as base OS
(installed
> using > the makesd-tool) does boot fine. On two tries, it freezed after > around
10
> - 12 > minutes sitting around idle. And about 2 - 3 minutes running the
high-load
> script (which ups the CPU to 1,5GHz with some loop). > > I will do more testings to see if it's always around the same time. > > I've made a small script that logs power supply, uptime and
temperature to
> see > if it has something to do with that. > > The result (incl. bootlog) is attached. > I went to bed while it was running, but you can see in the log that > it > freezes > after 12 minutes. Temperature is fine, voltage also seems to be fine
and
> when > using the high-load script, the temperature is higher after a few
seconds
> and it > still takes minutes to freeze - so we can assume that the freeze
doesn't
> have > anything to do with temperature CPU load. > > Interestingly, when I came back, the frozen unit was still on (and
showed
> the > frozen desktop screen), but pretty warm - a lot warmer compared to > it
just
> idling. > > I've added the new temperature and charging state at the end of
freeze.txt
> as > well. > > Some things we know: > > * Both askarus' and aTc's older Pyras (older Mainboard, 2GB RAM CPU
board
> without our optimizations) are stable with kernel 4.15. aTc > mentioned
he'd
> been > running it off USB for days whereas askarus' Pyra runs 10 hours off
battery
> without freezing and in normal usage (internet, LibreOffice, etc.) > Unfortunately, the released kernel 4.15 doesn't support the new
mainboards
> / CPU > boards for some reason, so I can't compare them. > > * According to hns, even the uEVM freezes randomly with the latest
kernel
> 4.18 > > * These freezes don't seem to be connected with temperature or CPU > load > (both > Askarus and aTcs Pyras don't have any heat sink applied, whereas the
4GB
> RAM > unit I've tested it on does). > > * Sometimes the freeze shows random graphic glitches on the display, > sometimes > the display simply freezes (memory errors?) > > * The freeze might NOT be a full freeze! When using hns' QuantumStep > OS > (which > uses the LEDs as heartbeat), the unit also seemed to have frozen > after
a
> couple > of minutes, as the LEDs stopped blinking. However, the LCD was still > updating > the graphics - VERY very slowly (1 character per 30 seconds on the > terminal), > but it was still working - even though it seemed frozen at first. > This doesn't seem to be the case on LXDE, but maybe it's even slower > because > there's so much going on. > > * On the unit I tested, sometimes u-Boot reported that it expected
4000000
> of > memory size but only found 4. This might be related to the unit - or
maybe
> some > timings are still wrong for the 4GB RAM so there's some issue here? > > * Running the uevm with little less than 5V increases the > instability.
As
> aTc > and askarus Pyra work fine with less than 5V, this shouldn't be a
problem
> - but > I thought I'd mention it here anyways. > > So I'd like to find out the reason for these random freezes are. > I've got various PCBs and cases, I can run tests - but as I can't
code, I
> can't > play around with kernel parameters or similar, so I'm pretty limited > finding out > what it could be. > > That's where I need your help! > Does anyone have any ideas what the issue could be and can provide > different > kernels, board files, etc. for me so I can test if that changes the > behaviour? > > It would also be good if someone could provide a minimal image with > all > non- > needed drivers disabled per default incl. instructions how I can > load
them
> afterwards one by one (for example, let me know the modules to load > and > what > they're used for). > > So basically, starting with minimal features to see when it fails. > Maybe further debug notices can be added so we see something on > serial
out
> just > before the crash...? > > As far as I know, Tony and notaz did a memory test run for hours, so
for
> some > reason, it didn't freeze on their boards. > > Who would be willing to help me providing me files and guiding me
through
> the > tests? > > Here are some ideas that could be the reason for the freezes: > > * SD-Card issues - we're running the Linux off SD Card. Not sure it
likes
> it > when for example power breaks down or the driver does something > wrong
with
> the > SD Card (though that should probably lead to a kernel panic). > > * Issues switching between various voltages / frequencies (are all > the > voltages > we're using in the dtb approved by TI?) > > * RAM timing issues > > * PALMAS power supply / timing issues. If PALMAS does something > wrong,
the
> system could easily freeze > > > The biggest issue is, that for the new PCBs both hardware and > software
have
> changed, so it's hard to compare it and rule out hardware / software > issues. > > Kernel 4.15 and old 2GB RAM hardware or uEVM works fine. > Kernel 4.18 doesn't work stable on old OR new hardware and neither > on
uEVM.
> > It could be hardware, software - or a mixture of both. > Well, software seems to be ONE reason at least, as 4.18 doesn't work > on > uEVM as > well. > But it means I can't properly test the new hardware right now - > which
is
> critical :) > > What are your ideas, how can we approach this? > > Thanks a lot in advance! > > -- > Greetings, > > Michael Mrozek > > ----------------------- > OpenPandora GmbH > CEO: Michael Mrozek > > Schäffbräustr. 11 > 85049 Ingolstadt > Germany > Tel.: +49 841 990 5548 > http://www.openpandora.de/ > HRB 4879, Amtsgericht Ingolstadt > ----------------------- > eMail: mrozek@openpandora.org > _______________________________________________ > Kernel mailing list > Kernel@pyra-handheld.com > http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel >
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
Am Donnerstag, den 20.09.2018, 15:45 +0200 schrieb H. Nikolaus Schaller:
Hi,
Or: Can we change some EMIF values to raise stability (maybe at the cost of tiny bit performance?
As far as I remember, Tony did send some EMIF setup patch that is in notaz and letux uboot now, but noted that it might be reasonable to change some more parameters. Unfortunately I do not really understand more about them than you... So I have never played with them.
I thought EMIF is in the MLO, not in U-boot...? Or is the MLO combined with U-Boot on the Pyra?
I'm confused :)
Well, I'm perfectly fine doing trial and error with the values.
The datasheet usually has minimum and maximum listed, so I should be able to safely change them.
I'd change one value at a time between max and min to see if it improves things (or worsens them).
Also, simply comparing the values to the datasheet might help as well.
Am 20.09.2018 um 15:57 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Donnerstag, den 20.09.2018, 15:45 +0200 schrieb H. Nikolaus Schaller:
Hi,
Or: Can we change some EMIF values to raise stability (maybe at the cost of tiny bit performance?
As far as I remember, Tony did send some EMIF setup patch that is in notaz and letux uboot now, but noted that it might be reasonable to change some more parameters. Unfortunately I do not really understand more about them than you... So I have never played with them.
I thought EMIF is in the MLO, not in U-boot...? Or is the MLO combined with U-Boot on the Pyra?
Yes. There is just a single u-boot source and two compile phases make
MLO (called "SPL") u-boot.img
out of it. Up to ca. 5 years ago there was a separate x-loader source tree for MLO.
I'm confused :)
Well, I'm perfectly fine doing trial and error with the values.
They are in a big array of magic hex numbers...
http://git.goldelico.com/?p=letux-uboot.git;a=blob;f=board/goldelico/letux-c...
The datasheet usually has minimum and maximum listed, so I should be able to safely change them.
I'd change one value at a time between max and min to see if it improves things (or worsens them).
Also, simply comparing the values to the datasheet might help as well.
-- Greetings,
Michael Mrozek
OpenPandora GmbH CEO: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Germany Tel.: +49 841 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
Am Donnerstag, den 20.09.2018, 16:08 +0200 schrieb H. Nikolaus Schaller:
Okay, thanks!
Looks like those hex values can be created with the EMIF-Tool by TI. I think I played around with that already. It was an excel sheet where you entered the values according to the datasheet and the tool spits out the hex values.
I think it can also do it the other way round, so we know what has been set up with the Pyra.
I'll search for that.
Or maybe Tony can share some light, as he knows what he's doing :D
BTW: https://patchwork.ozlabs.org/patch/479808/
The X15 has very different hex values compared to us, so yes, there might be some lions sleeping there!
Also: They enabled hardware leveling in 2015 - do we have hardware leveling enabled or is ours still based on older values?
I'll try to find out :)
Am 20.09.2018 um 15:57 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Donnerstag, den 20.09.2018, 15:45 +0200 schrieb H. Nikolaus Schaller:
Hi,
Or: Can we change some EMIF values to raise stability (maybe at the cost of tiny bit performance?
As far as I remember, Tony did send some EMIF setup patch that is in notaz and letux uboot now, but noted that it might be reasonable to change some more parameters. Unfortunately I do not really understand more about them than you... So I have never played with them.
I thought EMIF is in the MLO, not in U-boot...? Or is the MLO combined with U-Boot on the Pyra?
Yes. There is just a single u-boot source and two compile phases make
MLO (called "SPL") u-boot.img
out of it. Up to ca. 5 years ago there was a separate x-loader source tree for MLO.
I'm confused :)
Well, I'm perfectly fine doing trial and error with the values.
They are in a big array of magic hex numbers...
http://git.goldelico.com/?p=letux-uboot.git;a=blob;f=board/goldelico/letux-c...
The datasheet usually has minimum and maximum listed, so I should be able to safely change them.
I'd change one value at a time between max and min to see if it improves things (or worsens them).
Also, simply comparing the values to the datasheet might help as well.
-- Greetings,
Michael Mrozek
OpenPandora GmbH CEO: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Germany Tel.: +49 841 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
* H. Nikolaus Schaller hns@goldelico.com [180920 14:13]:
Am 20.09.2018 um 15:57 schrieb Michael Mrozek EvilDragon@openpandora.org:
They are in a big array of magic hex numbers...
http://git.goldelico.com/?p=letux-uboot.git;a=blob;f=board/goldelico/letux-c...
The datasheet usually has minimum and maximum listed, so I should be able to safely change them.
I'd change one value at a time between max and min to see if it improves things (or worsens them).
Also, simply comparing the values to the datasheet might help as well.
Yes please do check again :)
Can you please try maxing out .sdram_tim3 T_RFC value from 0x027f8be8 to 0x027f8ff8, see commit 2b143578a07afe0e50567f8327f4f62e0703cb20.
Regards,
Tony
On Do, 2018-09-20 at 09:29 -0700, Tony Lindgren wrote:
Also, simply comparing the values to the datasheet might help as well.
Yes please do check again :)
Just tried doing that, I have a few questions though:
1. Is there an EMIF-Tool for the OMAP5 as well? I only found the EMIF RegisterConfig Rev 2.0.1 (which is for the DRA7 and similar. The one I used is this one: http://www.ti.com/lit/an/sprac36b/sprac36b.zip
2. Table 1 (Step1-SystemDetails) requires system details like ODT, Output Driver Impedance, etc. Aren't these dependant on our board layout? How can we get these? It says "** NOTE: IBIS models should be used for signal integrity analysis. I/O settings programmed should match simulation settings."... so can we get these values from Juliano?
3. Does the OMAP5 support Hardware Levelling and have we enabled that? Otherwise, we need the trace lengths (according to Step2-BoardDetails).
4. On Step3-DDRTimings, most timings are already exactly like they say in our datasheet. Some values are "minimum" or "maximum". Are these values we can play around with to give the RAM a bit more safety?
5.As mentioned, those specs are exactly the same as in our datasheet. Howeve, I noticed that tfRAW has the value 50 pre-setup in the tool (which is for 2KB page size) whereas the value tRRD has the value 7.5 pre-setup in the tool (which is for 1KB page size). Is that a mismatch or doesn't that matter?
6. Some values the tool spits out are different than what we have in U- Boot. Some examples: .sdram_config and .sdram_config_init: 0x61851BB2 in the EMIF tool, 0x61851B3A in U-Boot.
.sdram_tim1 is 0xCCCF36B5 according to the EMIF Tool, but we use 0xCCCF36B3 (so a bit lower).
.emif_ddr_phy_ctlr_1_init is 0x0024400B in the EMIF Tool, but we are using 0x0030400A.
.emif_rd_wr_exec_thresh is 0x00000305 in the EMIF Tool, we're using 0x40000305
.zq_config results in 0x5007190B and we're using 0xD007190B,
So yeah, there ARE some differences - though I have no idea what these values mean. Do they have something to do with those board specific values (where the DRA7 EVM surely differs from our setup)?
I hope that helps :)
Can you please try maxing out .sdram_tim3 T_RFC value from 0x027f8be8 to 0x027f8ff8, see commit 2b143578a07afe0e50567f8327f4f62e0703cb20.
Regards,
Tony
* Michael Mrozek EvilDragon@openpandora.org [180920 11:13]:
On Do, 2018-09-20 at 09:29 -0700, Tony Lindgren wrote:
Also, simply comparing the values to the datasheet might help as well.
Yes please do check again :)
Just tried doing that, I have a few questions though:
- Is there an EMIF-Tool for the OMAP5 as well? I only found the EMIF
RegisterConfig Rev 2.0.1 (which is for the DRA7 and similar. The one I used is this one: http://www.ti.com/lit/an/sprac36b/sprac36b.zip
Not that I know. Note that dra7 is different for at least levelling, so some registers might be different too. There's a tool for am437x that might be similar to omap5, but I don't think it supports 532MHz rate.
- Table 1 (Step1-SystemDetails) requires system details like ODT,
Output Driver Impedance, etc. Aren't these dependant on our board layout? How can we get these? It says "** NOTE: IBIS models should be used for signal integrity analysis. I/O settings programmed should match simulation settings."... so can we get these values from Juliano?
- Does the OMAP5 support Hardware Levelling and have we enabled that?
Otherwise, we need the trace lengths (according to Step2-BoardDetails).
Yes that's done AFAIK on boot. That too might have issues remaining.
- On Step3-DDRTimings, most timings are already exactly like they say
in our datasheet. Some values are "minimum" or "maximum". Are these values we can play around with to give the RAM a bit more safety?
I think they are the accepted JEDEC range.
5.As mentioned, those specs are exactly the same as in our datasheet. Howeve, I noticed that tfRAW has the value 50 pre-setup in the tool (which is for 2KB page size) whereas the value tRRD has the value 7.5 pre-setup in the tool (which is for 1KB page size). Is that a mismatch or doesn't that matter?
Hmm that might be worth testing.
- Some values the tool spits out are different than what we have in U-
Boot. Some examples: .sdram_config and .sdram_config_init: 0x61851BB2 in the EMIF tool, 0x61851B3A in U-Boot.
For sdram_config_init, it needs to have 2 chip selects per ddr so pyra-4g has 4 chip selects total. So ROWSIZE we seem to have wrong for 4Gb, it should be 16 not 15, but I think that's cosmetic.
.sdram_tim1 is 0xCCCF36B5 according to the EMIF Tool, but we use 0xCCCF36B3 (so a bit lower).
This value seems to make no difference, it works the same way and could be used.
.emif_ddr_phy_ctlr_1_init is 0x0024400B in the EMIF Tool, but we are using 0x0030400A.
And emif_ddr_phy_ctlr_1_init needs to be configured in addition to emif_ddr_phy_ctlr_1 with PHY_INVERT_CLKOUT cleared for emif_ddr_phy_ctlr_1_init. So this would translate to values:
.emif_ddr_phy_ctlr_1_init = 0x0020400B, .emif_ddr_phy_ctlr_1 = 0x0024400B,
They too work just the same and could be used.
.emif_rd_wr_exec_thresh is 0x00000305 in the EMIF Tool, we're using 0x40000305
This emif_rd_wr_exec_thresh is missing LL_BUBBLE_ENABLE, so no change to what we have.
.zq_config results in 0x5007190B and we're using 0xD007190B,
And zq_config is missing ZQ_CS1EN, so we have the same value after that already.
So yeah, there ARE some differences - though I have no idea what these values mean. Do they have something to do with those board specific values (where the DRA7 EVM surely differs from our setup)?
I think we need to check any idle dll affecting values again considering disabling the internal dll hack prevents the idle hang based on brief testing..
Regards,
Tony
I just tried running mtest on my test Pyra, as you can't get any more lowlevel and we can rule out kernel settings.
And yes, there surely is an issue with the memory, as you can see in the attached logfile (I stopped after 15 iterations).
Even the "FAILURE" has changed to "FABLURE" in the RAM.
I'll run the memtest with the other PCBs I have here as well!
On Do, 2018-09-20 at 20:11 +0200, Michael Mrozek wrote:
Quick update: Running mtest now on a different test unit. 18 iterations without a failure so far.
I'll let it continue, but it could really be a tolerance / timing issue.
I just tried running mtest on my test Pyra, as you can't get any more lowlevel and we can rule out kernel settings.
And yes, there surely is an issue with the memory, as you can see in the attached logfile (I stopped after 15 iterations).
Even the "FAILURE" has changed to "FABLURE" in the RAM.
I'll run the memtest with the other PCBs I have here as well!
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
On Do, 2018-09-20 at 20:19 +0200, Michael Mrozek wrote:
Oh, and: That PCB does NOT have the heat transfer foil from the OMAP / DRAMs to the heatsink installed... so maybe that copper DOES have an effect on the RAM...?
On Do, 2018-09-20 at 20:11 +0200, Michael Mrozek wrote:
Quick update: Running mtest now on a different test unit. 18 iterations without a failure so far.
I'll let it continue, but it could really be a tolerance / timing issue.
I just tried running mtest on my test Pyra, as you can't get any more lowlevel and we can rule out kernel settings.
And yes, there surely is an issue with the memory, as you can see in the attached logfile (I stopped after 15 iterations).
Even the "FAILURE" has changed to "FABLURE" in the RAM.
I'll run the memtest with the other PCBs I have here as well!
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Am Donnerstag, den 20.09.2018, 09:29 -0700 schrieb Tony Lindgren:
@Nikolaus: Please let me know once that is integrated into the Letux U-Boot so I can create a new SD Card and continue testing.
Thanks.
- H. Nikolaus Schaller hns@goldelico.com [180920 14:13]:
Am 20.09.2018 um 15:57 schrieb Michael Mrozek EvilDragon@openpandora.org :
They are in a big array of magic hex numbers...
http://git.goldelico.com/?p=letux-uboot.git;a=blob;f=board/goldelico/letux-c...
The datasheet usually has minimum and maximum listed, so I should be able to safely change them.
I'd change one value at a time between max and min to see if it improves things (or worsens them).
Also, simply comparing the values to the datasheet might help as well.
Yes please do check again :)
Can you please try maxing out .sdram_tim3 T_RFC value from 0x027f8be8 to 0x027f8ff8, see commit 2b143578a07afe0e50567f8327f4f62e0703cb20.
Regards,
Tony
Just a quick question: Is there a tool available that can dump all the setup configs and timings for the DRAM in Linux? That would be a good way to compare our running setup with the datasheet and check for some things like is hardware levelling enabled or not.
I can imagine that a disabled hardware levelling could cause exact these issues, as no one has measured or calculated our traces. Or if hardware levelling doesn't work properly for some reason - so dumping these values to see if the hardware levelling results differ with each boot would be a good help
On 24 September 2018 12:39:48 CEST, Michael Mrozek EvilDragon@openpandora.org wrote:
Am Donnerstag, den 20.09.2018, 09:29 -0700 schrieb Tony Lindgren:
@Nikolaus: Please let me know once that is integrated into the Letux U-Boot so I can create a new SD Card and continue testing.
Thanks.
- H. Nikolaus Schaller hns@goldelico.com [180920 14:13]:
Am 20.09.2018 um 15:57 schrieb Michael Mrozek
:
They are in a big array of magic hex numbers...
http://git.goldelico.com/?p=letux-uboot.git;a=blob;f=board/goldelico/letux-c...
The datasheet usually has minimum and maximum listed, so I should
be able
to safely change them.
I'd change one value at a time between max and min to see if it
improves
things (or worsens them).
Also, simply comparing the values to the datasheet might help as
well.
Yes please do check again :)
Can you please try maxing out .sdram_tim3 T_RFC value from 0x027f8be8
to
0x027f8ff8, see commit 2b143578a07afe0e50567f8327f4f62e0703cb20.
Regards,
Tony
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
* Michael Mrozek EvilDragon@openpandora.org [180926 10:18]:
Just a quick question: Is there a tool available that can dump all the setup configs and timings for the DRAM in Linux? That would be a good way to compare our running setup with the datasheet and check for some things like is hardware levelling enabled or not.
I've been using Tomi's rwmem tool from github for dumping out register ranges:
https://github.com/tomba/rwmem
I can imagine that a disabled hardware levelling could cause exact these issues, as no one has measured or calculated our traces. Or if hardware levelling doesn't work properly for some reason - so dumping these values to see if the hardware levelling results differ with each boot would be a good help
Yes that might provide some more clues.
Regards,
Tony
I've had a look at the uboot for the igepv5 , and they set the ram timings in a slightly nicer way than just magical hex values. It's probably completely different ram, but maybe it's useful to compare how they set it up.
https://git.isee.biz/arm-boot/u-boot-arm/blob/u-boot-2014.01.y-omap5/board/i...
https://git.isee.biz/arm-boot/u-boot-arm/blob/u-boot-2014.01.y-omap5/board/i...
Hello everyone,
in the forums EvilDragon said: "4GB RAM boards were working with only 2GB RAM enabled (as far as I remember - as that was the reason I ordered the RAM modules back then!)"
Did anyone with a 4GB Pyra try running it with only 2GB enabled? Might it work if we only populate it with half the amount of RAM chips?
These are important questions. If one of these options does actually work it will be fine for the first batch.
Of course it's better to fix things but we are at a state where we should start production as soon as possible. Either of these possibilities would bring some Pyras to life and be of no additional cost.
Maybe that's silly but if there is a chance that EvilDragon's memories are true, then for me it's obvious what to do.
Best Wishes Askarus
Hi Askarus,
Am 01.10.2018 um 19:45 schrieb Nino Bock askarus@openpandora.org:
Hello everyone,
in the forums EvilDragon said: "4GB RAM boards were working with only 2GB RAM enabled (as far as I remember - as that was the reason I ordered the RAM modules back then!)"
Did anyone with a 4GB Pyra try running it with only 2GB enabled?
Yes. We did that several times. Sometimes even unplanned (copying a non-LPAE kernel to the SD card).
Might it work if we only populate it with half the amount of RAM chips?
Unlikely. There are two separate busses from OMAP5 to 2 RAM chips each. So if one bus makes trouble it still will do if the other one is not used.
The only differences may be power demand and cross-talk.
These are important questions. If one of these options does actually work it will be fine for the first batch.
Of course it's better to fix things but we are at a state where we should start production as soon as possible. Either of these possibilities would bring some Pyras to life and be of no additional cost.
Maybe that's silly but if there is a chance that EvilDragon's memories are true, then for me it's obvious what to do.
BR, Nikolaus
Hello,
I'm back and I've done some tests, no useful conclusions though. I have 2 4GB boards now, I'll call them old one and new one.
- the old board also hangs, although it seems to take much longer (3 hours at least) - the new one hangs in minutes, but occasionally it can run for half an hour or so - u-boot tester doesn't fail on new one, although I haven't tried it for too long (less than 10 minutes) - memtester (on Linux) doesn't fail on new one, it also seems to allow the new board to survive much longer. I think it survived overnight but hung later when I was at work.
ideas: - perhaps the problem has something to do with idle/power saving. Need to check if there are RAM registers related to that. Does anyone know if it currently tries to do RAM self-refresh while idle? - try without any TILER patches. I can vaguely remember it was said the TILER can't be used at all because of some errata, but Matt proved otherwise. Maybe with 4GB/LPAE the problem actually triggers.
Gražvydas
On Mon, Oct 1, 2018 at 9:15 PM H. Nikolaus Schaller hns@goldelico.com wrote:
Hi Askarus,
Am 01.10.2018 um 19:45 schrieb Nino Bock askarus@openpandora.org:
Hello everyone,
in the forums EvilDragon said: "4GB RAM boards were working with only 2GB RAM enabled (as far as I remember - as that was the reason I ordered the RAM modules back then!)"
Did anyone with a 4GB Pyra try running it with only 2GB enabled?
Yes. We did that several times. Sometimes even unplanned (copying a non-LPAE kernel to the SD card).
Might it work if we only populate it with half the amount of RAM chips?
Unlikely. There are two separate busses from OMAP5 to 2 RAM chips each. So if one bus makes trouble it still will do if the other one is not used.
The only differences may be power demand and cross-talk.
These are important questions. If one of these options does actually work it will be fine for the first batch.
Of course it's better to fix things but we are at a state where we should start production as soon as possible. Either of these possibilities would bring some Pyras to life and be of no additional cost.
Maybe that's silly but if there is a chance that EvilDragon's memories are true, then for me it's obvious what to do.
BR, Nikolaus
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Am Tue, 2 Oct 2018 14:39:28 +0300 hat Grazvydas Ignotas notasas@gmail.com geschrieben:
Hi,
I'm back and I've done some tests, no useful conclusions though. I have 2 4GB boards now, I'll call them old one and new one.
- the old board also hangs, although it seems to take much longer (3
hours at least)
- the new one hangs in minutes, but occasionally it can run for half
an hour or so
- u-boot tester doesn't fail on new one, although I haven't tried it
for too long (less than 10 minutes)
I've run them for half an hour without any issues, so yes, it doesn't seem to fail.
- memtester (on Linux) doesn't fail on new one, it also seems to allow
the new board to survive much longer. I think it survived overnight but hung later when I was at work.
THAT is interesting...
ideas:
- perhaps the problem has something to do with idle/power saving. Need
to check if there are RAM registers related to that. Does anyone know if it currently tries to do RAM self-refresh while idle?
Hm, but would idle / power saving trigger when the system is doing something? Because when doing my test runs, I always run a script that spits out the voltages, temperatures and remaining temperature left every 10 seconds. So Linux doesn't really idle around, but freezes anyways after some minutes. And sometimes it even freezes while booting - and it's anything but idle there.
Or do you mean something else?
- try without any TILER patches. I can vaguely remember it was said
the TILER can't be used at all because of some errata, but Matt proved otherwise. Maybe with 4GB/LPAE the problem actually triggers.
I remember that (that's why it was so slow at the beginning), haven't found much about TILER in the errata though. Only one which doesn't seem to be too critical.
But it's a good idea trying that, yes. If someone compiles me a kernel, I'll test anything on various boards whatever ideas you have :)
I can't code much, but I can test.
Here's what I planned to do:
* See if a non-LPAE-Kernel has a different effect * Boot the kernel directly into a shell (without loading any other drivers except for the ones directly included in the kernel). Test whether this runs more stable and then modprobe the remaining drivers one after the other. After all, Tony was running the memtest similar as well, and that didn't fail.
Any other ideas or comments are very welcome :)
Gražvydas
On Mon, Oct 1, 2018 at 9:15 PM H. Nikolaus Schaller hns@goldelico.com wrote:
Hi Askarus,
Am 01.10.2018 um 19:45 schrieb Nino Bock askarus@openpandora.org:
Hello everyone,
in the forums EvilDragon said: "4GB RAM boards were working with only 2GB RAM enabled (as far as I remember - as that was the reason I ordered the RAM modules back then!)"
Did anyone with a 4GB Pyra try running it with only 2GB enabled?
Yes. We did that several times. Sometimes even unplanned (copying a non-LPAE kernel to the SD card).
Might it work if we only populate it with half the amount of RAM chips?
Unlikely. There are two separate busses from OMAP5 to 2 RAM chips each. So if one bus makes trouble it still will do if the other one is not used.
The only differences may be power demand and cross-talk.
These are important questions. If one of these options does actually work it will be fine for the first batch.
Of course it's better to fix things but we are at a state where we should start production as soon as possible. Either of these possibilities would bring some Pyras to life and be of no additional cost.
Maybe that's silly but if there is a chance that EvilDragon's memories are true, then for me it's obvious what to do.
BR, Nikolaus
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
This may be unrelated, but my uevm has been freezing too, and that seems to have started after plugging in the hdmi cable again. I've had it running stable for weeks before, without a display connected. There has been some weirdness with the linux framebuffer console on these newer kernels anyway. On pyra hardware you get a black screen, on uevm the image jitters all over the place.
It might be worth checking if the freezes happen without a display driver or with no screen connected.
Am 02.10.2018 um 19:51 schrieb aTc atc@k-n-p.org:
This may be unrelated, but my uevm has been freezing too, and that seems to have started after plugging in the hdmi cable again. I've had it running stable for weeks before, without a display connected. There has been some weirdness with the linux framebuffer console on these newer kernels anyway. On pyra hardware you get a black screen, on uevm the image jitters all over the place.
It might be worth checking if the freezes happen without a display driver or with no screen connected.
Just for the records, we have two reports that the igepv5 has (had) also some issue with warm reset and abe_dpll ("reboot").
a) one reported recently by forum PM to me:
Blacklisting snd_soc_omap_abe_twl6040 also solves the reboot issue on the igep.
I used the 4.19-rc5 mainline with only the omap2plus_defconfig, no option changed, GCC-8.2.0. I didn't touch my ARM boards for a long time (I'm on fpga now), now LPAE is broken, but maybe it's linked to the old u-boot.
b) an old mail from March 2016 by Tony Lindgren:
- Warm reset seems to fail on igepv5, the clk code seems
to be missing clk_prepare_enable() for the abe_dpll. I need to debug this more as it seems that with LPAE reboot is still flakey on igepv5. Looks like omap5-uevm does not have this issue, but it has a newer u-boot as well.
* H. Nikolaus Schaller hns@goldelico.com [181004 06:22]:
b) an old mail from March 2016 by Tony Lindgren:
- Warm reset seems to fail on igepv5, the clk code seems
to be missing clk_prepare_enable() for the abe_dpll. I need to debug this more as it seems that with LPAE reboot is still flakey on igepv5. Looks like omap5-uevm does not have this issue, but it has a newer u-boot as well.
Sorry I don't remember how I got to this conclusion, it probably was just initial gut feeling on the issue. At least I never got anything working for the reboot beyond blacklisting snd_soc_omap_abe_twl6040.
Regards,
Tony
Hi Tony,
Am 04.10.2018 um 17:12 schrieb Tony Lindgren tony@atomide.com:
- H. Nikolaus Schaller hns@goldelico.com [181004 06:22]:
b) an old mail from March 2016 by Tony Lindgren:
- Warm reset seems to fail on igepv5, the clk code seems
to be missing clk_prepare_enable() for the abe_dpll. I need to debug this more as it seems that with LPAE reboot is still flakey on igepv5. Looks like omap5-uevm does not have this issue, but it has a newer u-boot as well.
Sorry I don't remember how I got to this conclusion, it probably was just initial gut feeling on the issue. At least I never got anything working for the reboot beyond blacklisting snd_soc_omap_abe_twl6040.
Ok, then at least we know that we are not duplicating work and just forgot to merge some known solution.
I have set up and started to learn kgdb. I can already debug the "sync" syscall as suggested by documentation...
The idea is to step beyond the known call path when modprobing pwm_omap_dmtimer. At some point there must be a write to critical control registers that influence the abe_dpll.
BR and thanks, Nikolaus
Hi,
Am 02.10.2018 um 17:29 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Tue, 2 Oct 2018 14:39:28 +0300 hat Grazvydas Ignotas notasas@gmail.com geschrieben:
Hi,
I'm back and I've done some tests, no useful conclusions though. I have 2 4GB boards now, I'll call them old one and new one.
- the old board also hangs, although it seems to take much longer (3
hours at least)
- the new one hangs in minutes, but occasionally it can run for half
an hour or so
- u-boot tester doesn't fail on new one, although I haven't tried it
for too long (less than 10 minutes)
I've run them for half an hour without any issues, so yes, it doesn't seem to fail.
- memtester (on Linux) doesn't fail on new one, it also seems to allow
the new board to survive much longer. I think it survived overnight but hung later when I was at work.
THAT is interesting...
ideas:
- perhaps the problem has something to do with idle/power saving. Need
to check if there are RAM registers related to that. Does anyone know if it currently tries to do RAM self-refresh while idle?
Hm, but would idle / power saving trigger when the system is doing something?
Well, I think it will trigger if the system is NOT doing something :)
Because when doing my test runs, I always run a script that spits out the voltages, temperatures and remaining temperature left every 10 seconds. So Linux doesn't really idle around, but freezes anyways after some minutes. And sometimes it even freezes while booting - and it's anything but idle there.
Or do you mean something else?
I remember that there is a patch which does control some CPU idling and is reported to have an effect in the range of minutes:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h...
But:
a) it is also in omap5evm mainline kernels b) I have tried reverting this patch some time ago but don't remember to have an effect wrt. freezes (I didn't measure power demand)
- try without any TILER patches. I can vaguely remember it was said
the TILER can't be used at all because of some errata, but Matt proved otherwise. Maybe with 4GB/LPAE the problem actually triggers.
I remember that (that's why it was so slow at the beginning), haven't found much about TILER in the errata though. Only one which doesn't seem to be too critical.
I tried that a while ago (perhaps several times), but it would be good to repeat and cross-check. My findings may have been disturbed by somthing else.
But it's a good idea trying that, yes. If someone compiles me a kernel, I'll test anything on various boards whatever ideas you have :)
I can't code much, but I can test.
Here's what I planned to do:
- See if a non-LPAE-Kernel has a different effect
- Boot the kernel directly into a shell (without loading any other
drivers except for the ones directly included in the kernel). Test whether this runs more stable and then modprobe the remaining drivers one after the other. After all, Tony was running the memtest similar as well, and that didn't fail.
Any other ideas or comments are very welcome :)
BTW: what I found is that Freezes seem to have the characteristics that the Reset button has no function. I.e. the OMAP5 isn't responding to external NRESWARM any more. This seems to be different from the omap5evm freezes I have. There I can always reboot with RESET if I remember correctly.
And I had a bad setup on one SD card some days ago for the LC15-eval board. This repeatably triggered a Freeze ca. 3 seconds into boot time (last messages were that LDO.8 and LDO.5 were initialized). The RESET button was responsive up to ca. 0.1 seconds before that, but not after.
Currently, I am trying to understand the details of the setup that makes this happen. Most likely it was a wrong U-Boot version for the board or a wrong DTB.
BR, Nikolaus
Some more tests: - disabling LPAE has no effect, same hang after few minutes - attempting to use 2GB MLO/u-boot on new 4GB board makes it very unstable (crashes or hangs early while booting the kernel)
On Tue, Oct 2, 2018 at 6:29 PM Michael Mrozek EvilDragon@openpandora.org wrote:
ideas:
- perhaps the problem has something to do with idle/power saving. Need
to check if there are RAM registers related to that. Does anyone know if it currently tries to do RAM self-refresh while idle?
Hm, but would idle / power saving trigger when the system is doing something? Because when doing my test runs, I always run a script that spits out the voltages, temperatures and remaining temperature left every 10 seconds.
And what does it do in between those 10s? If the system has no work for any periods, even as short as 1ms, it's called idle and power saving mechanisms start.
So Linux doesn't really idle around, but freezes anyways after some minutes. And sometimes it even freezes while booting - and it's anything but idle there.
It can be partially idle while booting, for example CPU can have nothing to do while data is arriving from the SD card (data is copied to RAM by SD card controller and CPU is not used). And because it is some important data, like loading systemd itself, kernel would have no other process to run and would put the CPU on idle for a short while.
- See if a non-LPAE-Kernel has a different effect
no
- Boot the kernel directly into a shell (without loading any other drivers except for the ones directly included in the kernel). Test whether this runs more stable and then modprobe the remaining drivers one after the other. After all, Tony was running the memtest similar as well, and that didn't fail.
Yeah that's something to try. Perhaps just move /lib/modules somewhere so they don't autoload (not sure if they go to the ramdisk currently).
On Tue, Oct 2, 2018 at 8:54 PM H. Nikolaus Schaller hns@goldelico.com wrote:
And I had a bad setup on one SD card some days ago for the LC15-eval board. This repeatably triggered a Freeze ca. 3 seconds into boot time (last messages were that LDO.8 and LDO.5 were initialized). The RESET button was responsive up to ca. 0.1 seconds before that, but not after.
I've seen such behavior too, it all just becomes super-unstable for no reason, and it persists resets. I then have to remove the battery for some time to recover, and then it comes back to hang-in-a-few-minutes mode. This is without changing anything on the SD card. Really hard to make any sense from all this.
Gražvydas
On x86 I had problems with a Bad idle driver before. Limiting the c-states fixed my problems until the idle driver got fixed. Do you have the possibilitly to disable powersave states in omap5 ?
Am 3. Oktober 2018 01:12:29 MESZ schrieb Grazvydas Ignotas notasas@gmail.com:
Some more tests:
- disabling LPAE has no effect, same hang after few minutes
- attempting to use 2GB MLO/u-boot on new 4GB board makes it very
unstable (crashes or hangs early while booting the kernel)
On Tue, Oct 2, 2018 at 6:29 PM Michael Mrozek EvilDragon@openpandora.org wrote:
ideas:
- perhaps the problem has something to do with idle/power saving.
Need
to check if there are RAM registers related to that. Does anyone
know
if it currently tries to do RAM self-refresh while idle?
Hm, but would idle / power saving trigger when the system is doing something? Because when doing my test runs, I always run a script that spits out the voltages, temperatures and remaining temperature left every 10 seconds.
And what does it do in between those 10s? If the system has no work for any periods, even as short as 1ms, it's called idle and power saving mechanisms start.
So Linux doesn't really idle around, but freezes anyways after some minutes. And sometimes it even freezes while booting - and it's anything but idle there.
It can be partially idle while booting, for example CPU can have nothing to do while data is arriving from the SD card (data is copied to RAM by SD card controller and CPU is not used). And because it is some important data, like loading systemd itself, kernel would have no other process to run and would put the CPU on idle for a short while.
- See if a non-LPAE-Kernel has a different effect
no
- Boot the kernel directly into a shell (without loading any other drivers except for the ones directly included in the kernel). Test whether this runs more stable and then modprobe the remaining
drivers
one after the other. After all, Tony was running the memtest
similar
as well, and that didn't fail.
Yeah that's something to try. Perhaps just move /lib/modules somewhere so they don't autoload (not sure if they go to the ramdisk currently).
On Tue, Oct 2, 2018 at 8:54 PM H. Nikolaus Schaller hns@goldelico.com wrote:
And I had a bad setup on one SD card some days ago for the LC15-eval
board.
This repeatably triggered a Freeze ca. 3 seconds into boot time (last messages were that LDO.8 and LDO.5 were initialized). The RESET
button
was responsive up to ca. 0.1 seconds before that, but not after.
I've seen such behavior too, it all just becomes super-unstable for no reason, and it persists resets. I then have to remove the battery for some time to recover, and then it comes back to hang-in-a-few-minutes mode. This is without changing anything on the SD card. Really hard to make any sense from all this.
Gražvydas _______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
On Mi, 2018-10-03 at 02:12 +0300, Grazvydas Ignotas wrote:
Some more tests:
- disabling LPAE has no effect, same hang after few minutes
- attempting to use 2GB MLO/u-boot on new 4GB board makes it very
unstable (crashes or hangs early while booting the kernel)
Ah, thanks for the info. It's interesting that LPAE doesn't have an effect, as it had one with our first 4GB RAM board tests, afair. But that was with a different kernel and DT files.
The 2GB MLO / u-boot has different timings, so it seems the timings certainly has an effect.
Hm, but would idle / power saving trigger when the system is doing something? Because when doing my test runs, I always run a script that spits out the voltages, temperatures and remaining temperature left every 10 seconds.
And what does it do in between those 10s? If the system has no work for any periods, even as short as 1ms, it's called idle and power saving mechanisms start.
Oh, okay. Didn't know powersaving is THAT aggressive. That's a good idea then!
BTW: When I ran the 4GB unit from the SD Card that Nikolaus gave me (with QuantumSTEP installed), it didn't fully freeze. While the Heartbeat of the LEDs stopped and it looked at first glance it had frozen, it was still running incredibly slow, as the touchscreen still reacted.
I didn't have it connected to serial out, so I have no idea if this was still reacting or not. But at least some part of i2c stopped working, hence the frozen LEDs.
Here's a video of that (don't mind the German comments ;)) https://cloud.openpandora.org/index.php/s/mfGobZfeFRyKS6P
I'm not sure whether it even really ever froze on QuantumStep. And this "half-freeze" usually happened after 5 - 10 minutes, so the same time it freezes with out Linux.
Not sure if this is the same or even related, but I thought I'd mention it.
- Boot the kernel directly into a shell (without loading any other drivers except for the ones directly included in the kernel).
Test whether this runs more stable and then modprobe the remaining drivers one after the other. After all, Tony was running the memtest similar as well, and that didn't fail.
Yeah that's something to try. Perhaps just move /lib/modules somewhere so they don't autoload (not sure if they go to the ramdisk currently).
Shouldn't it be enough booting into /bin/bash as initrd?
On Di, 2018-10-02 at 14:39 +0300, Grazvydas Ignotas wrote:
Quick update:
I've now renamed the modules-directory on my SD Card and booted it up normally.
It's now happily idling around for 45 minutes already - it never survived that long before.
I'll let it run as long as possible (no idea how much battery it has left), but it could very well be that one driver is causing the freeze.
I'm planning to setup a small script that modprobes one driver after the other, waits for 10 minutes, and then loads up the next one - until we have the freeze.
Question: Is there a way to find out what modules are being loaded normally? So I know which ones I need to load manually with the script?
Thanks!
Hello,
I'm back and I've done some tests, no useful conclusions though. I have 2 4GB boards now, I'll call them old one and new one.
- the old board also hangs, although it seems to take much longer (3
hours at least)
- the new one hangs in minutes, but occasionally it can run for half
an hour or so
- u-boot tester doesn't fail on new one, although I haven't tried it
for too long (less than 10 minutes)
- memtester (on Linux) doesn't fail on new one, it also seems to
allow the new board to survive much longer. I think it survived overnight but hung later when I was at work.
ideas:
- perhaps the problem has something to do with idle/power saving.
Need to check if there are RAM registers related to that. Does anyone know if it currently tries to do RAM self-refresh while idle?
- try without any TILER patches. I can vaguely remember it was said
the TILER can't be used at all because of some errata, but Matt proved otherwise. Maybe with 4GB/LPAE the problem actually triggers.
Gražvydas
On Mon, Oct 1, 2018 at 9:15 PM H. Nikolaus Schaller < hns@goldelico.com> wrote:
Hi Askarus,
Am 01.10.2018 um 19:45 schrieb Nino Bock <askarus@openpandora.org
:
Hello everyone,
in the forums EvilDragon said: "4GB RAM boards were working with only 2GB RAM enabled (as far as I remember - as that was the reason I ordered the RAM modules back then!)"
Did anyone with a 4GB Pyra try running it with only 2GB enabled?
Yes. We did that several times. Sometimes even unplanned (copying a non-LPAE kernel to the SD card).
Might it work if we only populate it with half the amount of RAM chips?
Unlikely. There are two separate busses from OMAP5 to 2 RAM chips each. So if one bus makes trouble it still will do if the other one is not used.
The only differences may be power demand and cross-talk.
These are important questions. If one of these options does actually work it will be fine for the first batch.
Of course it's better to fix things but we are at a state where we should start production as soon as possible. Either of these possibilities would bring some Pyras to life and be of no additional cost.
Maybe that's silly but if there is a chance that EvilDragon's memories are true, then for me it's obvious what to do.
BR, Nikolaus
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Lsmod will show you the currently loaded modules. https://linux.die.net/man/8/lsmod
Am 4. Oktober 2018 02:30:58 MESZ schrieb Michael Mrozek EvilDragon@openpandora.org:
On Di, 2018-10-02 at 14:39 +0300, Grazvydas Ignotas wrote:
Quick update:
I've now renamed the modules-directory on my SD Card and booted it up normally.
It's now happily idling around for 45 minutes already - it never survived that long before.
I'll let it run as long as possible (no idea how much battery it has left), but it could very well be that one driver is causing the freeze.
I'm planning to setup a small script that modprobes one driver after the other, waits for 10 minutes, and then loads up the next one - until we have the freeze.
Question: Is there a way to find out what modules are being loaded normally? So I know which ones I need to load manually with the script?
Thanks!
Hello,
I'm back and I've done some tests, no useful conclusions though. I have 2 4GB boards now, I'll call them old one and new one.
- the old board also hangs, although it seems to take much longer (3
hours at least)
- the new one hangs in minutes, but occasionally it can run for half
an hour or so
- u-boot tester doesn't fail on new one, although I haven't tried it
for too long (less than 10 minutes)
- memtester (on Linux) doesn't fail on new one, it also seems to
allow the new board to survive much longer. I think it survived overnight but hung later when I was at work.
ideas:
- perhaps the problem has something to do with idle/power saving.
Need to check if there are RAM registers related to that. Does anyone know if it currently tries to do RAM self-refresh while idle?
- try without any TILER patches. I can vaguely remember it was said
the TILER can't be used at all because of some errata, but Matt proved otherwise. Maybe with 4GB/LPAE the problem actually triggers.
Gražvydas
On Mon, Oct 1, 2018 at 9:15 PM H. Nikolaus Schaller < hns@goldelico.com> wrote:
Hi Askarus,
Am 01.10.2018 um 19:45 schrieb Nino Bock <askarus@openpandora.org
:
Hello everyone,
in the forums EvilDragon said: "4GB RAM boards were working with only 2GB RAM enabled (as far as I remember - as that was the reason I ordered the RAM modules back then!)"
Did anyone with a 4GB Pyra try running it with only 2GB enabled?
Yes. We did that several times. Sometimes even unplanned (copying a non-LPAE kernel to the SD card).
Might it work if we only populate it with half the amount of RAM chips?
Unlikely. There are two separate busses from OMAP5 to 2 RAM chips each. So if one bus makes trouble it still will do if the other one is not used.
The only differences may be power demand and cross-talk.
These are important questions. If one of these options does actually work it will be fine for the first batch.
Of course it's better to fix things but we are at a state where we should start production as soon as possible. Either of these possibilities would bring some Pyras to life and be of no additional cost.
Maybe that's silly but if there is a chance that EvilDragon's memories are true, then for me it's obvious what to do.
BR, Nikolaus
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Hi,
Am 04.10.2018 um 02:30 schrieb Michael Mrozek EvilDragon@openpandora.org:
On Di, 2018-10-02 at 14:39 +0300, Grazvydas Ignotas wrote:
Quick update:
I've now renamed the modules-directory on my SD Card and booted it up normally.
It's now happily idling around for 45 minutes already - it never survived that long before.
Interesting observation.
I am currently hunting another observation: on my lc15 eval board the lpae kernel freezes during boot with an older 2GB RAM board that othweise works without freezes... So it might have to do something with LPAE.
I'll let it run as long as possible (no idea how much battery it has left), but it could very well be that one driver is causing the freeze.
I'm planning to setup a small script that modprobes one driver after the other, waits for 10 minutes, and then loads up the next one - until we have the freeze.
Question: Is there a way to find out what modules are being loaded normally? So I know which ones I need to load manually with the script?
lsmod | cut -d ' ' -f 1 | fgrep -v 'Module'
omapdrm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm drm_panel_orientation_quirks bnep bluetooth ecdh_generic usb_f_ecm g_ether usb_f_rndis u_ether libcomposite configfs ipv6 arc4 wl18xx wlcore mac80211 snd_soc_omap_hdmi_audio snd_soc_dmic panel_boe_w677l cfg80211 dwc3 snd_soc_simple_card snd_soc_omap_abe_twl6040 snd_soc_simple_card_utils snd_soc_twl6040 leds_gpio snd_soc_gtm601 pwm_omap_dmtimer wwan_on_off encoder_tpd12s015 connector_hdmi omapdss snd_soc_w2cbw003_bt omapdss_base generic_adc_battery cec pwm_bl ehci_omap wlcore_sdio dwc3_omap snd_soc_ts3a227e bmp280_spi bq27xxx_battery_i2c leds_is31fl319x bq2429x_charger bq27xxx_battery crtouch_mt tsc2007 ina2xx tca8418_keypad as5013 gpio_twl6040 twl6040_vibra palmas_gpadc bmp280_i2c palmas_pwrbutton bmc150_magn_i2c bmc150_accel_i2c bmp280 bmc150_magn bmc150_accel_core bno055 industrialio_triggered_buffer kfifo_buf industrialio usb3503 w2cbw003_bluetooth snd_soc_omap_aess snd_soc_omap_mcpdm snd_soc_omap_mcbsp snd_soc_sdma
Some modules may be loaded automatically by another one.
Thanks!
Hello,
I'm back and I've done some tests, no useful conclusions though. I have 2 4GB boards now, I'll call them old one and new one.
- the old board also hangs, although it seems to take much longer (3
hours at least)
- the new one hangs in minutes, but occasionally it can run for half
an hour or so
- u-boot tester doesn't fail on new one, although I haven't tried it
for too long (less than 10 minutes)
- memtester (on Linux) doesn't fail on new one, it also seems to
allow the new board to survive much longer. I think it survived overnight but hung later when I was at work.
ideas:
- perhaps the problem has something to do with idle/power saving.
Need to check if there are RAM registers related to that. Does anyone know if it currently tries to do RAM self-refresh while idle?
- try without any TILER patches. I can vaguely remember it was said
the TILER can't be used at all because of some errata, but Matt proved otherwise. Maybe with 4GB/LPAE the problem actually triggers.
Gražvydas
On Mon, Oct 1, 2018 at 9:15 PM H. Nikolaus Schaller < hns@goldelico.com> wrote:
Hi Askarus,
Am 01.10.2018 um 19:45 schrieb Nino Bock <askarus@openpandora.org
:
Hello everyone,
in the forums EvilDragon said: "4GB RAM boards were working with only 2GB RAM enabled (as far as I remember - as that was the reason I ordered the RAM modules back then!)"
Did anyone with a 4GB Pyra try running it with only 2GB enabled?
Yes. We did that several times. Sometimes even unplanned (copying a non-LPAE kernel to the SD card).
Might it work if we only populate it with half the amount of RAM chips?
Unlikely. There are two separate busses from OMAP5 to 2 RAM chips each. So if one bus makes trouble it still will do if the other one is not used.
The only differences may be power demand and cross-talk.
These are important questions. If one of these options does actually work it will be fine for the first batch.
Of course it's better to fix things but we are at a state where we should start production as soon as possible. Either of these possibilities would bring some Pyras to life and be of no additional cost.
Maybe that's silly but if there is a chance that EvilDragon's memories are true, then for me it's obvious what to do.
BR, Nikolaus
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
On Do, 2018-10-04 at 07:51 +0200, H. Nikolaus Schaller wrote:
Well, didn't help it seems, just delayed it.
It now froze after 3 hours and 10 minutes.
I'll do another start to see if this was just random or if it also survives longer this time.
I am currently hunting another observation: on my lc15 eval board the lpae kernel freezes during boot with an older 2GB RAM board that othweise works without freezes... So it might have to do something with LPAE.
Hm, even though NOT using LPAE didn't fix the freeze for notaz' 4GB unit.
I'll let it run as long as possible (no idea how much battery it has left), but it could very well be that one driver is causing the freeze.
I'm planning to setup a small script that modprobes one driver after the other, waits for 10 minutes, and then loads up the next one - until we have the freeze.
Question: Is there a way to find out what modules are being loaded normally? So I know which ones I need to load manually with the script?
lsmod | cut -d ' ' -f 1 | fgrep -v 'Module'
omapdrm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm drm_panel_orientation_quirks bnep bluetooth ecdh_generic usb_f_ecm g_ether usb_f_rndis u_ether libcomposite configfs ipv6 arc4 wl18xx wlcore mac80211 snd_soc_omap_hdmi_audio snd_soc_dmic panel_boe_w677l cfg80211 dwc3 snd_soc_simple_card snd_soc_omap_abe_twl6040 snd_soc_simple_card_utils snd_soc_twl6040 leds_gpio snd_soc_gtm601 pwm_omap_dmtimer wwan_on_off encoder_tpd12s015 connector_hdmi omapdss snd_soc_w2cbw003_bt omapdss_base generic_adc_battery cec pwm_bl ehci_omap wlcore_sdio dwc3_omap snd_soc_ts3a227e bmp280_spi bq27xxx_battery_i2c leds_is31fl319x bq2429x_charger bq27xxx_battery crtouch_mt tsc2007 ina2xx tca8418_keypad as5013 gpio_twl6040 twl6040_vibra palmas_gpadc bmp280_i2c palmas_pwrbutton bmc150_magn_i2c bmc150_accel_i2c bmp280 bmc150_magn bmc150_accel_core bno055 industrialio_triggered_buffer kfifo_buf industrialio usb3503 w2cbw003_bluetooth snd_soc_omap_aess snd_soc_omap_mcpdm snd_soc_omap_mcbsp snd_soc_sdma
Some modules may be loaded automatically by another one.
Thanks!
Hello,
I'm back and I've done some tests, no useful conclusions though. I have 2 4GB boards now, I'll call them old one and new one.
- the old board also hangs, although it seems to take much longer
(3 hours at least)
- the new one hangs in minutes, but occasionally it can run for
half an hour or so
- u-boot tester doesn't fail on new one, although I haven't tried
it for too long (less than 10 minutes)
- memtester (on Linux) doesn't fail on new one, it also seems to
allow the new board to survive much longer. I think it survived overnight but hung later when I was at work.
ideas:
- perhaps the problem has something to do with idle/power saving.
Need to check if there are RAM registers related to that. Does anyone know if it currently tries to do RAM self-refresh while idle?
- try without any TILER patches. I can vaguely remember it was
said the TILER can't be used at all because of some errata, but Matt proved otherwise. Maybe with 4GB/LPAE the problem actually triggers.
Gražvydas
On Mon, Oct 1, 2018 at 9:15 PM H. Nikolaus Schaller < hns@goldelico.com> wrote:
Hi Askarus,
Am 01.10.2018 um 19:45 schrieb Nino Bock < askarus@openpandora.org
:
Hello everyone,
in the forums EvilDragon said: "4GB RAM boards were working with only 2GB RAM enabled (as far as I remember - as that was the reason I ordered the RAM modules back then!)"
Did anyone with a 4GB Pyra try running it with only 2GB enabled?
Yes. We did that several times. Sometimes even unplanned (copying a non-LPAE kernel to the SD card).
Might it work if we only populate it with half the amount of RAM chips?
Unlikely. There are two separate busses from OMAP5 to 2 RAM chips each. So if one bus makes trouble it still will do if the other one is not used.
The only differences may be power demand and cross-talk.
These are important questions. If one of these options does actually work it will be fine for the first batch.
Of course it's better to fix things but we are at a state where we should start production as soon as possible. Either of these possibilities would bring some Pyras to life and be of no additional cost.
Maybe that's silly but if there is a chance that EvilDragon's memories are true, then for me it's obvious what to do.
BR, Nikolaus
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
On Do, 2018-10-04 at 09:41 +0200, Michael Mrozek wrote:
Okay, it froze again after 1:14 hours this time.
So it's improved but still freezes (it gets very hot after a freeze, btw).
Just some idea:
* As notaz mentioned, it doesn't seem to freeze when the CPU is under load. Tonys memtest with CPU load seems to confirm that.
* I remember that Nikolaus mentioned a while ago that he did a full apt-get upgrade with the 4GB board and that worked without any freeze as well.
* It freezes pretty fast when it's idling
* It also freezes but not that fast when no drivers are loaded (when it's still idling).
So, here is my idea: Could it be that the freeze happens when the OMAP switches between voltages / frequencies (well, not always but sometimes)? Or maybe if it switches to ONE special voltage / frequency?
Because:
1. Under full load, it doesn't switch voltage / frequency 2. When idling with drivers loaded, it probably changes voltage / frequency often, as the drivers ocassionally do something 3. When idling WITHOUT all drivers loaded, a lot less is going on and it would switch voltage / frequency less often (therefore, less likely freezes)
That would explain the behaviour we have here.
Can someone create me a finished U-Boot / MLO (or wherever that information is stored) that only allows one fixed voltage / frequency?
Thanks!
On Do, 2018-10-04 at 07:51 +0200, H. Nikolaus Schaller wrote:
Well, didn't help it seems, just delayed it.
It now froze after 3 hours and 10 minutes.
I'll do another start to see if this was just random or if it also survives longer this time.
I am currently hunting another observation: on my lc15 eval board the lpae kernel freezes during boot with an older 2GB RAM board that othweise works without freezes... So it might have to do something with LPAE.
Hm, even though NOT using LPAE didn't fix the freeze for notaz' 4GB unit.
I'll let it run as long as possible (no idea how much battery it has left), but it could very well be that one driver is causing the freeze.
I'm planning to setup a small script that modprobes one driver after the other, waits for 10 minutes, and then loads up the next one - until we have the freeze.
Question: Is there a way to find out what modules are being loaded normally? So I know which ones I need to load manually with the script?
lsmod | cut -d ' ' -f 1 | fgrep -v 'Module'
omapdrm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm drm_panel_orientation_quirks bnep bluetooth ecdh_generic usb_f_ecm g_ether usb_f_rndis u_ether libcomposite configfs ipv6 arc4 wl18xx wlcore mac80211 snd_soc_omap_hdmi_audio snd_soc_dmic panel_boe_w677l cfg80211 dwc3 snd_soc_simple_card snd_soc_omap_abe_twl6040 snd_soc_simple_card_utils snd_soc_twl6040 leds_gpio snd_soc_gtm601 pwm_omap_dmtimer wwan_on_off encoder_tpd12s015 connector_hdmi omapdss snd_soc_w2cbw003_bt omapdss_base generic_adc_battery cec pwm_bl ehci_omap wlcore_sdio dwc3_omap snd_soc_ts3a227e bmp280_spi bq27xxx_battery_i2c leds_is31fl319x bq2429x_charger bq27xxx_battery crtouch_mt tsc2007 ina2xx tca8418_keypad as5013 gpio_twl6040 twl6040_vibra palmas_gpadc bmp280_i2c palmas_pwrbutton bmc150_magn_i2c bmc150_accel_i2c bmp280 bmc150_magn bmc150_accel_core bno055 industrialio_triggered_buffer kfifo_buf industrialio usb3503 w2cbw003_bluetooth snd_soc_omap_aess snd_soc_omap_mcpdm snd_soc_omap_mcbsp snd_soc_sdma
Some modules may be loaded automatically by another one.
Thanks!
Hello,
I'm back and I've done some tests, no useful conclusions though. I have 2 4GB boards now, I'll call them old one and new one.
- the old board also hangs, although it seems to take much
longer (3 hours at least)
- the new one hangs in minutes, but occasionally it can run for
half an hour or so
- u-boot tester doesn't fail on new one, although I haven't
tried it for too long (less than 10 minutes)
- memtester (on Linux) doesn't fail on new one, it also seems
to allow the new board to survive much longer. I think it survived overnight but hung later when I was at work.
ideas:
- perhaps the problem has something to do with idle/power
saving. Need to check if there are RAM registers related to that. Does anyone know if it currently tries to do RAM self-refresh while idle?
- try without any TILER patches. I can vaguely remember it was
said the TILER can't be used at all because of some errata, but Matt proved otherwise. Maybe with 4GB/LPAE the problem actually triggers.
Gražvydas
On Mon, Oct 1, 2018 at 9:15 PM H. Nikolaus Schaller < hns@goldelico.com> wrote:
Hi Askarus,
Am 01.10.2018 um 19:45 schrieb Nino Bock < askarus@openpandora.org > :
Hello everyone,
in the forums EvilDragon said: "4GB RAM boards were working with only 2GB RAM enabled (as far as I remember - as that was the reason I ordered the RAM modules back then!)"
Did anyone with a 4GB Pyra try running it with only 2GB enabled?
Yes. We did that several times. Sometimes even unplanned (copying a non-LPAE kernel to the SD card).
Might it work if we only populate it with half the amount of RAM chips?
Unlikely. There are two separate busses from OMAP5 to 2 RAM chips each. So if one bus makes trouble it still will do if the other one is not used.
The only differences may be power demand and cross-talk.
These are important questions. If one of these options does actually work it will be fine for the first batch.
Of course it's better to fix things but we are at a state where we should start production as soon as possible. Either of these possibilities would bring some Pyras to life and be of no additional cost.
Maybe that's silly but if there is a chance that EvilDragon's memories are true, then for me it's obvious what to do.
BR, Nikolaus
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Am 04.10.2018 um 19:13 schrieb Michael Mrozek EvilDragon@openpandora.org:
On Do, 2018-10-04 at 09:41 +0200, Michael Mrozek wrote:
Okay, it froze again after 1:14 hours this time.
So it's improved but still freezes (it gets very hot after a freeze, btw).
Just some idea:
- As notaz mentioned, it doesn't seem to freeze when the CPU is under
load. Tonys memtest with CPU load seems to confirm that.
- I remember that Nikolaus mentioned a while ago that he did a full
apt-get upgrade with the 4GB board and that worked without any freeze as well.
It freezes pretty fast when it's idling
It also freezes but not that fast when no drivers are loaded (when
it's still idling).
So, here is my idea: Could it be that the freeze happens when the OMAP switches between voltages / frequencies (well, not always but sometimes)? Or maybe if it switches to ONE special voltage / frequency?
I have tried that before without seeing a difference.
Because:
- Under full load, it doesn't switch voltage / frequency
- When idling with drivers loaded, it probably changes voltage /
frequency often, as the drivers ocassionally do something 3. When idling WITHOUT all drivers loaded, a lot less is going on and it would switch voltage / frequency less often (therefore, less likely freezes)
That would explain the behaviour we have here.
Can someone create me a finished U-Boot / MLO (or wherever that information is stored) that only allows one fixed voltage / frequency?
It is defined in DTS. Attached is a patch.
Thanks!
On Do, 2018-10-04 at 07:51 +0200, H. Nikolaus Schaller wrote:
Well, didn't help it seems, just delayed it.
It now froze after 3 hours and 10 minutes.
I'll do another start to see if this was just random or if it also survives longer this time.
I am currently hunting another observation: on my lc15 eval board the lpae kernel freezes during boot with an older 2GB RAM board that othweise works without freezes... So it might have to do something with LPAE.
Hm, even though NOT using LPAE didn't fix the freeze for notaz' 4GB unit.
I'll let it run as long as possible (no idea how much battery it has left), but it could very well be that one driver is causing the freeze.
I'm planning to setup a small script that modprobes one driver after the other, waits for 10 minutes, and then loads up the next one - until we have the freeze.
Question: Is there a way to find out what modules are being loaded normally? So I know which ones I need to load manually with the script?
lsmod | cut -d ' ' -f 1 | fgrep -v 'Module'
omapdrm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm drm_panel_orientation_quirks bnep bluetooth ecdh_generic usb_f_ecm g_ether usb_f_rndis u_ether libcomposite configfs ipv6 arc4 wl18xx wlcore mac80211 snd_soc_omap_hdmi_audio snd_soc_dmic panel_boe_w677l cfg80211 dwc3 snd_soc_simple_card snd_soc_omap_abe_twl6040 snd_soc_simple_card_utils snd_soc_twl6040 leds_gpio snd_soc_gtm601 pwm_omap_dmtimer wwan_on_off encoder_tpd12s015 connector_hdmi omapdss snd_soc_w2cbw003_bt omapdss_base generic_adc_battery cec pwm_bl ehci_omap wlcore_sdio dwc3_omap snd_soc_ts3a227e bmp280_spi bq27xxx_battery_i2c leds_is31fl319x bq2429x_charger bq27xxx_battery crtouch_mt tsc2007 ina2xx tca8418_keypad as5013 gpio_twl6040 twl6040_vibra palmas_gpadc bmp280_i2c palmas_pwrbutton bmc150_magn_i2c bmc150_accel_i2c bmp280 bmc150_magn bmc150_accel_core bno055 industrialio_triggered_buffer kfifo_buf industrialio usb3503 w2cbw003_bluetooth snd_soc_omap_aess snd_soc_omap_mcpdm snd_soc_omap_mcbsp snd_soc_sdma
Some modules may be loaded automatically by another one.
Thanks!
Hello,
I'm back and I've done some tests, no useful conclusions though. I have 2 4GB boards now, I'll call them old one and new one.
- the old board also hangs, although it seems to take much
longer (3 hours at least)
- the new one hangs in minutes, but occasionally it can run for
half an hour or so
- u-boot tester doesn't fail on new one, although I haven't
tried it for too long (less than 10 minutes)
- memtester (on Linux) doesn't fail on new one, it also seems
to allow the new board to survive much longer. I think it survived overnight but hung later when I was at work.
ideas:
- perhaps the problem has something to do with idle/power
saving. Need to check if there are RAM registers related to that. Does anyone know if it currently tries to do RAM self-refresh while idle?
- try without any TILER patches. I can vaguely remember it was
said the TILER can't be used at all because of some errata, but Matt proved otherwise. Maybe with 4GB/LPAE the problem actually triggers.
Gražvydas
On Mon, Oct 1, 2018 at 9:15 PM H. Nikolaus Schaller < hns@goldelico.com> wrote:
Hi Askarus,
> Am 01.10.2018 um 19:45 schrieb Nino Bock < > askarus@openpandora.org >> : > > Hello everyone, > > in the forums EvilDragon said: > "4GB RAM boards were working with only 2GB RAM enabled (as > far as > I remember - as that was the reason I ordered the RAM > modules > back then!)" > > Did anyone with a 4GB Pyra try running it with only 2GB > enabled?
Yes. We did that several times. Sometimes even unplanned (copying a non-LPAE kernel to the SD card).
> Might it work if we only populate it with half the amount > of > RAM > chips?
Unlikely. There are two separate busses from OMAP5 to 2 RAM chips each. So if one bus makes trouble it still will do if the other one is not used.
The only differences may be power demand and cross-talk.
> These are important questions. > If one of these options does actually work it will be fine > for > the first batch. > > Of course it's better to fix things but we are at a state > where > we should start production as soon as possible. > Either of these possibilities would bring some Pyras to > life > and > be of no additional cost. > > Maybe that's silly but if there is a chance that > EvilDragon's > memories are true, then for me it's obvious what to do.
BR, Nikolaus
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
* H. Nikolaus Schaller hns@goldelico.com [181005 05:36]:
Am 04.10.2018 um 19:13 schrieb Michael Mrozek EvilDragon@openpandora.org: On Do, 2018-10-04 at 09:41 +0200, Michael Mrozek wrote:
- It also freezes but not that fast when no drivers are loaded (when
it's still idling).
So, here is my idea: Could it be that the freeze happens when the OMAP switches between voltages / frequencies (well, not always but sometimes)? Or maybe if it switches to ONE special voltage / frequency?
I have tried that before without seeing a difference.
Well for the idle freezing issue, curiously the old "disable internal dll" trick seems to work. At least for me the system keeps running just fine idling for several hours.
Also, looks like we have SDRAM_CONFIG ROWSIZE bits [9:7] set to 15 while it should be 16 for 8Gb memory. But I think that is just cosmetic, not sure if that is actually used.
Anyways, care to test with the following hack with internal dll disabled and ROWSIZE fixed?
If that works for the idle system problem, then we probably have some DLL timing related value misconfigured.
Regards,
Tony
8< -------------- diff --git a/board/goldelico/letux-cortex15/lc15.c b/board/goldelico/letux-cortex15/lc15.c --- a/board/goldelico/letux-cortex15/lc15.c +++ b/board/goldelico/letux-cortex15/lc15.c @@ -37,8 +37,8 @@ const struct omap_sysinfo sysinfo = { */
const struct emif_regs emif_regs_ddr3_532_mhz_2cs_es2 = { - .sdram_config_init = 0x61851B3A, - .sdram_config = 0x61851B3A, + .sdram_config_init = 0x61951BBA, + .sdram_config = 0x61951BBA, .sdram_config2 = 0x0, .ref_ctrl = 0x000040F1, .ref_ctrl_final = 0x00001035,
On Fri, Oct 5, 2018 at 5:49 PM Tony Lindgren tony@atomide.com wrote:
- H. Nikolaus Schaller hns@goldelico.com [181005 05:36]:
Am 04.10.2018 um 19:13 schrieb Michael Mrozek EvilDragon@openpandora.org: On Do, 2018-10-04 at 09:41 +0200, Michael Mrozek wrote:
- It also freezes but not that fast when no drivers are loaded (when
it's still idling).
So, here is my idea: Could it be that the freeze happens when the OMAP switches between voltages / frequencies (well, not always but sometimes)? Or maybe if it switches to ONE special voltage / frequency?
I have tried that before without seeing a difference.
Well for the idle freezing issue, curiously the old "disable internal dll" trick seems to work. At least for me the system keeps running just fine idling for several hours.
Confirmed, had it idle 15h and then running 2x memtester for 4h. As a side note, the temperature reached 82C @1GHz with the memtesters but it kept going. This is in the latest case (I think?) with the heatsinks.
Also, looks like we have SDRAM_CONFIG ROWSIZE bits [9:7] set to 15 while it should be 16 for 8Gb memory. But I think that is just cosmetic, not sure if that is actually used.
With only ROWSIZE adjusted I've got an undefined instruction panic after a few minutes. Second try is still running for over an hour, but I guess only the ROWSIZE change is insufficient (if it has any effect at all).
Gražvydas
On Fr, 2018-10-05 at 07:49 -0700, Tony Lindgren wrote:
Hi,
hm, that is interesting. I tried to find out a bit about the DLL on the internet (as the memory datasheet simply says that it should never be disabled for production use), but the only I did find it this article about the BIOS setup: https://www.techarp.com/bios-guide/digital-locked-loop-dll/
The interesting: "However, the delay-locked loop circuit of every DDR, DDR2, DDR3 or DDR4 chip is tuned for a certain fixed frequency range. If you run the chip beyond that frequency rate, the DLL circuit may not work correctly. That’s why DDR, DDR2, DDR3 and DDR4 SDRAM chips can have problems running at clock speeds slower than what they are rated for."
I wonder - what frequency are we running the memory with? Could it be that frequency is off?
- H. Nikolaus Schaller hns@goldelico.com [181005 05:36]:
Am 04.10.2018 um 19:13 schrieb Michael Mrozek < EvilDragon@openpandora.org>: On Do, 2018-10-04 at 09:41 +0200, Michael Mrozek wrote:
- It also freezes but not that fast when no drivers are loaded
(when it's still idling).
So, here is my idea: Could it be that the freeze happens when the OMAP switches between voltages / frequencies (well, not always but sometimes)? Or maybe if it switches to ONE special voltage / frequency?
I have tried that before without seeing a difference.
Well for the idle freezing issue, curiously the old "disable internal dll" trick seems to work. At least for me the system keeps running just fine idling for several hours.
Also, looks like we have SDRAM_CONFIG ROWSIZE bits [9:7] set to 15 while it should be 16 for 8Gb memory. But I think that is just cosmetic, not sure if that is actually used.
Anyways, care to test with the following hack with internal dll disabled and ROWSIZE fixed?
If that works for the idle system problem, then we probably have some DLL timing related value misconfigured.
Regards,
Tony
8< -------------- diff --git a/board/goldelico/letux-cortex15/lc15.c b/board/goldelico/letux-cortex15/lc15.c --- a/board/goldelico/letux-cortex15/lc15.c +++ b/board/goldelico/letux-cortex15/lc15.c @@ -37,8 +37,8 @@ const struct omap_sysinfo sysinfo = { */
const struct emif_regs emif_regs_ddr3_532_mhz_2cs_es2 = {
- .sdram_config_init = 0x61851B3A,
- .sdram_config = 0x61851B3A,
- .sdram_config_init = 0x61951BBA,
- .sdram_config = 0x61951BBA, .sdram_config2 = 0x0, .ref_ctrl = 0x000040F1, .ref_ctrl_final = 0x00001035,
Hi,
Am 08.10.2018 um 05:09 schrieb Michael Mrozek EvilDragon@openpandora.org:
On Fr, 2018-10-05 at 07:49 -0700, Tony Lindgren wrote:
Hi,
hm, that is interesting. I tried to find out a bit about the DLL on the internet (as the memory datasheet simply says that it should never be disabled for production use), but the only I did find it this article about the BIOS setup: https://www.techarp.com/bios-guide/digital-locked-loop-dll/
The interesting: "However, the delay-locked loop circuit of every DDR, DDR2, DDR3 or DDR4 chip is tuned for a certain fixed frequency range. If you run the chip beyond that frequency rate, the DLL circuit may not work correctly. That’s why DDR, DDR2, DDR3 and DDR4 SDRAM chips can have problems running at clock speeds slower than what they are rated for."
That is indeed an interesting finding!
I wonder - what frequency are we running the memory with?
AFAIR, the OMAP5 EMIF runs at 532 MHz while the chips are specified for 1066 (or 1200?) MHz. So that would be half clock rate.
Could it be that frequency is off?
IMHO: yes!
BR, Nikolaus
- H. Nikolaus Schaller hns@goldelico.com [181005 05:36]:
Am 04.10.2018 um 19:13 schrieb Michael Mrozek < EvilDragon@openpandora.org>: On Do, 2018-10-04 at 09:41 +0200, Michael Mrozek wrote:
- It also freezes but not that fast when no drivers are loaded
(when it's still idling).
So, here is my idea: Could it be that the freeze happens when the OMAP switches between voltages / frequencies (well, not always but sometimes)? Or maybe if it switches to ONE special voltage / frequency?
I have tried that before without seeing a difference.
Well for the idle freezing issue, curiously the old "disable internal dll" trick seems to work. At least for me the system keeps running just fine idling for several hours.
Also, looks like we have SDRAM_CONFIG ROWSIZE bits [9:7] set to 15 while it should be 16 for 8Gb memory. But I think that is just cosmetic, not sure if that is actually used.
Anyways, care to test with the following hack with internal dll disabled and ROWSIZE fixed?
If that works for the idle system problem, then we probably have some DLL timing related value misconfigured.
Regards,
Tony
8< -------------- diff --git a/board/goldelico/letux-cortex15/lc15.c b/board/goldelico/letux-cortex15/lc15.c --- a/board/goldelico/letux-cortex15/lc15.c +++ b/board/goldelico/letux-cortex15/lc15.c @@ -37,8 +37,8 @@ const struct omap_sysinfo sysinfo = { */
const struct emif_regs emif_regs_ddr3_532_mhz_2cs_es2 = {
- .sdram_config_init = 0x61851B3A,
- .sdram_config = 0x61851B3A,
- .sdram_config_init = 0x61951BBA,
- .sdram_config = 0x61951BBA, .sdram_config2 = 0x0, .ref_ctrl = 0x000040F1, .ref_ctrl_final = 0x00001035,
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
Am Montag, den 08.10.2018, 08:29 +0200 schrieb H. Nikolaus Schaller:
Ah well... that would explain things indeed.
The speed ratings in the datasheet for the memory are: DDR3-800 DDR3-1066 DDR3-1333 DDR3-1600 DDR3-1866
So if we're using 532, we're indeed running at a lower clock speed (unless it means dual-die is 2x532 - but even then we're off, as that would be 1064 instead of 1066... very weird).. so it should be 533.
Is there a reason the OMAP is set to 532? Can't it go higher?
And checking the JEDEC documents, there are only the above mentioned speeds for DDR3, 532 (or 533) isn't mentioned anywhere.
Hi,
Am 08.10.2018 um 05:09 schrieb Michael Mrozek EvilDragon@openpandora.org:
On Fr, 2018-10-05 at 07:49 -0700, Tony Lindgren wrote:
Hi,
hm, that is interesting. I tried to find out a bit about the DLL on the internet (as the memory datasheet simply says that it should never be disabled for production use), but the only I did find it this article about the BIOS setup: https://www.techarp.com/bios-guide/digital-locked-loop-dll/
The interesting: "However, the delay-locked loop circuit of every DDR, DDR2, DDR3 or DDR4 chip is tuned for a certain fixed frequency range. If you run the chip beyond that frequency rate, the DLL circuit may not work correctly. That’s why DDR, DDR2, DDR3 and DDR4 SDRAM chips can have problems running at clock speeds slower than what they are rated for."
That is indeed an interesting finding!
I wonder - what frequency are we running the memory with?
AFAIR, the OMAP5 EMIF runs at 532 MHz while the chips are specified for 1066 (or 1200?) MHz. So that would be half clock rate.
Could it be that frequency is off?
IMHO: yes!
BR, Nikolaus
- H. Nikolaus Schaller hns@goldelico.com [181005 05:36]:
Am 04.10.2018 um 19:13 schrieb Michael Mrozek < EvilDragon@openpandora.org>: On Do, 2018-10-04 at 09:41 +0200, Michael Mrozek wrote:
- It also freezes but not that fast when no drivers are loaded
(when it's still idling).
So, here is my idea: Could it be that the freeze happens when the OMAP switches between voltages / frequencies (well, not always but sometimes)? Or maybe if it switches to ONE special voltage / frequency?
I have tried that before without seeing a difference.
Well for the idle freezing issue, curiously the old "disable internal dll" trick seems to work. At least for me the system keeps running just fine idling for several hours.
Also, looks like we have SDRAM_CONFIG ROWSIZE bits [9:7] set to 15 while it should be 16 for 8Gb memory. But I think that is just cosmetic, not sure if that is actually used.
Anyways, care to test with the following hack with internal dll disabled and ROWSIZE fixed?
If that works for the idle system problem, then we probably have some DLL timing related value misconfigured.
Regards,
Tony
8< -------------- diff --git a/board/goldelico/letux-cortex15/lc15.c b/board/goldelico/letux-cortex15/lc15.c --- a/board/goldelico/letux-cortex15/lc15.c +++ b/board/goldelico/letux-cortex15/lc15.c @@ -37,8 +37,8 @@ const struct omap_sysinfo sysinfo = { */
const struct emif_regs emif_regs_ddr3_532_mhz_2cs_es2 = {
- .sdram_config_init = 0x61851B3A,
- .sdram_config = 0x61851B3A,
- .sdram_config_init = 0x61951BBA,
- .sdram_config = 0x61951BBA, .sdram_config2 = 0x0, .ref_ctrl = 0x000040F1, .ref_ctrl_final = 0x00001035,
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
On Mon, Oct 8, 2018 at 2:39 PM Michael Mrozek EvilDragon@openpandora.org wrote:
Am Montag, den 08.10.2018, 08:29 +0200 schrieb H. Nikolaus Schaller:
Ah well... that would explain things indeed.
The speed ratings in the datasheet for the memory are: DDR3-800 DDR3-1066 DDR3-1333 DDR3-1600 DDR3-1866
So if we're using 532, we're indeed running at a lower clock speed (unless it means dual-die is 2x532 - but even then we're off, as that would be 1064 instead of 1066... very weird).. so it should be 533.
Is there a reason the OMAP is set to 532? Can't it go higher?
532MHz is specified as the maximum supported frequency by the OMAP5 datasheet. And DDR3-1066 (for example) refers to bits per second per pin, as it's DDR (dual data rate) it's 533MHz.
So which RAM chips do we have, DDR3-1866?
We could try overclocking OMAP to 533, it's < 1% overclock so maybe it would not hurt. But I guess there must be a reason why TI capped it to 532.
Gražvydas
Am Montag, den 08.10.2018, 15:09 +0300 schrieb Grazvydas Ignotas:
Hi,
So if we're using 532, we're indeed running at a lower clock speed (unless it means dual-die is 2x532 - but even then we're off, as that would be 1064 instead of 1066... very weird).. so it should be 533. Is there a reason the OMAP is set to 532? Can't it go higher?
532MHz is specified as the maximum supported frequency by the OMAP5 datasheet. And DDR3-1066 (for example) refers to bits per second per pin, as it's DDR (dual data rate) it's 533MHz.
That's pretty weird, if the JEDEC Standard is 533MHz.
So which RAM chips do we have, DDR3-1866?
No, we have the DDR3L-1600 (11-11-11).
We could try overclocking OMAP to 533, it's < 1% overclock so maybe it would not hurt. But I guess there must be a reason why TI capped it to 532.
The 2GB is also clocked at 532MHz then? Maybe this is why some 2GB RAM boards (even uEVMs) freeze randomly as well.
With DLL enabled, it might work fine with most of them, but for some, the tolerance level might be worse, so it causes hiccups.
Am Montag, den 08.10.2018, 08:29 +0200 schrieb H. Nikolaus Schaller:
Just noticed: Clock rate and data rate are different.
For example, for the IMGTech memory, the website says:
Data Rate: DDR3-1333 Clock Rate: 667MHz
And the Samsung Data sheet says:
400 MHz f CK for 800Mb/sec/pin, 533MHz f CK for 1066Mb/sec/pin, 667MHz f CK for 1333Mb/sec/pin, 800MHz f CK for 1600Mb/sec/pin, 933MHz f CK for 1866Mb/sec/pin,
So 533MHz should be fine.
Hi,
Am 08.10.2018 um 05:09 schrieb Michael Mrozek EvilDragon@openpandora.org:
On Fr, 2018-10-05 at 07:49 -0700, Tony Lindgren wrote:
Hi,
hm, that is interesting. I tried to find out a bit about the DLL on the internet (as the memory datasheet simply says that it should never be disabled for production use), but the only I did find it this article about the BIOS setup: https://www.techarp.com/bios-guide/digital-locked-loop-dll/
The interesting: "However, the delay-locked loop circuit of every DDR, DDR2, DDR3 or DDR4 chip is tuned for a certain fixed frequency range. If you run the chip beyond that frequency rate, the DLL circuit may not work correctly. That’s why DDR, DDR2, DDR3 and DDR4 SDRAM chips can have problems running at clock speeds slower than what they are rated for."
That is indeed an interesting finding!
I wonder - what frequency are we running the memory with?
AFAIR, the OMAP5 EMIF runs at 532 MHz while the chips are specified for 1066 (or 1200?) MHz. So that would be half clock rate.
Could it be that frequency is off?
IMHO: yes!
BR, Nikolaus
- H. Nikolaus Schaller hns@goldelico.com [181005 05:36]:
Am 04.10.2018 um 19:13 schrieb Michael Mrozek < EvilDragon@openpandora.org>: On Do, 2018-10-04 at 09:41 +0200, Michael Mrozek wrote:
- It also freezes but not that fast when no drivers are loaded
(when it's still idling).
So, here is my idea: Could it be that the freeze happens when the OMAP switches between voltages / frequencies (well, not always but sometimes)? Or maybe if it switches to ONE special voltage / frequency?
I have tried that before without seeing a difference.
Well for the idle freezing issue, curiously the old "disable internal dll" trick seems to work. At least for me the system keeps running just fine idling for several hours.
Also, looks like we have SDRAM_CONFIG ROWSIZE bits [9:7] set to 15 while it should be 16 for 8Gb memory. But I think that is just cosmetic, not sure if that is actually used.
Anyways, care to test with the following hack with internal dll disabled and ROWSIZE fixed?
If that works for the idle system problem, then we probably have some DLL timing related value misconfigured.
Regards,
Tony
8< -------------- diff --git a/board/goldelico/letux-cortex15/lc15.c b/board/goldelico/letux-cortex15/lc15.c --- a/board/goldelico/letux-cortex15/lc15.c +++ b/board/goldelico/letux-cortex15/lc15.c @@ -37,8 +37,8 @@ const struct omap_sysinfo sysinfo = { */
const struct emif_regs emif_regs_ddr3_532_mhz_2cs_es2 = {
- .sdram_config_init = 0x61851B3A,
- .sdram_config = 0x61851B3A,
- .sdram_config_init = 0x61951BBA,
- .sdram_config = 0x61951BBA, .sdram_config2 = 0x0, .ref_ctrl = 0x000040F1, .ref_ctrl_final = 0x00001035,
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
* H. Nikolaus Schaller hns@goldelico.com [181008 06:33]:
Hi,
Am 08.10.2018 um 05:09 schrieb Michael Mrozek EvilDragon@openpandora.org:
On Fr, 2018-10-05 at 07:49 -0700, Tony Lindgren wrote:
Hi,
hm, that is interesting. I tried to find out a bit about the DLL on the internet (as the memory datasheet simply says that it should never be disabled for production use), but the only I did find it this article about the BIOS setup: https://www.techarp.com/bios-guide/digital-locked-loop-dll/
The interesting: "However, the delay-locked loop circuit of every DDR, DDR2, DDR3 or DDR4 chip is tuned for a certain fixed frequency range. If you run the chip beyond that frequency rate, the DLL circuit may not work correctly. That’s why DDR, DDR2, DDR3 and DDR4 SDRAM chips can have problems running at clock speeds slower than what they are rated for."
That is indeed an interesting finding!
Yeah I agree that might explain why things hang at random if dll fails to lock coming out of self-refresh. And that would also explain the complete hang as DDR3 is not functional at all coming out of idle.
Regards,
Tony
On Do, 2018-09-20 at 09:29 -0700, Tony Lindgren wrote:
Hi Tony,
well, that change seems to have an effect - but it seems to WORSEN things.
None of the three Pyras survived longer than 3 minutes now.
And it's definitely not the temperature: Sat Jan 1 00:02:41 UTC 2000 35�° 35�° 37�° 22�° 3564mV 1000MHz 00:02:41 up 2 min, 1 user, load average: 1.96, 0.96, 0.38
(this was the last entry one of the Pyras spit out).
mtest from U-Boot, however, was working fine (I had it running for over 100 iterations).
None of the units have ever crashed running U-Boot - but then again, a lot less is going on there.
How did you run your memtest? From a loaded kernel or from U-Boot as well?
BTW: One unit spit out the following when it froze:
[ 236.148138] Unable to handle kernel NULL pointer dereference at virtual address 0000022b [ 236.148283] Unable to handle kernel paging request at virtual address a0100101 [ 236.148418] Unable to handle kernel paging request at virtual address a0100101 [ 236.148553] Unable to handle kernel paging request at virtual address a0100101 [ 236.148687] Unable to handle kernel paging request at virtual address a0100101 [ 236.148821] Unable to handle kernel paging request at virtual address a0100101 [ 236.148955] Unable to handle kernel paging request at virtual address a0100101 [ 236.149089] Unable to handle kernel paging request at virtual address a0100101 [ 236.149221] Unable to handle kernel paging request at virtual address a0100101 [ 236.149354] Unable to handle kernel paging request at virtual address a0100101 [ 236.149486] Unable to handle kernel paging request at virtual address a0100101 [ 236.149618] Unable to handle kernel paging request at virtual address a0100101 [ 236.149751] Unable to handle kernel paging request at virtual address a0100101 [ 236.149883] Unable to handle kernel paging request at virtual address a0100101 [ 236.150016] Unable to handle kernel paging request at virtual address a0100101 [ 236.150148] Unable to handle kernel paging request at virtual address a0100101 [ 236.150282] Unable to handle kernel paging request at virtual address a0100101 [ 236.150415] Unable to handle kernel paging request at virtual address a0100101 [ 236.150548] Unable to handle kernel paging request at virtual address a0100101 [ 236.150682] Unable to handle kernel paging request at virtual address a0100101 [ 236.150814] Unable to handle kernel paging request at virtual address a0100101 [ 236.150947] Unable to handle kernel paging request at virtual address a0100101 [ 236.151079] Unable to handle kernel paging request at virtual address a0100101 [ 236.151212] Unable to handle kernel paging request at virtual address a0100101 [ 236.358133] Unable to handle kernel NULL pointer dereference at virtual address 0000026b
Does that help?
@Tony: We do have an adaptor for JTAG access available for the Pyra, but we have no one who know is very knowledgable with that. Have you ever used the JTAG and would that help you debugging the memory issue? Then Nikolaus could send that adaptor to you.
- H. Nikolaus Schaller hns@goldelico.com [180920 14:13]:
Am 20.09.2018 um 15:57 schrieb Michael Mrozek < EvilDragon@openpandora.org>:
They are in a big array of magic hex numbers...
http://git.goldelico.com/?p=letux-uboot.git;a=blob;f=board/goldelico/letux-c...
The datasheet usually has minimum and maximum listed, so I should be able to safely change them.
I'd change one value at a time between max and min to see if it improves things (or worsens them).
Also, simply comparing the values to the datasheet might help as well.
Yes please do check again :)
Can you please try maxing out .sdram_tim3 T_RFC value from 0x027f8be8 to 0x027f8ff8, see commit 2b143578a07afe0e50567f8327f4f62e0703cb20.
Regards,
Tony
* Michael Mrozek EvilDragon@openpandora.org [180926 03:47]:
On Do, 2018-09-20 at 09:29 -0700, Tony Lindgren wrote:
Hi Tony,
well, that change seems to have an effect - but it seems to WORSEN things.
None of the three Pyras survived longer than 3 minutes now.
Hmm interesting. What about the datasheet value mentioned in one of my commits? I used a slightly larger value to make the old CPU board revisions to work.
How did you run your memtest? From a loaded kernel or from U-Boot as well?
Only with kernel with two instances of memtester running.
Does that help?
If the memory is not working the errors are usually random oopses. Not sure why it would always just hang.
@Tony: We do have an adaptor for JTAG access available for the Pyra, but we have no one who know is very knowledgable with that. Have you ever used the JTAG and would that help you debugging the memory issue? Then Nikolaus could send that adaptor to you.
Sure I could use one. Also the main board in my rack is an old version with no WLAN working that forces me to use an external SDIO WLAN adapter on it so I'd like to replace that if possible.
Regards,
Tony
Just a quick summary with some questions with what we know so far, maybe it helps finding ideas.
1. With the same rootfs and kernel version, 2GB RAM versions seem to run stable, 4GB RAM version don't. Besides the RAM, they use a different MLO/U-Boot and a different kernel (LPAE-enabled).
2. The maxed out value seem to make the unit freeze a lot more often.
3. A freeze is often accompanied with graphical glitches on the screen
4. But the serial log usually doesn't show anything. It simply freezes. Shouldn't a kernel panic occur when there are memory errors? Sure, if the error is THAT bad that it will instantly freeze the CPU, then nothing will happen - but how likely is it that Linux NEVER reports bad memory but instead freezes right away?
5. U-Boot has never crashed so far. I had memory errors with mtest on one PCB with the older sdram3-values, but I have only run the test there once. At least with the new values, all of them seem to run fine (tested for over 100 iterations)
6. According to the simulation using the IBIS files, the DRAM- and PALMAS-Connections are within the given specs of TI. So while a hardware error IS still possible, it should hopefully be not the reason. One thing to notice is that the improved PCBs (that are a lot better according to the simulation) don't seem to have improved at all compared to the older PCBs (which were out of specs according to the simulation). One should expect at least a bit of improvement.
As the 2GB RAM boards are working fine, we should check what's DIFFERENT with the 4GB RAM boards.
1. Obviously, the RAM Chips are very different (two chips in one, probably needing twice the bus speed, correct?) as well as their timings.
2. DTB-File used
3. Kernel (2GB RAM: non-LPAE). Do we even need LPAE? non-LPAE should support memory up to 4GB...?
4. Power usage - the double-die memory chips might use more power, maybe?
So, anyone has any ideas what to try? I'm ready to test everything ASAP :)
On 26/09/2018 04:57, Michael Mrozek wrote:
- The maxed out value seem to make the unit freeze a lot more often.
what happens when you try the most conservative settings possible?
I'm not sure how to set that up. We've got a datasheet for the RAM, a calculator toolsheet from TI and then it spits out Hex values you use in your Bootloader to setup the RAM. For most of them, there's not really a max or min - but that also includes the value we have changed, so there are possible ways to tweak the stuff.
I have some hex values which are different from what the datasheet spits out - I've asked about them here a couple of days ago. Tony is probably our best guess to know about that, so I'm hoping he can help out a bit more.
I do wonder though if we can dump the values of the running RAM somehow, so we can make sure they are what they should be.
This would be especially helpful with some of the config settings. If hardware levelling is switched off, for example, then it would be a miracle if the RAM works fine...
On 26 September 2018 10:49:07 CEST, Mr C Camacho chriscamacho@virginmedia.com wrote:
On 26/09/2018 04:57, Michael Mrozek wrote:
- The maxed out value seem to make the unit freeze a lot more often.
what happens when you try the most conservative settings possible?
-- http://Bedroomcoders.co.uk (coding adventures of a geek) software and electronics blog _______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
* Michael Mrozek EvilDragon@openpandora.org [180926 09:05]:
I have some hex values which are different from what the datasheet spits out - I've asked about them here a couple of days ago. Tony is probably our best guess to know about that, so I'm hoping he can help out a bit more.
Not sure I know which message you refer to here, what's the message-id for that mail so I can check if I have it here?
Regards,
Tony
Am Mittwoch, den 26.09.2018, 07:46 -0700 schrieb Tony Lindgren:
That a mail full of questions regarding the timings, written 2018-09-20.
Message-ID: 398d8a20c2b5b608726e20c8028be752c98cced7.camel@openpandora.org
Topic 6 lists my findings which are different from what the tool spit out.
- Michael Mrozek EvilDragon@openpandora.org [180926 09:05]:
I have some hex values which are different from what the datasheet spits out
- I've asked about them here a couple of days ago.
Tony is probably our best guess to know about that, so I'm hoping he can help out a bit more.
Not sure I know which message you refer to here, what's the message-id for that mail so I can check if I have it here?
Regards,
Tony
On 9/20/18 3:57 PM, Michael Mrozek wrote:
Am Donnerstag, den 20.09.2018, 15:45 +0200 schrieb H. Nikolaus Schaller:
Hi,
Or: Can we change some EMIF values to raise stability (maybe at the cost of tiny bit performance?
As far as I remember, Tony did send some EMIF setup patch that is in notaz and letux uboot now, but noted that it might be reasonable to change some more parameters. Unfortunately I do not really understand more about them than you... So I have never played with them.
I thought EMIF is in the MLO, not in U-boot...? Or is the MLO combined with U-Boot on the Pyra?
I'm confused :)
Well, I'm perfectly fine doing trial and error with the values.
The datasheet usually has minimum and maximum listed, so I should be able to safely change them.
I'd change one value at a time between max and min to see if it improves things (or worsens them).
Also, simply comparing the values to the datasheet might help as well.
https://dev.pyra-handheld.com/notaz/pyra-u-boot/commit/4063458c6290b4e15df9d...
That has some explanation about the timings.
To compile I use make -j6 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- distclean make -j6 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- pandora_pyra_lc15_defconfig make -j6 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-
for the 4gb version use pandora_pyra_lc15_4gb_defconfig
To install to sd (for pyraos at least, letux uses a different method, and im not sure if copying a new MLO file on top of the old one will actually work there. for letux you need to use the letux uboot anyway)
dd if="MLO" of=${DISK} count=1 seek=1 bs=128k dd if="u-boot.img" of=${DISK} count=2 seek=1 bs=384k
where ${DISK} is the sd card device (just /dev/sdh , or /dev/mmcblk0 , not the partitions)
Hi,
okay, from what I understand, this COULD very well be an issue with the timings of the RAM chips.
What we know:
* 2GB RAM boards seem work fine without any freeze on 4.15 and 4.18 kernel * 4GB RAM boards seem to freeze after 5 - 10 minutes on 4.15 and 4.18 kernel * Memtester did work for Tony with the new, optimized 4GB RAM PCB (don't know which kernel, MLO, U-Boot, etc. has been used) * Memtester did work fine with Tonys patches for notaz on the OLD, unoptimized 4GB RAM PCB.
So, it doesn't seem to have much to do with the voltage drop (otherwise the old 4GB RAM PCB wouldn't have survived the memtester). But it can't simply be the new PCB either, as it worked for Tony with memtesting.
Could it be noise? (simply running memtest, maybe even without a screen attached should have less noise than a fully booted Pyra). (I'll test if it crashes as well when I only connect a PCB and also if it crashes if I idle around in U-Boot).
Could it be that the timings are on the edge and therefore it works for some 4GB RAM PCBs but not for others? That would also explain why the bootloader even sometimes fails to detect the RAM properly in my test unit.
In that case, I'd love to play around with some timings.
Could someone help me here? As far as I understand, the EMIF timings are setup in the MLO. I've got a Debian server, I've got an ArchLinux system at home.
Can anyone let me know where to grab the MLO sourcecode and how to compile it for the Pyra? I can provide access to the debian server if someone wants to set it up for me.
I guess the EMIF timings are properly documented in the MLO sourcecode, so I can compare them with the datasheet and change them?
On 9/20/18 1:55 PM, Michael Mrozek wrote:
- According to hns, even the uEVM freezes randomly with the latest kernel 4.18
I just checked, and my uevm has been running the 4.18.4-letux-pyra kernel for 8 days 19+ hours now. That's under pretty much no load however, and without a monitor connected.
-
aTc -
Grazvydas Ignotas -
H. Nikolaus Schaller -
Michael Mrozek -
Mr C Camacho -
Nino Bock -
Patrick Brünn -
Tony Lindgren