Hi, some additions below.
Am 20.09.2018 um 13:55 schrieb Michael Mrozek EvilDragon@openpandora.org:
I've got 22 Mainboard + CPU board combination (latest 4GB RAM versions) lying around at home and would like to ensure that the hardware is working fine - so I can place the order for the mass production.
However, I'm running into issues which I can't solve myself (as I am no developer and can't even do small changes to a kernel, etc.) I'm good with Linux, but not with developing - so I need some serious help here.
The unit I'm testing right now is one of latest batch.
Our own PyraOS usually freezes the system right after 30 seconds to 1 minute, so not really usable.
The latest "stable" Letux (kernel 4.18.8) with LXDE as base OS (installed using the makesd-tool) does boot fine. On two tries, it freezed after around 10 - 12 minutes sitting around idle. And about 2 - 3 minutes running the high-load script (which ups the CPU to 1,5GHz with some loop).
I will do more testings to see if it's always around the same time.
I've made a small script that logs power supply, uptime and temperature to see if it has something to do with that.
The result (incl. bootlog) is attached. I went to bed while it was running, but you can see in the log that it freezes after 12 minutes. Temperature is fine, voltage also seems to be fine and when using the high-load script, the temperature is higher after a few seconds and it still takes minutes to freeze - so we can assume that the freeze doesn't have anything to do with temperature CPU load.
Interestingly, when I came back, the frozen unit was still on (and showed the frozen desktop screen), but pretty warm - a lot warmer compared to it just idling.
I've added the new temperature and charging state at the end of freeze.txt as well.
Some things we know:
- Both askarus' and aTc's older Pyras (older Mainboard, 2GB RAM CPU board
without our optimizations) are stable with kernel 4.15. aTc mentioned he'd been running it off USB for days whereas askarus' Pyra runs 10 hours off battery without freezing and in normal usage (internet, LibreOffice, etc.) Unfortunately, the released kernel 4.15 doesn't support the new mainboards / CPU boards for some reason, so I can't compare them.
- According to hns, even the uEVM freezes randomly with the latest kernel 4.18
Not only 4.18. I also tried several archived system binaries back to 4.9 and both EVMs did freeze. What I did not do is to combine older u-boot, older kernel and older rootfs binaries.
It is very unclear to me what the factors are. At some days I start it in the morning and have to reboot 3 or 4 times. Then it freezes when accessing SD card or USB. Or just when not looking at it. After reset it runs fine for hours - to freeze again unexpectedly.
- These freezes don't seem to be connected with temperature or CPU load (both
Askarus and aTcs Pyras don't have any heat sink applied, whereas the 4GB RAM unit I've tested it on does).
- Sometimes the freeze shows random graphic glitches on the display, sometimes
the display simply freezes (memory errors?)
- The freeze might NOT be a full freeze! When using hns' QuantumStep OS (which
uses the LEDs as heartbeat), the unit also seemed to have frozen after a couple of minutes, as the LEDs stopped blinking. However, the LCD was still updating the graphics - VERY very slowly (1 character per 30 seconds on the terminal), but it was still working - even though it seemed frozen at first. This doesn't seem to be the case on LXDE, but maybe it's even slower because there's so much going on.
- On the unit I tested, sometimes u-Boot reported that it expected 4000000 of
memory size but only found 4. This might be related to the unit - or maybe some timings are still wrong for the 4GB RAM so there's some issue here?
- Running the uevm with little less than 5V increases the instability. As aTc
and askarus Pyra work fine with less than 5V, this shouldn't be a problem - but I thought I'd mention it here anyways.
So I'd like to find out the reason for these random freezes are. I've got various PCBs and cases, I can run tests - but as I can't code, I can't play around with kernel parameters or similar, so I'm pretty limited finding out what it could be.
That's where I need your help! Does anyone have any ideas what the issue could be and can provide different kernels, board files, etc. for me so I can test if that changes the behaviour?
It would also be good if someone could provide a minimal image with all non- needed drivers disabled per default incl. instructions how I can load them afterwards one by one (for example, let me know the modules to load and what they're used for).
So basically, starting with minimal features to see when it fails. Maybe further debug notices can be added so we see something on serial out just before the crash...?
As far as I know, Tony and notaz did a memory test run for hours, so for some reason, it didn't freeze on their boards.
Who would be willing to help me providing me files and guiding me through the tests?
Here are some ideas that could be the reason for the freezes:
- SD-Card issues - we're running the Linux off SD Card. Not sure it likes it
when for example power breaks down or the driver does something wrong with the SD Card (though that should probably lead to a kernel panic).
- Issues switching between various voltages / frequencies (are all the voltages
we're using in the dtb approved by TI?)
I have one kernel patch that restricts to 1GHz and official OPP voltage only, but it also freezes from time to time.
RAM timing issues
PALMAS power supply / timing issues. If PALMAS does something wrong, the
system could easily freeze
The biggest issue is, that for the new PCBs both hardware and software have changed, so it's hard to compare it and rule out hardware / software issues.
Kernel 4.15 and old 2GB RAM hardware or uEVM works fine. Kernel 4.18 doesn't work stable on old OR new hardware and neither on uEVM.
For me, any old/new kernel works stable on 2GB RAM but not any more on uEVM (it was rock solid up to ca. 6 months ago) or 4GB RAM.
With one exception: when I tried to analyse the "reboot" issue I had pwm_bl and the dmtimer modules disabled. So there was *no* backlight. And in that setup the 2GB Pyra did freeze several times just sitting at the login: promt.
Although that is anecdotical evidence only, these problems might be related. Tero has made a suggestion about the NRESWARM line. I have started analysis but not yet finished.
It could be hardware, software - or a mixture of both. Well, software seems to be ONE reason at least, as 4.18 doesn't work on uEVM as well. But it means I can't properly test the new hardware right now - which is critical :)
What are your ideas, how can we approach this?
Thanks a lot in advance!
-- Greetings,
Michael Mrozek
OpenPandora GmbH CEO: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Germany Tel.: +49 841 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org <freeze.txt>_______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel