Am Donnerstag, den 20.09.2018, 21:49 +0900 schrieb Grazvydas Ignotas:

Hi,

that's bad news :(
AFAIR, you were running memtester on the new 4GB RAM PCB I've sent you earlier
as well for a few hours.

Were you using a different kernel / dtb here for the test?
Because it's very weird that the PCB survives a memtest for both Tony and you
but then has issues when simply idling around.

What's different here?

> I'm traveling now and ED sent me a new board just before I left, and
> unfortunately it's unusable because of these random hangs. I wasn't able to
> correlate these hangs to anything too, it just hangs in like 1-10 minutes.
> Most often it starts showing garbage suggesting RAM issues. I'm using 4.15
> for working tiler, with adjusted dts for the new board. Nubs and sesors are
> not responding on that board too.
>
> Pyra would have helped a lot during the travel, but unfortunately it's just
> dead weight I have to carry around.
>
> notaz
>
> On Thu, 20 Sep 2018, 20:55 Michael Mrozek, <EvilDragon@openpandora.org>
> wrote:
>
> > I've got 22 Mainboard + CPU board combination (latest 4GB RAM versions)
> > lying
> > around at home and would like to ensure that the hardware is working fine
> > - so I
> > can place the order for the mass production.
> >
> > However, I'm running into issues which I can't solve myself (as I am no
> > developer and can't even do small changes to a kernel, etc.)
> > I'm good with Linux, but not with developing - so I need some serious help
> > here.
> >
> > The unit I'm testing right now is one of latest batch.
> >
> > Our own PyraOS usually freezes the system right after 30 seconds to 1
> > minute, so
> > not really usable.
> >
> > The latest "stable" Letux (kernel 4.18.8) with LXDE as base OS (installed
> > using
> > the makesd-tool) does boot fine. On two tries, it freezed after around 10
> > - 12
> > minutes sitting around idle. And about 2 - 3 minutes running the high-load
> > script (which ups the CPU to 1,5GHz with some loop).
> >
> > I will do more testings to see if it's always around the same time.
> >
> > I've made a small script that logs power supply, uptime and temperature to
> > see
> > if it has something to do with that.
> >
> > The result (incl. bootlog) is attached.
> > I went to bed while it was running, but you can see in the log that it
> > freezes
> > after 12 minutes. Temperature is fine, voltage also seems to be fine and
> > when
> > using the high-load script, the temperature is higher after a few seconds
> > and it
> > still takes minutes to freeze - so we can assume that the freeze doesn't
> > have
> > anything to do with temperature CPU load.
> >
> > Interestingly, when I came back, the frozen unit was still on (and showed
> > the
> > frozen desktop screen), but pretty warm - a lot warmer compared to it just
> > idling.
> >
> > I've added the new temperature and charging state at the end of freeze.txt
> > as
> > well.
> >
> > Some things we know:
> >
> > * Both askarus' and aTc's older Pyras (older Mainboard, 2GB RAM CPU board
> > without our optimizations) are stable with kernel 4.15. aTc mentioned he'd
> > been
> > running it off USB for days whereas askarus' Pyra runs 10 hours off battery
> > without freezing and in normal usage (internet, LibreOffice, etc.)
> > Unfortunately, the released kernel 4.15 doesn't support the new mainboards
> > / CPU
> > boards for some reason, so I can't compare them.
> >
> > * According to hns, even the uEVM freezes randomly with the latest kernel
> > 4.18
> >
> > * These freezes don't seem to be connected with temperature or CPU load
> > (both
> > Askarus and aTcs Pyras don't have any heat sink applied, whereas the 4GB
> > RAM
> > unit I've tested it on does).
> >
> > * Sometimes the freeze shows random graphic glitches on the display,
> > sometimes
> > the display simply freezes (memory errors?)
> >
> > * The freeze might NOT be a full freeze! When using hns' QuantumStep OS
> > (which
> > uses the LEDs as heartbeat), the unit also seemed to have frozen after a
> > couple
> > of minutes, as the LEDs stopped blinking. However, the LCD was still
> > updating
> > the graphics - VERY very slowly (1 character per 30 seconds on the
> > terminal),
> > but it was still working - even though it seemed frozen at first.
> > This doesn't seem to be the case on LXDE, but maybe it's even slower
> > because
> > there's so much going on.
> >
> > * On the unit I tested, sometimes u-Boot reported that it expected 4000000
> > of
> > memory size but only found 4. This might be related to the unit - or maybe
> > some
> > timings are still wrong for the 4GB RAM so there's some issue here?
> >
> > * Running the uevm with little less than 5V increases the instability. As
> > aTc
> > and askarus Pyra work fine with less than 5V, this shouldn't be a problem
> > - but
> > I thought I'd mention it here anyways.
> >
> > So I'd like to find out the reason for these random freezes are.
> > I've got various PCBs and cases, I can run tests - but as I can't code, I
> > can't
> > play around with kernel parameters or similar, so I'm pretty limited
> > finding out
> > what it could be.
> >
> > That's where I need your help!
> > Does anyone have any ideas what the issue could be and can provide
> > different
> > kernels, board files, etc. for me so I can test if that changes the
> > behaviour?
> >
> > It would also be good if someone could provide a minimal image with all
> > non-
> > needed drivers disabled per default incl. instructions how I can load them
> > afterwards one by one (for example, let me know the modules to load and
> > what
> > they're used for).
> >
> > So basically, starting with minimal features to see when it fails.
> > Maybe further debug notices can be added so we see something on serial out
> > just
> > before the crash...?
> >
> > As far as I know, Tony and notaz did a memory test run for hours, so for
> > some
> > reason, it didn't freeze on their boards.
> >
> > Who would be willing to help me providing me files and guiding me through
> > the
> > tests?
> >
> > Here are some ideas that could be the reason for the freezes:
> >
> > * SD-Card issues - we're running the Linux off SD Card. Not sure it likes
> > it
> > when for example power breaks down or the driver does something wrong with
> > the
> > SD Card (though that should probably lead to a kernel panic).
> >
> > * Issues switching between various voltages / frequencies (are all the
> > voltages
> > we're using in the dtb approved by TI?)
> >
> > * RAM timing issues
> >
> > * PALMAS power supply / timing issues. If PALMAS does something wrong, the
> > system could easily freeze
> >
> >
> > The biggest issue is, that for the new PCBs both hardware and software have
> > changed, so it's hard to compare it and rule out hardware / software
> > issues.
> >
> > Kernel 4.15 and old 2GB RAM hardware or uEVM works fine.
> > Kernel 4.18 doesn't work stable on old OR new hardware and neither on uEVM.
> >
> > It could be hardware, software - or a mixture of both.
> > Well, software seems to be ONE reason at least, as 4.18 doesn't work on
> > uEVM as
> > well.
> > But it means I can't properly test the new hardware right now - which is
> > critical :)
> >
> > What are your ideas, how can we approach this?
> >
> > Thanks a lot in advance!
> >
> > --
> > Greetings,
> >
> > Michael Mrozek
> >
> > -----------------------
> > OpenPandora GmbH
> > CEO: Michael Mrozek
> >
> > Schäffbräustr. 11
> > 85049 Ingolstadt
> > Germany
> > Tel.: +49 841 990 5548
> > http://www.openpandora.de/
> > HRB 4879, Amtsgericht Ingolstadt
> > -----------------------
> > eMail: mrozek@openpandora.org
> > _______________________________________________
> > Kernel mailing list
> > Kernel@pyra-handheld.com
> > http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
> >
--
Mit freundlichen Grüßen,

Michael Mrozek

-----------------------
OpenPandora GmbH
Geschäftsführer: Michael Mrozek

Schäffbräustr. 11
85049 Ingolstadt
Deutschland
Tel.: 0841 / 990 5548
http://www.openpandora.de/
HRB 4879, Amtsgericht Ingolstadt
-----------------------
eMail: mrozek@openpandora.org