Am 20.09.2018 um 15:37 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Donnerstag, den 20.09.2018, 15:24 +0200 schrieb H. Nikolaus Schaller:
Hi,
neither u-boot nor kernel nor DTS modify any DDR3 (EMIF) timing. This all is defined by the SPL (MLO). But the exact timings might be different for each board.
Hm, so could it be that the timings in the MLO are a bit off for our RAM? That it didn't have anything to do with optimized or unoptimized CPU boards but mostly with timings being a bit on the edge?
Maybe even WRONG for the RAM we use?
I'm a bit confused by all the EMIF stuff. According to JEDEC Specs, these should be the same for all RAM types, but I remember that I found some differences when I compared both datasheets (for the old and new RAM).
So do we need to setup EMIF according to our RAM or not...?
It looks so. We did have two different RAM brands and they definitively did need different settings.
Or: Can we change some EMIF values to raise stability (maybe at the cost of tiny bit performance?)
As far as I remember, Tony did send some EMIF setup patch that is in notaz and letux uboot now, but noted that it might be reasonable to change some more parameters. Unfortunately I do not really understand more about them than you... So I have never played with them.
It was the older 4gb board (I have 2 now) and different dts, remaining
things should be the same I think (same SD). I intentionally tried to minimize changes from my working setup before leaving for the trip. Before reading this thread I just assumed my new board is faulty, but now it's clear more testing is needed. One obvious difference from the older board is the heatsink and some sort of new shielding (?), could it be messing something up like shorting or reflecting high freq signals?
On Thu, 20 Sep 2018, 21:56 Michael Mrozek, EvilDragon@openpandora.org wrote:
Am Donnerstag, den 20.09.2018, 21:49 +0900 schrieb Grazvydas Ignotas:
Hi,
that's bad news :( AFAIR, you were running memtester on the new 4GB RAM PCB I've sent you earlier as well for a few hours.
Were you using a different kernel / dtb here for the test? Because it's very weird that the PCB survives a memtest for both Tony and you but then has issues when simply idling around.
What's different here?
I'm traveling now and ED sent me a new board just before I left, and unfortunately it's unusable because of these random hangs. I wasn't able
to
correlate these hangs to anything too, it just hangs in like 1-10
minutes.
Most often it starts showing garbage suggesting RAM issues. I'm using
4.15
for working tiler, with adjusted dts for the new board. Nubs and sesors
are
not responding on that board too.
Pyra would have helped a lot during the travel, but unfortunately it's
just
dead weight I have to carry around.
notaz
On Thu, 20 Sep 2018, 20:55 Michael Mrozek, <EvilDragon@openpandora.org > wrote:
> I've got 22 Mainboard + CPU board combination (latest 4GB RAM > versions) > lying > around at home and would like to ensure that the hardware is working
fine
> - so I > can place the order for the mass production. > > However, I'm running into issues which I can't solve myself (as I am > no > developer and can't even do small changes to a kernel, etc.) > I'm good with Linux, but not with developing - so I need some > serious
help
> here. > > The unit I'm testing right now is one of latest batch. > > Our own PyraOS usually freezes the system right after 30 seconds to > 1 > minute, so > not really usable. > > The latest "stable" Letux (kernel 4.18.8) with LXDE as base OS
(installed
> using > the makesd-tool) does boot fine. On two tries, it freezed after > around
10
> - 12 > minutes sitting around idle. And about 2 - 3 minutes running the
high-load
> script (which ups the CPU to 1,5GHz with some loop). > > I will do more testings to see if it's always around the same time. > > I've made a small script that logs power supply, uptime and
temperature to
> see > if it has something to do with that. > > The result (incl. bootlog) is attached. > I went to bed while it was running, but you can see in the log that > it > freezes > after 12 minutes. Temperature is fine, voltage also seems to be fine
and
> when > using the high-load script, the temperature is higher after a few
seconds
> and it > still takes minutes to freeze - so we can assume that the freeze
doesn't
> have > anything to do with temperature CPU load. > > Interestingly, when I came back, the frozen unit was still on (and
showed
> the > frozen desktop screen), but pretty warm - a lot warmer compared to > it
just
> idling. > > I've added the new temperature and charging state at the end of
freeze.txt
> as > well. > > Some things we know: > > * Both askarus' and aTc's older Pyras (older Mainboard, 2GB RAM CPU
board
> without our optimizations) are stable with kernel 4.15. aTc > mentioned
he'd
> been > running it off USB for days whereas askarus' Pyra runs 10 hours off
battery
> without freezing and in normal usage (internet, LibreOffice, etc.) > Unfortunately, the released kernel 4.15 doesn't support the new
mainboards
> / CPU > boards for some reason, so I can't compare them. > > * According to hns, even the uEVM freezes randomly with the latest
kernel
> 4.18 > > * These freezes don't seem to be connected with temperature or CPU > load > (both > Askarus and aTcs Pyras don't have any heat sink applied, whereas the
4GB
> RAM > unit I've tested it on does). > > * Sometimes the freeze shows random graphic glitches on the display, > sometimes > the display simply freezes (memory errors?) > > * The freeze might NOT be a full freeze! When using hns' QuantumStep > OS > (which > uses the LEDs as heartbeat), the unit also seemed to have frozen > after
a
> couple > of minutes, as the LEDs stopped blinking. However, the LCD was still > updating > the graphics - VERY very slowly (1 character per 30 seconds on the > terminal), > but it was still working - even though it seemed frozen at first. > This doesn't seem to be the case on LXDE, but maybe it's even slower > because > there's so much going on. > > * On the unit I tested, sometimes u-Boot reported that it expected
4000000
> of > memory size but only found 4. This might be related to the unit - or
maybe
> some > timings are still wrong for the 4GB RAM so there's some issue here? > > * Running the uevm with little less than 5V increases the > instability.
As
> aTc > and askarus Pyra work fine with less than 5V, this shouldn't be a
problem
> - but > I thought I'd mention it here anyways. > > So I'd like to find out the reason for these random freezes are. > I've got various PCBs and cases, I can run tests - but as I can't
code, I
> can't > play around with kernel parameters or similar, so I'm pretty limited > finding out > what it could be. > > That's where I need your help! > Does anyone have any ideas what the issue could be and can provide > different > kernels, board files, etc. for me so I can test if that changes the > behaviour? > > It would also be good if someone could provide a minimal image with > all > non- > needed drivers disabled per default incl. instructions how I can > load
them
> afterwards one by one (for example, let me know the modules to load > and > what > they're used for). > > So basically, starting with minimal features to see when it fails. > Maybe further debug notices can be added so we see something on > serial
out
> just > before the crash...? > > As far as I know, Tony and notaz did a memory test run for hours, so
for
> some > reason, it didn't freeze on their boards. > > Who would be willing to help me providing me files and guiding me
through
> the > tests? > > Here are some ideas that could be the reason for the freezes: > > * SD-Card issues - we're running the Linux off SD Card. Not sure it
likes
> it > when for example power breaks down or the driver does something > wrong
with
> the > SD Card (though that should probably lead to a kernel panic). > > * Issues switching between various voltages / frequencies (are all > the > voltages > we're using in the dtb approved by TI?) > > * RAM timing issues > > * PALMAS power supply / timing issues. If PALMAS does something > wrong,
the
> system could easily freeze > > > The biggest issue is, that for the new PCBs both hardware and > software
have
> changed, so it's hard to compare it and rule out hardware / software > issues. > > Kernel 4.15 and old 2GB RAM hardware or uEVM works fine. > Kernel 4.18 doesn't work stable on old OR new hardware and neither > on
uEVM.
> > It could be hardware, software - or a mixture of both. > Well, software seems to be ONE reason at least, as 4.18 doesn't work > on > uEVM as > well. > But it means I can't properly test the new hardware right now - > which
is
> critical :) > > What are your ideas, how can we approach this? > > Thanks a lot in advance! > > -- > Greetings, > > Michael Mrozek > > ----------------------- > OpenPandora GmbH > CEO: Michael Mrozek > > Schäffbräustr. 11 > 85049 Ingolstadt > Germany > Tel.: +49 841 990 5548 > http://www.openpandora.de/ > HRB 4879, Amtsgericht Ingolstadt > ----------------------- > eMail: mrozek@openpandora.org > _______________________________________________ > Kernel mailing list > Kernel@pyra-handheld.com > http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel >
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org