Hi all, we know that the Pyra CPU boards (at least the 3 units we have running) make problems when we use OPP to allow for 1.5GHz. The kernel suddenly hangs without obvious and repeatable error messages.
At 1.0 GHz (or 1.5 GHz and disabling the second core) the OMAP5432 works.
To get some more insights I have done some tests.
* Board M4+C19 w/o display * Kernel: letux-4.7.0 * 500MHz + 750MHz OPP runs accoding to default DT * 1.0GHz OPP in DT modified to check what happens * temperature driven by /root/high-load (prints 3 temperature hwmon values every second)
A) 1.0GHz@1060000uV kernel boot: ok high-load: runs unlimited
Comment: This is what works since march 2016.
B) 1.1GHz@1060000uV kernel boot: ok high-load: reaches 97°C after 25 min cpufreq-info: 96%@1.1 GHz
Comment: I remember that temperature was ~92°C at 1.0 GHz so this drives the temperature up by 5K.
C1) 1.3GHz@1060000uV kernel boot: hangs during initial boot
Note: hang means the CPU isn't responding on serial interface and status LEDs are no longer blinking
Comment: obviously the voltage is too low for 1.3 GHz.
C3) 1.0GHz@1060000uV + 1.3GHz@1150000uV kernel boot: ok high-load: hangs after 15 seconds after reaching 63°C
repeated boot attempts: C3a) high-load: hangs after some seconds at 64°C
C3b) high-load: runs >15 min ramps up to 100-103°C suddenly jumps down to 82-95°C. reaches after ca. 10 sec again >100°C. As if some over temperature protection throttles the CPU clock PCB temperature: 80°C cpufreq-info: just 73%@1.3 GHz
C3c) high-load: hangs after 15 sec at 65°C
Comment: This means it runs not 100% reliable at this OPP and the effect seems to be temperature dependent. But if the OMAP runs it comes into a temperature limit which triggers some overtemp protection built into the kernel.
D) 1.0GHz@1060000uV + 1.5GHz@1150000uV kernel boot: hangs after 4.3-4.4 sek (3 times reproducible)
E1) 1.5GHz@1250000uV: kernel boot: hangs after 6.3-6.5 sek (3 times reproducible)
E2) 1.5GHz@1300000uV (close to upper limit according to "Data Manual Operating Condition Addendum Version 0.6"): kernel boot: hangs at 6.6 sek
F) test E1 + OMAP5_ERRATA_801819 enabled kernel boot: hangs again 6.4 sek (in "Synthesizing the initial hotplug events...")
Comment: it was not possible to boot in dual core 1.5 GHz mode. Very strange and unexpected is that the kernel hangs repeatedly at 6.3-6.6 seconds as if there is something in the code which increases the risk of a hang (deadlock).
So it is either a kernel software issue (something critical is running faster at 1.5 Ghz resulting in a deadlock). Or the voltage is still too low. But I did not dare to increase it further since it may destroy the valuable CPU board...
G) [Kernel] omap5 mpu bridge dividers Matthijs recently reported a potential issue here with the above subject line.
A simple test would be to boot at 1.5Ghz and then run
omapconf write 0x4A004320 0x06000001
But I can't even boot at 1.5Ghz so I have no chance to test.
Summary / Discussion: * it looks as if 1 GHz (or single core 1.5 GHz) works without problems * for 1.3 GHz we have to increase CPU voltage or the kernel hangs * at 1.5 GHz I wasn't able to boot even with increasing CPU voltage
The data sheets hint at using AVS and ABB.
"4.3.1 AVS and ABB Requirements Adaptive Voltage Scaling (AVS) and Adaptive Body Biasing (ABB) are required on most of the VDD_* domains as defined in Table 4-7"
Table 4-7 indirectly defines all operation points >1.0 GHz as required.
" • The AVS Voltages are device-dependent, voltage domain-dependent, and OPP-dependent. They must be read from the CONTROL_STD_FUSE_OPP_VDD Registers in the Control Module Section of the TRM."
From this I read that every sigle OMAP chip is slightly different and TI measures these differences during production.
This should be done by the AVS drivers.
We did not have CONFIG_POWER_AVS_OMAP enabled but only CONFIG_POWER_AVS.
But although I changed that and did some additional tests, it has no influence. And the AVS seems to be incomplete and non-operational anyways:
[ 4.977605] sr_init: No PMIC hook to init smartreflex [ 4.982922] driver_register 'smartreflex' [ 4.987747] sr_init: platform driver register failed for SR
... and the kernel hangs again at 6.43 sec. As if there is a watchdog timer in the OMAP that is only running in 1.5GHz mode...
So I think a hardware issue is quite unlikely, especially as the 1.5 GHz setup hangs always at the same 6.3-6.6 seconds after Linux Start.
And before I waste more and more weeks on looking for really difficult to grab hardware issues I would like to hear kernel-specialist's opinions first.
BR and thanks, Nikolaus
On Fri, Jul 29, 2016 at 5:00 PM, H. Nikolaus Schaller hns@goldelico.com wrote:
G) [Kernel] omap5 mpu bridge dividers Matthijs recently reported a potential issue here with the above subject line.
A simple test would be to boot at 1.5Ghz and then run
omapconf write 0x4A004320 0x06000001
But I can't even boot at 1.5Ghz so I have no chance to test.
It used to be possible to select default cpufreq governor in .config or maybe kernel arguments, so perhaps you could try booting with powersave governor, change the divider and then switch to the performance governor.
Gražvydas
Am 29.07.2016 um 19:45 schrieb Grazvydas Ignotas notasas@gmail.com:
On Fri, Jul 29, 2016 at 5:00 PM, H. Nikolaus Schaller hns@goldelico.com wrote:
G) [Kernel] omap5 mpu bridge dividers Matthijs recently reported a potential issue here with the above subject line.
A simple test would be to boot at 1.5Ghz and then run
omapconf write 0x4A004320 0x06000001
But I can't even boot at 1.5Ghz so I have no chance to test.
It used to be possible to select default cpufreq governor in .config or maybe kernel arguments, so perhaps you could try booting with powersave governor, change the divider and then switch to the performance governor.
good idea!
Was a little more difficult. I had to
1. change defconfig # CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set -# CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set +CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE=y # CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set -CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y +# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
2. echo "ENABLE=false" >$SD/etc/default/cpufrequtils or Debian would try to enable the ondemand govenor at the end of the boot process.
3. After adding two OPP for 1.25GHz and 1.5GHz to DT I could run additional tests.
cpufreq-set -f 1Ghz runs fine. No problems. cpufreq-set -f 1.25GHz might hang cpufreq-set -f 1.5GHz hangs almost immediately within a second or so. Even as the CPU is almost idle (waiting for shell and blinking LEDs)
4. Clock divider devmem2 0x4A004320 w 0x06000001 /dev/mem opened. Memory mapped at address 0xb6fae000. Value at address 0x4A004320 (0xb6fae320): 0x1000001 Written 0x6000001; readback 0x6000001
has no obvious influence. Neither @500 MHz nor @1.5GHz.
So unfortunately we do not know more than before.
The only thing that remains really stable is that the board is instable @>1GHz.
We simply do not know the handle to turn for an improvement. And we all can imagine a lot of handles... The trick is to find the right one.
BR, Nikolaus
Am 30.07.2016 um 19:53 schrieb H. Nikolaus Schaller hns@goldelico.com:
Am 29.07.2016 um 19:45 schrieb Grazvydas Ignotas notasas@gmail.com:
On Fri, Jul 29, 2016 at 5:00 PM, H. Nikolaus Schaller hns@goldelico.com wrote:
G) [Kernel] omap5 mpu bridge dividers Matthijs recently reported a potential issue here with the above subject line.
A simple test would be to boot at 1.5Ghz and then run
omapconf write 0x4A004320 0x06000001
But I can't even boot at 1.5Ghz so I have no chance to test.
It used to be possible to select default cpufreq governor in .config or maybe kernel arguments, so perhaps you could try booting with powersave governor, change the divider and then switch to the performance governor.
good idea!
Was a little more difficult. I had to
- change defconfig
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set -# CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set +CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE=y # CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set -CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y +# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
- echo "ENABLE=false" >$SD/etc/default/cpufrequtils
or Debian would try to enable the ondemand govenor at the end of the boot process.
- After adding two OPP for 1.25GHz and 1.5GHz to DT I could run
additional tests.
cpufreq-set -f 1Ghz runs fine. No problems. cpufreq-set -f 1.25GHz might hang cpufreq-set -f 1.5GHz hangs almost immediately within a second or so. Even as the CPU is almost idle (waiting for shell and blinking LEDs)
- Clock divider
devmem2 0x4A004320 w 0x06000001 /dev/mem opened. Memory mapped at address 0xb6fae000. Value at address 0x4A004320 (0xb6fae320): 0x1000001 Written 0x6000001; readback 0x6000001
has no obvious influence. Neither @500 MHz nor @1.5GHz.
So unfortunately we do not know more than before.
The only thing that remains really stable is that the board is instable @>1GHz.
We simply do not know the handle to turn for an improvement. And we all can imagine a lot of handles... The trick is to find the right one.
Ah, there is room for another test for the hardware now!
The CPU board I am running these test on has additional wires which allow to measure the voltages VDD_MPU, VDD_CORE, VDD_DDR3, VDD_MM, 1V8, VSYS.
So I can better look what happens as soon as the cpufreq-set -f 1.5GHz command is started. For example if the voltages become more noisy or something significant changes. E.g. if they are really what the DT OPP defines.
Will do as soon as I find time to make this setup.
Anyways, please all kernel developers think about potential kernel issues (scheduling, SMP, locking, interrupts, I&D-caches) that might lead to such a behaviour. And potential tests (I can add printk etc. where needed).
One thing I also want to mention: the DDR3 speed remains the same (I think 532 MHz) in all these experiments. So an A15 running at 1.5GHz might have to wait longer for external RAM than I&D-cache which gives different execution patterns.
BR, Nikolaus
Am Sat, 30 Jul 2016 20:02:14 +0200 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:
Hi,
The CPU board I am running these test on has additional wires which allow to measure the voltages VDD_MPU, VDD_CORE, VDD_DDR3, VDD_MM, 1V8, VSYS.
This might be a good idea. Maybe the combination of Palmas and the charger / battery chip causes an issue here and the voltage isn't increased for some reason.
What you could also test would be setting 1GHz to run with a higher voltage and measure whether the voltage REALLY is increased. This way, the unit doesn't crash.
Another thing I'd like to remind here: 1,5GHz work fine (even with full load) with the EVM and the iGEPV5 board. So it probably is something that's different between those and our boards.
* Are there differences in the board files that could cause our CPU to misbehave?
* Is there a difference in our hardware setup that could lead to that? We got different RAM, a different quartz, different power setup (we don't have a simple AC like the devboards, but a battery / charger circuit)
Anyways, please all kernel developers think about potential kernel issues (scheduling, SMP, locking, interrupts, I&D-caches) that might lead to such a behaviour. And potential tests (I can add printk etc. where needed).
Hi,
Am 30.07.2016 um 21:14 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Sat, 30 Jul 2016 20:02:14 +0200 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:
Hi,
The CPU board I am running these test on has additional wires which allow to measure the voltages VDD_MPU, VDD_CORE, VDD_DDR3, VDD_MM, 1V8, VSYS.
This might be a good idea. Maybe the combination of Palmas and the charger / battery chip causes an issue here and the voltage isn't increased for some reason.
Very unlikely that the charger / battery has such an influence. How should it know that I increase the cpu-freq...
There are other factors (e.g. Speakers on/off, Display on/off) which also have a high influence on VSYS and currents flowing through connectors but turning them on or off makes no such difference as cpufreq.
The Palmas has several programmable voltage regulators between VSYS and all these voltages above. So they should block changes in VSYS.
If the voltages turn out that they are not increased, there would be a software bug.
What you could also test would be setting 1GHz to run with a higher voltage and measure whether the voltage REALLY is increased. This way, the unit doesn't crash.
Yes, we will see that.
Another thing I'd like to remind here: 1,5GHz work fine (even with full load) with the EVM and the iGEPV5 board.
I haven't tested the Letux kernel on the EVM for a while, but I should do that as well.
So it probably is something that's different between those and our boards.
- Are there differences in the board files that could cause
our CPU to misbehave?
Yes. The Pyra has a different DT file than EVM. And for the IGEP I don't even know what it has. There is omap5-board-common but we have a lot of extensions.
And there could be some problem in a driver for a device that does not exist on either EVM or IGEP.
In addition there are a lot of changes going on in this area, like:
https://lkml.org/lkml/2016/5/4/735
- Is there a difference in our hardware setup that could lead to that?
We got different RAM, a different quartz, different power setup (we don't have a simple AC like the devboards, but a battery / charger circuit)
RAM and power setup / battery / charger is something I would remove from the list of potential reasons. Because they are exactly the same at 500 MHz, 1 GHz or 1.5 GHz. There is nothing controlled differently.
All these external clocks run all at the same speed and voltage. I.e. neither RAM, nor I2C nor video signals run faster when changing the cpufreq.
The quartz also runs at the same frequency and voltage.
The only thing we had discussed a while ago was a potential temperature dependency. But I can also exclude that with the tests done today.
The SoC die temperature when running at 500 MHz is ~55°C. And when switching to 1.5 GHz it rises to 65°C in 2-3 seconds and then the CPU hangs. This does not change the quartz temperature.
Anyways, please all kernel developers think about potential kernel issues (scheduling, SMP, locking, interrupts, I&D-caches) that might lead to such a behaviour. And potential tests (I can add printk etc. where needed).
What puzzles me most is that the system hangs at low system load. As soon as cpufreq-set goes to >ca. 1 GHz.
In that situation the OMAP should be in idle 98% of the time and is just blinking some I2C LEDs and waiting that I type the next command over UART console. Well, DSS is also running in the background. But not much more.
BR, Nikolaus
Am Sat, 30 Jul 2016 21:54:13 +0200 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:
Hi,
The CPU board I am running these test on has additional wires which allow to measure the voltages VDD_MPU, VDD_CORE, VDD_DDR3, VDD_MM, 1V8, VSYS.
This might be a good idea. Maybe the combination of Palmas and the charger / battery chip causes an issue here and the voltage isn't increased for some reason.
Very unlikely that the charger / battery has such an influence. How should it know that I increase the cpu-freq...
It provides the power that the Palmas requests, does it not? Or does it always provide the same power and the Palmas changes it based on what you set it to?
There are other factors (e.g. Speakers on/off, Display on/off) which also have a high influence on VSYS and currents flowing through connectors but turning them on or off makes no such difference as cpufreq.
One thing just came to my mind: You mention that it runs fine with SINGLE core 1,5GHz. If that's the case, then the power has to be set correctly, otherwise, the single core would crash as well right away.
So the question is: Why does it run with 200% CPU on dualcore 1GHz but crashes as soon as you switch it to 1,5GHz. 1,5GHz without doing anything should need less power than 200% at 1GHz, so it can't probably be the overall power consumption.
Hm, well, there is one difference:
I=U/R... if you have a higher voltage, you need more Ampere as well, depending on the resistance the traces have. Maybe it can't handle to keep the voltage high enough for that reason, or peaks.
If the voltages turn out that they are not increased, there would be a software bug.
But then it wouldn't work single core 1,5GHz, so you can probably rule that out.
- Are there differences in the board files that could cause
our CPU to misbehave?
Yes. The Pyra has a different DT file than EVM. And for the IGEP I don't even know what it has. There is omap5-board-common but we have a lot of extensions.
I know they have different DT-Files, but the question is: Are the power / voltage / timing settings in them different?
And there could be some problem in a driver for a device that does not exist on either EVM or IGEP.
Well, in that case it should work when all drivers are disabled and the system is being run under a minimum condition.
- Is there a difference in our hardware setup that could lead to
that? We got different RAM, a diffrent quartz, different power setup (we don't have a simple AC like the devboards, but a battery / charger circuit)
RAM and power setup / battery / charger is something I would remove from the list of potential reasons. Because they are exactly the same at 500 MHz, 1 GHz or 1.5 GHz. There is nothing controlled differently.
Well, except if they cap the maximum voltage for some reason, so the PALMAS can't change to the requested voltage. This would explain why 200% CPU at 1GHz would still work, as not the power is the issue, but the voltage.
The SoC die temperature when running at 500 MHz is ~55°C. And when switching to 1.5 GHz it rises to 65°C in 2-3 seconds and then the CPU hangs. This does not change the quartz temperature.
10°C more in 2-3 seconds? That sounds strange - unless you are running performance as governor, setting the CPU speed to 1,5GHz shouldn't really do much as the CPU would still be idling... so why does the temperature rise that fast?
Anyways, please all kernel developers think about potential kernel issues (scheduling, SMP, locking, interrupts, I&D-caches) that might lead to such a behaviour. And potential tests (I can add printk etc. where needed).
What puzzles me most is that the system hangs at low system load. As soon as cpufreq-set goes to >ca. 1 GHz. In that situation the OMAP should be in idle 98% of the time and is just blinking some I2C LEDs and waiting that I type the next command over UART console. Well, DSS is also running in the background. But not much more.
Yes, but that sudden increase of temperature does NOT sound like idling.
Am 31.07.2016 um 00:57 schrieb Michael Mrozek EvilDragon@openpandora.org:
Am Sat, 30 Jul 2016 21:54:13 +0200 hat "H. Nikolaus Schaller" hns@goldelico.com geschrieben:
Hi,
The CPU board I am running these test on has additional wires which allow to measure the voltages VDD_MPU, VDD_CORE, VDD_DDR3, VDD_MM, 1V8, VSYS.
This might be a good idea. Maybe the combination of Palmas and the charger / battery chip causes an issue here and the voltage isn't increased for some reason.
Very unlikely that the charger / battery has such an influence. How should it know that I increase the cpu-freq...
It provides the power that the Palmas requests, does it not? Or does it always provide the same power and the Palmas changes it based on what you set it to?
The latter.
Power flow is
USB -----+ bq24297 ---> VSYS=3.7V ---> Palmas - LDO ----> OMAP5 Battery --+
And all this voltage stepping is done by the Palmas LDOs (or switching regulators).
There are other factors (e.g. Speakers on/off, Display on/off) which also have a high influence on VSYS and currents flowing through connectors but turning them on or off makes no such difference as cpufreq.
One thing just came to my mind: You mention that it runs fine with SINGLE core 1,5GHz.
Yes, back in March that was a workaround to get the boards booted.
If that's the case, then the power has to be set correctly, otherwise, the single core would crash as well right away.
Yes. This is why I think we could be hit by some SMP synchronization issue at high speed.
So the question is: Why does it run with 200% CPU on dualcore 1GHz but crashes as soon as you switch it to 1,5GHz. 1,5GHz without doing anything should need less power than 200% at 1GHz, so it can't probably be the overall power consumption.
Yes. That is what makes me puzzled as well.
Hm, well, there is one difference:
I=U/R... if you have a higher voltage, you need more Ampere as well, depending on the resistance the traces have. Maybe it can't handle to keep the voltage high enough for that reason, or peaks.
There is a feedback line going from the end of the trace at the OMAP5 back to the Palmas so that it can compensate for voltage loss along the high-current traces.
But: this depends on how fast the regulator can react. So if there are fast current peaks it might regulate too slow.
If the voltages turn out that they are not increased, there would be a software bug.
But then it wouldn't work single core 1,5GHz, so you can probably rule that out.
Yes.
- Are there differences in the board files that could cause
our CPU to misbehave?
Yes. The Pyra has a different DT file than EVM. And for the IGEP I don't even know what it has. There is omap5-board-common but we have a lot of extensions.
I know they have different DT-Files, but the question is: Are the power / voltage / timing settings in them different?
Not that I am aware of. Well, there is one difference: we have added 500 MHz and 750 MHz OPPs.
And there could be some problem in a driver for a device that does not exist on either EVM or IGEP.
Well, in that case it should work when all drivers are disabled and the system is being run under a minimum condition.
That is difficult to do since it does not boot in all cases (for example the bq24297 driver).
- Is there a difference in our hardware setup that could lead to
that? We got different RAM, a diffrent quartz, different power setup (we don't have a simple AC like the devboards, but a battery / charger circuit)
RAM and power setup / battery / charger is something I would remove from the list of potential reasons. Because they are exactly the same at 500 MHz, 1 GHz or 1.5 GHz. There is nothing controlled differently.
Well, except if they cap the maximum voltage for some reason, so the PALMAS can't change to the requested voltage. This would explain why 200% CPU at 1GHz would still work, as not the power is the issue, but the voltage.
I am not sure but the Palmas should detect over-current situations and report an Interrupt. But it might be too late for the Linux kernel to print a message.
The SoC die temperature when running at 500 MHz is ~55°C. And when switching to 1.5 GHz it rises to 65°C in 2-3 seconds and then the CPU hangs. This does not change the quartz temperature.
10°C more in 2-3 seconds? That sounds strange
Well, my statement wasn't precise - it is not idle temperature but running high-load.
- unless you are running performance as governor,
setting the CPU speed to 1,5GHz shouldn't really do much as the CPU would still be idling... so why does the temperature rise that fast?
I should write a tool to measure the temperature in idle mode.
Anyways, please all kernel developers think about potential kernel issues (scheduling, SMP, locking, interrupts, I&D-caches) that might lead to such a behaviour. And potential tests (I can add printk etc. where needed).
What puzzles me most is that the system hangs at low system load. As soon as cpufreq-set goes to >ca. 1 GHz. In that situation the OMAP should be in idle 98% of the time and is just blinking some I2C LEDs and waiting that I type the next command over UART console. Well, DSS is also running in the background. But not much more.
Yes, but that sudden increase of temperature does NOT sound like idling.
Sorry for the misunderstanding created by slightly imprecise description.
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org
More test results / observations:
A) Pyra: MIPI/DSI is not working if govenor "powersave" and 500 MHz is selected at boot (and OMAP5 running @500MHz only) I get "DSI protocol violations" which means the CPU is too slow to process something. To me this indicates that something is CPU clock frequency dependent which should not be the case since DSI should be asynchronous, or not?
B) Pyra: LEDs seem to be blinking slightly irregular @500 MHz only boot (may be harmless) Sometimes the serial console appears to show delays (looks as if some IRQ blocking process runs a little slow?)
C) 5432EVM (letux-4.7.0 w/o Pyra Mainboard connected) I have modified the DT so that it also has 500 / 750 / 1250 MHz OPPs like for the Pyra. Termperature measurements (with cooling fingers installed)
idle@500MHz => ca. 46°C high-load@500MHz => ca. 52°C
idle@1000MHz => ca. 50°C high-load@1000MHz => ca. 73°C
idle@1500MHz => ca. 55°C high-load@1500MHz => ca. 99°C cools down to 73° within 5 seconds...
But in one attempt @1.5GHz my EVM did suddenly hang! Unfortunately not reproducible.
D) 5432EVM with Pyra Mainboard connected (booted at 500 MHz)
* bq2429x_charger 1-006b: bq24296_battery_probe(): Failed in reading register 0x0a I think it is normal that it needs either battery or USB power to respond on I2C * ssd2858 was initialized w/o errors - but does not show anything * re-initialization of ssd fails with DSI protocol errors (have not analysed futher - may be a setup problem of the old prototype hardware)
* kernel did hang once @1.5Ghz after typing "reboot" and did not shut down. * did hang another time @1.5GHz with no obvious reason * kernel did hang once immerdiately after typing cpufreq-set -f 1.5GHz over ssh/ethernet and ./high-load
hang means: * heartbeat LEDs stop blinking (usually they blink until kernel prints [info] Will now halt. [ 293.933938] dsi: mipi_debug_disable() [ 293.969283] reboot: Power down * Palmas does not react to power-on (because it is still on) * EVM reboots on Reset
So on my EVM there is also a problem with 1.5GHz although not as strong as on the Pyra CPU board. For better analysis someone else should be able to reproduce and then apply JTAG to find out what the OMAP is doing (or not doing) in that state.
E) Pyra CPU Temperature measurements (same as test C)
dle@500MHz => ca. 42°C high-load@500MHz => ca. 54°C
idle@1000MHz => ca. 62°C high-load@1000MHz => ca. 90°C
idle@1500MHz heartbeat stops almost immediately after switching to 1.5 GHz
BTW: display also doesn't work (maybe it is related to 500 MHz boot) any more in the current setup
I booted again and was able to make this log
root@letux:~# cpufreq-set -f 1.5ghz root@letux:~# ls b_host.sh femtocom.c mic-test tam batt findhwmon mipi-test temperatures bl findiio modprobe-test test_omapfb_vsync.c blanviewd.c findregulator mqtmoko tvout boe-w677l findthermal palmas-dump twl boe-w677l.prog flash-nand panelselect useful bootstrap fm-demo pin vibra.py bt-scan gps-demo pppd video-demo camera-demo gps-on profile volumed capture-demo hello.c sd-maximise wlan-on capture.c high-load si4721.c wlan-scan charger hw-test somefile.mp3 wwan config.tgz kbl somefile.wav wwan-off dcs ledtest somefile4.mp3 wwan-on debugdsi lsHSO sound-demo wwan-status dial makerootfs ssd2858 wwan.conf fbpng.c measure-power success-s90451 x fbtest measure-suspend suspend-test femtocom mic-present sys-profile root@letux:~# ls -l
Then everything hangs. Heartbeat stops blinking.
Interestingly, I have *never* seen any console message when it hangs!
This raises another question: what would happen if one core runs wild? Does the other core continue in SMP mode? Would it be able to print some messages? Is there some special CONFIG for kernel debugging of SMP or scheduling?
F) Pyra: Voltage measurements before and after cpufreq-set -f 1.5GHz
Measured by Oscilloscope (GND reference on Mainboard!) after boot cpufreq-set cpufreq-set 1GHz 1.5GHz VDD_MPU: 0.96V 1.16V 1.4V - on one case Palmas did shut down so I could not measure VDD_DDR3: 1.32V 1.32V 1.32V VDD_CORE: 1.12V 1.16V 1.16V
Measurement by external Voltmeter (reference to CPU GND) after boot cpufreq-set cpufreq-set hangs 1GHz 1.5GHz @1.5GHz VDD_MPU: 0.843V 1.053V 1.241V 1.247V VDD_CORE: 1.035V 1.034V 1.036V 1.036V VDD_DDR3: 1.209V 1.210V 1.201V 1.203V VDD_MM: 1.041V 1.041V 1.041V 10.40V 1V8: 1.788V 1.789V 1.799V 1.790V VSYS: 3.624V 3.622V 3.627V 3.562V
operating-points = < /* kHz uV */ 500000 850000 750000 950000 1000000 1060000 1250000 1150000 1500000 1250000 >;
So the VDD_MPU is very precise! Just 7mV lower than defined by software.
other observations: * no visible difference when high-load is started @500MHz * it appears as if noise increases on VDD_MPU if high-load is started @1GHz * it appears as if noise increases even more on VDD_MPU if high-load is started @1.5GHz - until activities break down
Interesting is that sometimes the Palmas shuts down after cpufreq-set to 1.5 GHz instead of making the CPU just hang. But I have seen it first time after connecting the Oscilloscope probes. This would mean that the Palmas detects some regulator overcurrent or something else going wrong.
Sometimes the CPU immediately hangs after switching to 1.5GHz. But one time it continued to blink the LEDs so that I could measure all the voltages. After starting high-load it stopped immediately.
And it might be that we have too much noise on VDD_MPU @1.5GHz. This is now a test scenario that I can compare to the EVM as soon as I find time for doing that.
This was a good hint to boot in "powersave" mode and manually switch to 1.5GHz. Before this, I wasn't able to do such measurements. And without measurements it is not possible to judge if something is ok or not ok - or if some change is an improvement.
Anyways these results now need some interpretation. Especially why VSSYS goes down from 3.6V to 3.5V! And the other voltages remain the same. This either means that the Palmas draws even more current from VSYS - or the bq24297 has reset itself to the default 3.5V regulation mode. Or something else?
Strange. Weird. Odd. Puzzling.
But we come closer to an understanding.
BR, Nikolaus
On Mon, Aug 1, 2016 at 12:37 AM, H. Nikolaus Schaller hns@goldelico.com wrote:
B) Pyra: LEDs seem to be blinking slightly irregular @500 MHz only boot (may be harmless) Sometimes the serial console appears to show delays (looks as if some IRQ blocking process runs a little slow?)
C) 5432EVM (letux-4.7.0 w/o Pyra Mainboard connected) I have modified the DT so that it also has 500 / 750 / 1250 MHz OPPs like for the Pyra. Termperature measurements (with cooling fingers installed)
500 was removed by TI, I guess there was some reason for that. 750 and 1250 were fabricated by you, not from some TI data, right? For uEVM stability testing I'd remove the "unofficial" frequencies, there could be some problems we don't know about, and uEVM has been always rock stable with the defaults (1 and 1.5 GHz only) for me.
And it might be that we have too much noise on VDD_MPU @1.5GHz. This is now a test scenario that I can compare to the EVM as soon as I find time for doing that.
Looking forward to that, I hope it will give some hints.
Anyways these results now need some interpretation. Especially why VSSYS goes down from 3.6V to 3.5V! And the other voltages remain the same. This either means that the Palmas draws even more current from VSYS - or the bq24297 has reset itself to the default 3.5V regulation mode. Or something else?
But 3.5V is still in spec, right? Maybe it would it be possible to mod the board to feed VSYS from some strong lab supply?
Gražvydas
Hi,
Am 02.08.2016 um 01:42 schrieb Grazvydas Ignotas notasas@gmail.com:
On Mon, Aug 1, 2016 at 12:37 AM, H. Nikolaus Schaller hns@goldelico.com wrote:
B) Pyra: LEDs seem to be blinking slightly irregular @500 MHz only boot (may be harmless) Sometimes the serial console appears to show delays (looks as if some IRQ blocking process runs a little slow?)
C) 5432EVM (letux-4.7.0 w/o Pyra Mainboard connected) I have modified the DT so that it also has 500 / 750 / 1250 MHz OPPs like for the Pyra. Termperature measurements (with cooling fingers installed)
500 was removed by TI, I guess there was some reason for that. 750 and 1250 were fabricated by you, not from some TI data, right?
Yes.
Theoretically any intermediate point should work...
For uEVM stability testing I'd remove the "unofficial" frequencies, there could be some problems we don't know about, and uEVM has been always rock stable with the defaults (1 and 1.5 GHz only) for me.
It had been stable until some months ago. At least in my anecdotical memory.
But shouldn't these points be completely ignored if we use the powersave / userspace govenor?
I see 0% useage by cpufreq-info for all points except the one that was booted.
And for the 1GHz boot I had removed the 500 & 750 entries.
And it might be that we have too much noise on VDD_MPU @1.5GHz. This is now a test scenario that I can compare to the EVM as soon as I find time for doing that.
Looking forward to that, I hope it will give some hints.
Anyways these results now need some interpretation. Especially why VSSYS goes down from 3.6V to 3.5V! And the other voltages remain the same. This either means that the Palmas draws even more current from VSYS - or the bq24297 has reset itself to the default 3.5V regulation mode. Or something else?
But 3.5V is still in spec, right?
Yes: 3.2 - 4.2V
The change itself is more interesting. Because it is a regulated voltage and the regulator is inside the bq24297. It can be programmed in 100mV steps.
Maybe it would it be possible to mod the board to feed VSYS from some strong lab supply?
That is on my to-do list... Needs some careful setup since I must reduce the risk to damage anything to 0%.
BR, Nikolaus
I'm missing the initial part of the thread and haven't read it all in detail yet, but I can offer my first thoughts:
I recently posted that I discovered that the Cortex-A15 subsystem's async bridges to the L3 and ABE interconnects are being overclocked (according to the datasheet) when the cpu is run at 1.5 GHz. The OS is supposed to reduce their clock dividers prior to switching to 1.5 GHz clock, but this did not seem to be happening.
If you check CM_MPU_MPU_CLKCTRL (0x4a004320), bits 25 and 26 are zero by default, which means L3bridge = cpu/4 and ABEbridge = cpu/8. They should be set to 1 before going to 1.5 GHz according to the datasheet, which makes the dividers /8 and /16 respectively. Unfortunately though this means that at OPP_HIGH they are clocked 25% slower than at OPP_NOM, and (based on testing) the bridge clock speed actually seems to be the bottleneck for L3 access. Yuck.
No idea why you'd have problems with it though when the uEVM seems to be fine with the L3 bridge apparently being overclocked by 50% (!!!).
This theory does have the benefit of being dependent specifically on cpu clock frequency and not on how heavily the cpu is actually being exercised. For everything else I can think of the only explanation would be "it just happens to be triggered by the way you exercise the hardware" which seems very weak. E.g. the cortex-A15 does have a few "nice" errata resulting in core deadlock. Of course it doesn't hurt to try enabling CONFIG_OMAP5_ERRATA_801819 if it isn't already (see https://patchwork.kernel.org/patch/6960921/ ).
Have you been able to check using JTAG in what sort of state the core(s) or SoC in general are? Even if the ARM subsystem is totally locked up, if you can connect to DAP then you can inspect things like PRCM registers and use I2C to check the configuration of the PMIC. (if you need it, I have some javascript helper code for CCS debugserver to perform I2C requests)
Matthijs
Hi Matthijs,
Am 02.08.2016 um 02:55 schrieb Matthijs van Duin matthijsvanduin@gmail.com:
I'm missing the initial part of the thread and haven't read it all in detail yet, but I can offer my first thoughts:
I recently posted that I discovered that the Cortex-A15 subsystem's async bridges to the L3 and ABE interconnects are being overclocked (according to the datasheet) when the cpu is run at 1.5 GHz. The OS is supposed to reduce their clock dividers prior to switching to 1.5 GHz clock, but this did not seem to be happening.
If you check CM_MPU_MPU_CLKCTRL (0x4a004320), bits 25 and 26 are zero by default, which means L3bridge = cpu/4 and ABEbridge = cpu/8. They should be set to 1 before going to 1.5 GHz according to the datasheet, which makes the dividers /8 and /16 respectively. Unfortunately though this means that at OPP_HIGH they are clocked 25% slower than at OPP_NOM, and (based on testing) the bridge clock speed actually seems to be the bottleneck for L3 access. Yuck.
So they should be modified dynamically when switching CPU frequencies?
No idea why you'd have problems with it though when the uEVM seems to be fine with the L3 bridge apparently being overclocked by 50% (!!!).
This theory does have the benefit of being dependent specifically on cpu clock frequency and not on how heavily the cpu is actually being exercised. For everything else I can think of the only explanation would be "it just happens to be triggered by the way you exercise the hardware" which seems very weak. E.g. the cortex-A15 does have a few "nice" errata resulting in core deadlock. Of course it doesn't hurt to try enabling CONFIG_OMAP5_ERRATA_801819 if it isn't already (see https://patchwork.kernel.org/patch/6960921/ ).
I had tried both (enable erratum and the CM_MPU_MPU_CLKCTRL change) but did not see an obvious difference.
Which does not prove that we don't need them anyways...
Have you been able to check using JTAG in what sort of state the core(s) or SoC in general are? Even if the ARM subsystem is totally locked up, if you can connect to DAP then you can inspect things like PRCM registers and use I2C to check the configuration of the PMIC. (if you need it, I have some javascript helper code for CCS debugserver to perform I2C requests)
Yes, that would be really a good tool to find out what is going on. And make some post-mortem register dumps.
Unfortunately I neither have JTAG equipment nor do we have access to the JTAG interface on the Pyra CPU board. Since neither is needed in normal situations. For a big pile of money we could have built some Pyra CPU boards where these signals are made available on external test points.
BR, Nikolaus
On 2 August 2016 at 08:45, H. Nikolaus Schaller hns@goldelico.com wrote:
So they should be modified dynamically when switching CPU frequencies?
Yes, and the datasheet seems quite emphatic about it. See section 4.2.1 of the omap5 TRM and the large note at the end of section 4.3.2 (Operating Performance Points -> MPU) of the omap5432 datasheet.
(Random side note: the same restriction apparently applies to AM57xx/DRA7xx based on section 4.2.1 of its TRM, although it's not waved in your face as much as on the omap5.)
Unfortunately I neither have JTAG equipment nor do we have access to the JTAG interface on the Pyra CPU board. Since neither is needed in normal situations. For a big pile of money we could have built some Pyra CPU boards where these signals are made available on external test points.
Ouch. You can get a perfectly adequate JTAG debugger (e.g. XDS100v2) for less than $100, but not having JTAG pinned out is a problem.
I read in the TRM that JTAG can alternatively be muxed to the SD card slot pins (intended for space-constrained applications like smartphones). I have no idea however whether that needs any special config other than setting the pinmux.
Matthijs
Hi,
Am 02.08.2016 um 09:52 schrieb Matthijs van Duin matthijsvanduin@gmail.com:
On 2 August 2016 at 08:45, H. Nikolaus Schaller hns@goldelico.com wrote:
So they should be modified dynamically when switching CPU frequencies?
Yes, and the datasheet seems quite emphatic about it. See section 4.2.1 of the omap5 TRM and the large note at the end of section 4.3.2 (Operating Performance Points -> MPU) of the omap5432 datasheet.
(Random side note: the same restriction apparently applies to AM57xx/DRA7xx based on section 4.2.1 of its TRM, although it's not waved in your face as much as on the omap5.)
Ok, so it is something that should be more fundamentally fixed in the kernel.
Unfortunately I neither have JTAG equipment nor do we have access to the JTAG interface on the Pyra CPU board. Since neither is needed in normal situations. For a big pile of money we could have built some Pyra CPU boards where these signals are made available on external test points.
Ouch. You can get a perfectly adequate JTAG debugger (e.g. XDS100v2) for less than $100, but not having JTAG pinned out is a problem.
Well, having it pinned out is a space/pcb layout problem...
$100 is not the problem and I have an Openmoko Debug board (which is also a JTAG debugger).
The problem is the time to set it up (which software to installation) and learn how to use it appropriately. Like it takes 10 min to buy a pan at some super market... But learning to cook usually needs more time.
I read in the TRM that JTAG can alternatively be muxed to the SD card slot pins (intended for space-constrained applications like smartphones).
Ah yes, I remember. Section 30.2.4 Narrow Interface for Debug and Trace (NIDnT) has some description - but I have not understood how to enable it before boot or if that is done automatically.
So we sould need a NIDnT compatible JTAG probe.
I have no idea however whether that needs any special config other than setting the pinmux.
Interesting idea for someone who wants to volunteer to experiment with that. The setup will be a little tricky since we boot from this SD card but I guess it is doable.
On 2 August 2016 at 10:24, H. Nikolaus Schaller hns@goldelico.com wrote:
So we sould need a NIDnT compatible JTAG probe.
As far as I can tell it's just an unusual method of connection, but if you manage to patch up wires (or find an adapter) then any probe should work. (Provided it works with CCS)
The config part is hopefully just muxing the normal JTAG pins to safe mode and muxing the SD pins to their JTAG functions
Am 02.08.2016 um 10:49 schrieb Matthijs van Duin matthijsvanduin@gmail.com:
On 2 August 2016 at 10:24, H. Nikolaus Schaller hns@goldelico.com wrote:
So we sould need a NIDnT compatible JTAG probe.
As far as I can tell it's just an unusual method of connection, but if you manage
well, we need to find someone to try this in parallel to me. Or postpone until other (simpler) means to identify the issue show no success.
to patch up wires (or find an adapter) then any probe should work. (Provided it works with CCS)
The config part is hopefully just muxing the normal JTAG pins to safe mode and muxing the SD pins to their JTAG functions
Yes, that can likely be done by devmem2 writing pinmux registers before switching to1.5GHz.
On 2 August 2016 at 11:55, H. Nikolaus Schaller hns@goldelico.com wrote:
well, we need to find someone to try this in parallel to me. Or postpone until other (simpler) means to identify the issue show no success.
Well I wasn't planning on attempting to solder anything onto my uEVM even if I had the time, but I found a moment to at least make a simple setup for CCS debugserver that only attaches to DAP and makes no attempt to touch the Cortex-A15 subsystem (which might be deadlocked).
https://github.com/mvduin/omap5-dap
If your setup matched mine you should be able to start the ./dss commandline, wait a bit, then press enter to connect to the target. You may however need to tweak paths hardcoded in the dss script and/or adjust the target configuration for a different JTAG adapter.
It includes utility code for I2C and an untested example for dumping pmic registers. The i2c code was working on am335x so it should work on omap5 too. I just commented out the clock calculation stuff since it would need to be revised and linux will have initialized those registers anyway. The problem is that the code only works with irqs enabled because this causes the peripheral to change behaviour (I don't immediately recall the details), but obviously doing this makes it impossible to sneak behind linux' back.
If linux has crashed sufficiently well however then you *should* be able to call i2c0_init() and then e.g. pmic_read(0x100,7) would read the pmic's RTC registers.
It's all a bit unpolished, but maybe it is of use
Matthijs
Another thing I'd like to remind here: 1,5GHz work fine (even with full load) with the EVM and the iGEPV5 board.
I haven't tested the Letux kernel on the EVM for a while, but I should do that as well.
As far as I know, PyraOS is using your kernel, so anyone in IRC using the EVM is using that kernel without any issues. (and they compile on that and do x86-emulation at 1,5GHz, so they really make use of that CPU).
Yeah I Currently using the 4.70 letux kernel on the EVM.
On Jul 30, 2016 7:07 PM, "Michael Mrozek" EvilDragon@openpandora.org wrote:
Another thing I'd like to remind here: 1,5GHz work fine (even with full load) with the EVM and the iGEPV5 board.
I haven't tested the Letux kernel on the EVM for a while, but I should do that as well.
As far as I know, PyraOS is using your kernel, so anyone in IRC using the EVM is using that kernel without any issues. (and they compile on that and do x86-emulation at 1,5GHz, so they really make use of that CPU).
-- Mit freundlichen Grüßen,
Michael Mrozek
OpenPandora GmbH Geschäftsführer: Michael Mrozek
Schäffbräustr. 11 85049 Ingolstadt Deutschland Tel.: 0841 / 990 5548 http://www.openpandora.de/ HRB 4879, Amtsgericht Ingolstadt
eMail: mrozek@openpandora.org _______________________________________________ Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Hi Nikolaus,
On Fri, Jul 29, 2016 at 4:00 PM, H. Nikolaus Schaller hns@goldelico.com wrote:
Hi all, we know that the Pyra CPU boards (at least the 3 units we have running) make problems when we use OPP to allow for 1.5GHz. The kernel suddenly hangs without obvious and repeatable error messages.
What about posting this message to LKML and add some TI people? Just an idea. I have experience only with single core CPU and usually it is hanging due to the fact that cpu freq is high but voltage isn't properly set.
At 1.0 GHz (or 1.5 GHz and disabling the second core) the OMAP5432 works.
To get some more insights I have done some tests.
- Board M4+C19 w/o display
- Kernel: letux-4.7.0
- 500MHz + 750MHz OPP runs accoding to default DT
- 1.0GHz OPP in DT modified to check what happens
- temperature driven by /root/high-load (prints 3 temperature hwmon values
every second)
A) 1.0GHz@1060000uV kernel boot: ok high-load: runs unlimited
Comment: This is what works since march 2016.
B) 1.1GHz@1060000uV kernel boot: ok high-load: reaches 97°C after 25 min cpufreq-info: 96%@1.1 GHz
Comment: I remember that temperature was ~92°C at 1.0 GHz so this drives the temperature up by 5K.
C1) 1.3GHz@1060000uV kernel boot: hangs during initial boot
Note: hang means the CPU isn't responding on serial interface and status LEDs are no longer blinking
Comment: obviously the voltage is too low for 1.3 GHz.
C3) 1.0GHz@1060000uV + 1.3GHz@1150000uV kernel boot: ok high-load: hangs after 15 seconds after reaching 63°C
repeated boot attempts: C3a) high-load: hangs after some seconds at 64°C
C3b) high-load: runs >15 min ramps up to 100-103°C suddenly jumps down to 82-95°C. reaches after ca. 10 sec again >100°C. As if some over temperature protection throttles the CPU clock PCB temperature: 80°C cpufreq-info: just 73%@1.3 GHz
C3c) high-load: hangs after 15 sec at 65°C
Comment: This means it runs not 100% reliable at this OPP and the effect seems to be temperature dependent. But if the OMAP runs it comes into a temperature limit which triggers some overtemp protection built into the kernel.
D) 1.0GHz@1060000uV + 1.5GHz@1150000uV kernel boot: hangs after 4.3-4.4 sek (3 times reproducible)
E1) 1.5GHz@1250000uV: kernel boot: hangs after 6.3-6.5 sek (3 times reproducible)
E2) 1.5GHz@1300000uV (close to upper limit according to "Data Manual Operating Condition Addendum Version 0.6"): kernel boot: hangs at 6.6 sek
F) test E1 + OMAP5_ERRATA_801819 enabled kernel boot: hangs again 6.4 sek (in "Synthesizing the initial hotplug events...")
Comment: it was not possible to boot in dual core 1.5 GHz mode. Very strange and unexpected is that the kernel hangs repeatedly at 6.3-6.6 seconds as if there is something in the code which increases the risk of a hang (deadlock).
So it is either a kernel software issue (something critical is running faster at 1.5 Ghz resulting in a deadlock). Or the voltage is still too low. But I did not dare to increase it further since it may destroy the valuable CPU board...
G) [Kernel] omap5 mpu bridge dividers Matthijs recently reported a potential issue here with the above subject line.
A simple test would be to boot at 1.5Ghz and then run
omapconf write 0x4A004320 0x06000001
But I can't even boot at 1.5Ghz so I have no chance to test.
Summary / Discussion:
- it looks as if 1 GHz (or single core 1.5 GHz) works without problems
- for 1.3 GHz we have to increase CPU voltage or the kernel hangs
- at 1.5 GHz I wasn't able to boot even with increasing CPU voltage
The data sheets hint at using AVS and ABB.
"4.3.1 AVS and ABB Requirements Adaptive Voltage Scaling (AVS) and Adaptive Body Biasing (ABB) are required on most of the VDD_* domains as defined in Table 4-7"
Table 4-7 indirectly defines all operation points >1.0 GHz as required.
" • The AVS Voltages are device-dependent, voltage domain-dependent, and OPP-dependent. They must be read from the CONTROL_STD_FUSE_OPP_VDD Registers in the Control Module Section of the TRM."
From this I read that every sigle OMAP chip is slightly different and TI measures these differences during production.
This should be done by the AVS drivers.
We did not have CONFIG_POWER_AVS_OMAP enabled but only CONFIG_POWER_AVS.
But although I changed that and did some additional tests, it has no influence. And the AVS seems to be incomplete and non-operational anyways:
[ 4.977605] sr_init: No PMIC hook to init smartreflex [ 4.982922] driver_register 'smartreflex' [ 4.987747] sr_init: platform driver register failed for SR
... and the kernel hangs again at 6.43 sec. As if there is a watchdog timer in the OMAP that is only running in 1.5GHz mode...
So I think a hardware issue is quite unlikely, especially as the 1.5 GHz setup hangs always at the same 6.3-6.6 seconds after Linux Start.
And before I waste more and more weeks on looking for really difficult to grab hardware issues I would like to hear kernel-specialist's opinions first.
BR and thanks, Nikolaus
http://projects.goldelico.com/p/gta04-kernel/ Letux-kernel mailing list Letux-kernel@openphoenux.org http://lists.goldelico.com/mailman/listinfo.cgi/letux-kernel
BR,
marek
Hi Marek,
Am 30.07.2016 um 22:14 schrieb Belisko Marek marek.belisko@gmail.com:
Hi Nikolaus,
On Fri, Jul 29, 2016 at 4:00 PM, H. Nikolaus Schaller hns@goldelico.com wrote:
Hi all, we know that the Pyra CPU boards (at least the 3 units we have running) make problems when we use OPP to allow for 1.5GHz. The kernel suddenly hangs without obvious and repeatable error messages.
What about posting this message to LKML and add some TI people? Just an idea.
I think it is too early to address the broad audience of LKML. They do not know or have the hardware to test. AFAIK, some TI people are also reading the kernel@pyra-handheld.com list.
And I don't exactly know what to tell them besides that it does not work reliably >1Ghz. And we can not 100% exclude a hardware issue where the LKML can't help (except telling us to exclude that).
I have experience only with single core CPU and usually it is hanging due to the fact that cpu freq is high but voltage isn't properly set.
Michael also pointed out that it works on OMAP5 EVM and IGEP5. And they use the same voltage settings as we use. But that could already be the problem.
I did read the data sheets and TRM in a way that the values for voltage settings should be modified according to some PROM value. And there is no driver for that...
So we simply might be unlucky that the unmodified values do not work for us :(
At 1.0 GHz (or 1.5 GHz and disabling the second core) the OMAP5432 works.
To get some more insights I have done some tests.
- Board M4+C19 w/o display
- Kernel: letux-4.7.0
- 500MHz + 750MHz OPP runs accoding to default DT
- 1.0GHz OPP in DT modified to check what happens
- temperature driven by /root/high-load (prints 3 temperature hwmon values
every second)
A) 1.0GHz@1060000uV kernel boot: ok high-load: runs unlimited
Comment: This is what works since march 2016.
B) 1.1GHz@1060000uV kernel boot: ok high-load: reaches 97°C after 25 min cpufreq-info: 96%@1.1 GHz
Comment: I remember that temperature was ~92°C at 1.0 GHz so this drives the temperature up by 5K.
C1) 1.3GHz@1060000uV kernel boot: hangs during initial boot
Note: hang means the CPU isn't responding on serial interface and status LEDs are no longer blinking
Comment: obviously the voltage is too low for 1.3 GHz.
C3) 1.0GHz@1060000uV + 1.3GHz@1150000uV kernel boot: ok high-load: hangs after 15 seconds after reaching 63°C
repeated boot attempts: C3a) high-load: hangs after some seconds at 64°C
C3b) high-load: runs >15 min ramps up to 100-103°C suddenly jumps down to 82-95°C. reaches after ca. 10 sec again >100°C. As if some over temperature protection throttles the CPU clock PCB temperature: 80°C cpufreq-info: just 73%@1.3 GHz
C3c) high-load: hangs after 15 sec at 65°C
Comment: This means it runs not 100% reliable at this OPP and the effect seems to be temperature dependent. But if the OMAP runs it comes into a temperature limit which triggers some overtemp protection built into the kernel.
D) 1.0GHz@1060000uV + 1.5GHz@1150000uV kernel boot: hangs after 4.3-4.4 sek (3 times reproducible)
E1) 1.5GHz@1250000uV: kernel boot: hangs after 6.3-6.5 sek (3 times reproducible)
E2) 1.5GHz@1300000uV (close to upper limit according to "Data Manual Operating Condition Addendum Version 0.6"): kernel boot: hangs at 6.6 sek
F) test E1 + OMAP5_ERRATA_801819 enabled kernel boot: hangs again 6.4 sek (in "Synthesizing the initial hotplug events...")
Comment: it was not possible to boot in dual core 1.5 GHz mode. Very strange and unexpected is that the kernel hangs repeatedly at 6.3-6.6 seconds as if there is something in the code which increases the risk of a hang (deadlock).
So it is either a kernel software issue (something critical is running faster at 1.5 Ghz resulting in a deadlock). Or the voltage is still too low. But I did not dare to increase it further since it may destroy the valuable CPU board...
G) [Kernel] omap5 mpu bridge dividers Matthijs recently reported a potential issue here with the above subject line.
A simple test would be to boot at 1.5Ghz and then run
omapconf write 0x4A004320 0x06000001
But I can't even boot at 1.5Ghz so I have no chance to test.
Summary / Discussion:
- it looks as if 1 GHz (or single core 1.5 GHz) works without problems
- for 1.3 GHz we have to increase CPU voltage or the kernel hangs
- at 1.5 GHz I wasn't able to boot even with increasing CPU voltage
The data sheets hint at using AVS and ABB.
"4.3.1 AVS and ABB Requirements Adaptive Voltage Scaling (AVS) and Adaptive Body Biasing (ABB) are required on most of the VDD_* domains as defined in Table 4-7"
Table 4-7 indirectly defines all operation points >1.0 GHz as required.
" • The AVS Voltages are device-dependent, voltage domain-dependent, and OPP-dependent. They must be read from the CONTROL_STD_FUSE_OPP_VDD Registers in the Control Module Section of the TRM."
From this I read that every sigle OMAP chip is slightly different and TI measures these differences during production.
This should be done by the AVS drivers.
We did not have CONFIG_POWER_AVS_OMAP enabled but only CONFIG_POWER_AVS.
But although I changed that and did some additional tests, it has no influence. And the AVS seems to be incomplete and non-operational anyways:
[ 4.977605] sr_init: No PMIC hook to init smartreflex [ 4.982922] driver_register 'smartreflex' [ 4.987747] sr_init: platform driver register failed for SR
... and the kernel hangs again at 6.43 sec. As if there is a watchdog timer in the OMAP that is only running in 1.5GHz mode...
So I think a hardware issue is quite unlikely, especially as the 1.5 GHz setup hangs always at the same 6.3-6.6 seconds after Linux Start.
And before I waste more and more weeks on looking for really difficult to grab hardware issues I would like to hear kernel-specialist's opinions first.
BR and thanks, Nikolaus
http://projects.goldelico.com/p/gta04-kernel/ Letux-kernel mailing list Letux-kernel@openphoenux.org http://lists.goldelico.com/mailman/listinfo.cgi/letux-kernel
BR,
marek
BR, Nikolaus
-
Belisko Marek -
Grazvydas Ignotas -
H. Nikolaus Schaller -
Matthijs van Duin -
Michael Mrozek -
Roy Gillotti