Hi Tony, I just recognized that I forgot to CC to the mailing lists...
For the readers: Letux-4.14-rc* boots well on GTA04 but doesn't on Pyra or OMAP5EVM. I just get Starting Kernel... Going back to Letux-4.13.* it boots on all machines.
Another problem which also might need bisection is that the Panel on the Pandora shows only the blue RGB channel in Letux-4.14-rc* while it works in Letux-4.13.*.
Am 08.10.2017 um 11:56 schrieb H. Nikolaus Schaller hns@goldelico.com:
Hi Tony,
Currently I am trying to bisect but it is quite difficult to debug our letux-tree since most bisect offers fail to compile (e.g. letux_defconfig missing).
I have finally found a way which looks promising. The key problem was so far that I only could compile the bad and good ends (Letux-4.13.0 and Letux-4.14-rc1) but never a bisected tree.
Now I take v4.13 as good and v4.14-rc1 as bad.
And I cherry pick some of the essential Letux patches on top of that. Plus https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h... (ignoring if it is already picked) Then I checkout the letux_defconfig from latest Letux.
It now seems as if I get the beast compiled and I can try to boot the OMAP5ECM with omap5-uevm.dtb
The good kernel boots while the bad one hangs after Starting kernel...
And the first bisect seems to boot as well. So the problem might be in the second half of 6000 patches.
So there are just ca. 13 bisect build & boot steps until I know more.
BR, Nikolaus
Hi Tony,
Am 09.10.2017 um 17:35 schrieb H. Nikolaus Schaller hns@goldelico.com:
Hi Tony, I just recognized that I forgot to CC to the mailing lists...
For the readers: Letux-4.14-rc* boots well on GTA04 but doesn't on Pyra or OMAP5EVM. I just get Starting Kernel... Going back to Letux-4.13.* it boots on all machines.
Another problem which also might need bisection is that the Panel on the Pandora shows only the blue RGB channel in Letux-4.14-rc* while it works in Letux-4.13.*.
So there are just ca. 13 bisect build & boot steps until I know more.
Well, I had to learn first that if I cherry-pick some patches on top of what git bisect good/bad proposes next, I have to remove them first before I do the next git bisect good/bad or git bisect tries to find out something about the cherry-picked patches...
Now after several days of work I have a result:
There are only 'skip'ped commits left to test. The first bad commit could be any of: d2b310b0234c20a656e2b9047b6f2a318ad39c35 fc23beb8a57723eecd04cd732e0722df72feaf70
Let's look into them:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i... https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
Both have indeed something to do with OMAP3/4/5 UARTs.
So this explains why there is no console message after they are applied, i.e. between v4.1 and v4.14-rc1.
But it does not explain: * why it works on GTA04 (OMAP3) * why the kernel doesn't start at all (no LEDs blinking) on omap5432uevm (or Pyra)
To document what kernel I have been running on the omap5432evm:
1) any bisect of kernel.org between v4.13 (good) and v4.14-rc1 (bad) 2) cherry pick 90c4ae4e2c1da9f1eaf846136861af43d4c1ff34 to fix a potential compile issue (may fail if already applied by bisect - or not needed if you have a better cross compiler) 3) cherry pick two patches from Letux for helping my local HOSTCC to compile sortextable and some selinux stuff: http://git.goldelico.com/?p=gta04-kernel.git;a=shortlog;h=refs/heads/work/le... They should not be necessary on different build hosts. 4) checkout http://git.goldelico.com/?p=gta04-kernel.git;a=blob;f=arch/arm/configs/letux... to get the letux defconfig which works for "good" on my setup (omap2plus_defconfig fails with my u-boot for all recent kernels) 5) make letux_defconfig
So with this information you should be able to confirm this problem on the omap5432evm.
Any ideas why it does break the EVM (and Pyra) that hard (no LEDs blinking)?
The commit description just talks about "there's no need to keep DEBUG_LL enabled", but not that it harms to have it enabled.
Note that letux_defconfig has CONFIG_DEBUG_LL=y by default for a long time and nobody did disable it so far.
I have finally tried with CONFIG_DEBUG_LL disabled and...
... I can boot letux-4.13-rc3
On both OMAP5432EVM and Pyra :)
So these offending patches should also enforce to disable CONFIG_DEBUG_LL on OMAP3/4/5 (all of them?). E.g. CONFIG_DEBUG_LL sould depend on !OMAP3 && !OMAP4 && !OMAP5 or so. Or the patches should explicitly depend on !CONFIG_DEBUG_LL. Or at least emit a #warning if CONFIG_DEBUG_LL is enabled.
Phew... That was quite challenging and time consuming to find out.
Anyways it was not a complete much waste of time, because I know two more bugs that need the same bisect approach: * Pandora display having blue channel only after going v4.13..v4.14-rc1 * Pandora WLAN stopped working between 4.6..4.7 - and even worse between v4.13..v4.14-rc1
BR, Nikolaus
Hi,
* H. Nikolaus Schaller hns@goldelico.com [171010 12:29]:
Now after several days of work I have a result:
There are only 'skip'ped commits left to test. The first bad commit could be any of: d2b310b0234c20a656e2b9047b6f2a318ad39c35 fc23beb8a57723eecd04cd732e0722df72feaf70
Let's look into them:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i... https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
Both have indeed something to do with OMAP3/4/5 UARTs.
So this explains why there is no console message after they are applied, i.e. between v4.1 and v4.14-rc1.
But it does not explain:
- why it works on GTA04 (OMAP3)
- why the kernel doesn't start at all (no LEDs blinking) on omap5432uevm (or Pyra)
Hmm OK that explains, sorry to hear about wasting time on this :(
The 8250 based DEBUG_LL uart will maintain the .config values for UART address etc. So just disable CONFIG_DEBUG_LL in all your configs and use the earlycon instead. DEBUG_LL is not portable across machines and should be only selected if nothing else helps.
$ grep "config DEBUG_UART_" arch/arm/Kconfig.debug config DEBUG_UART_PL01X config DEBUG_UART_8250 config DEBUG_UART_PHYS config DEBUG_UART_VIRT config DEBUG_UART_8250_SHIFT config DEBUG_UART_8250_WORD config DEBUG_UART_8250_PALMCHIP config DEBUG_UART_8250_FLOW_CONTROL
You probably enabled these for GTA04, and those addresses will hang early on anything else.
Regards,
Tony
Am 10.10.2017 um 21:43 schrieb Tony Lindgren tony@atomide.com:
Hi,
- H. Nikolaus Schaller hns@goldelico.com [171010 12:29]:
Now after several days of work I have a result:
There are only 'skip'ped commits left to test. The first bad commit could be any of: d2b310b0234c20a656e2b9047b6f2a318ad39c35 fc23beb8a57723eecd04cd732e0722df72feaf70
Let's look into them:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i... https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
Both have indeed something to do with OMAP3/4/5 UARTs.
So this explains why there is no console message after they are applied, i.e. between v4.1 and v4.14-rc1.
But it does not explain:
- why it works on GTA04 (OMAP3)
- why the kernel doesn't start at all (no LEDs blinking) on omap5432uevm (or Pyra)
Hmm OK that explains, sorry to hear about wasting time on this :(
The 8250 based DEBUG_LL uart will maintain the .config values for UART address etc. So just disable CONFIG_DEBUG_LL in all your configs and use the earlycon instead. DEBUG_LL is not portable across machines and should be only selected if nothing else helps.
$ grep "config DEBUG_UART_" arch/arm/Kconfig.debug config DEBUG_UART_PL01X config DEBUG_UART_8250
=y
config DEBUG_UART_PHYS
=0x49020000
config DEBUG_UART_VIRT
=0xfb020000
config DEBUG_UART_8250_SHIFT
=2
config DEBUG_UART_8250_WORD config DEBUG_UART_8250_PALMCHIP config DEBUG_UART_8250_FLOW_CONTROL
The others were not set. I don't remember when and why they were set...
You probably enabled these for GTA04, and those addresses will hang early on anything else.
Yes, that is well possible. And explains why OMAP3 did still work...
Anyways I think DEBUG_LL should be disabled or at least warned when compiling for OMAP5.
Still the best thing is that I have it working again on Pyra. So I can continue with some more hardware tests. We plan to replace the tca6424 (same as on EVM) with a pcal6524. It is already tested that the pcal can be operated as compatible="ti,tca6424" but the pcal has some better interrupt management which should be enabled in the driver and the DTS for the pyra should be updated as well. Now I can start to test such things (not with starting to maintain an older kernel...).
BR, Nikolaus
* H. Nikolaus Schaller hns@goldelico.com [171010 13:05]:
Anyways I think DEBUG_LL should be disabled or at least warned when compiling for OMAP5.
It should still work if enabled for omap5, but Kconfig will preserve the old value if set earlier for omap3.
Yeah I know it's confusing.. Best to not use DEBUG_LL.
Regards,
Tony
Am 10.10.2017 um 22:17 schrieb Tony Lindgren tony@atomide.com:
- H. Nikolaus Schaller hns@goldelico.com [171010 13:05]:
Anyways I think DEBUG_LL should be disabled or at least warned when compiling for OMAP5.
It should still work if enabled for omap5
well it didn't...
, but Kconfig will preserve the old value if set earlier for omap3.
Yeah I know it's confusing.. Best to not use DEBUG_LL.
The problem is: how should I (or someone else inheriting old .config) know without bisecting :) This is why I think some #warning / automatic disable would be nice.
BR, Nikolaus
* H. Nikolaus Schaller hns@goldelico.com [171010 23:42]:
Am 10.10.2017 um 22:17 schrieb Tony Lindgren tony@atomide.com:
- H. Nikolaus Schaller hns@goldelico.com [171010 13:05]:
Anyways I think DEBUG_LL should be disabled or at least warned when compiling for OMAP5.
It should still work if enabled for omap5
well it didn't...
, but Kconfig will preserve the old value if set earlier for omap3.
Yeah I know it's confusing.. Best to not use DEBUG_LL.
The problem is: how should I (or someone else inheriting old .config) know without bisecting :) This is why I think some #warning / automatic disable would be nice.
Yeah would be nice to fix it somehow. Those 8250 options are default values so Kconfig won't overwrite them. Maybe bring up the issue on LAKML?
Regards,
Tony