The Wifi Problem Will Be Found To Be Caused By:

[Grench]

From the Miscellaneous Information Access Sub-System (MIASS):

EMI harmonics from the Bluetooth transmitter is causing cross talk between the two antennas. Effectively transmissions are being bounced from the second antenna back to/from the primary resulting in the transmitted and received signals having a ghost signature that effectively blurs the packets. The solution will be to move the bluetooth antenna to the front right corner of the bottom half of the case - more than 50% of the 2.4ghz wave length of the bluetooth signal. (>50% of 12cm = >6cm).

That is just about as plausible as anything else I've seen put out there - has anyone tried the WiFi with the bluetooth antenna removed? Not just disconnected, but removed?

Shot - in - dark.

 

[torpor]

* Word-/page- alignment issues.

...would cause an exception and be quickly seen.

If the pbuffers for RX are not set at a 16-byte aligned address, and you tell the chipset to pass data to an address that is not so aligned, you may not see an exception, actually, but may just not get a full buffer transferred ..

* Subtle compiler differences between what gcc for x86 (where the existing driver was developed and has received most of its testing) is expecting to have to output for the packet_buffer, and ARM?

The driver was developed for N900 by Nokia and has been shown to work on N900 and various ARM HTC devices like G1.

Yes, but how was the driver *compiled* for the N900 - which version of gcc, which switches, etc. Just because Nokia got it built and working for N900/Maemo does not mean that the same code compiles exactly the same for Pandora/OpenEmbedded ..

We have an N900 development group here at Metalab - I'll inspect the ways this driver is used next time I get a chance to talk to some of them (tomorrow evening, most likely, CET)

* Tried building the module with all optimizations off?

I see no reason to do that. Multiple compilers were tested.

Which versions, which optimizations? Please keep in mind that not all gcc ARM targets are the same degree of high/low quality. Its really important to standardize on a compiler and not swing around much, else you're really going to set up some pain for breakfast.

Optimizations on/off is a really, really easy test. I would try this, if I had the hardware in front of me. It *can* make a difference for driver code, immensely so ..

* Incorrect register usage. (If I had a Pandora, this is the first thing I'd check - all registers are being used properly, in the right order, etc.)

Which registers do you have in mind?
MMC/SD on OMAP - same code drives both SD slots, also firmware upload works fine, it's not SDIO like I stated multiple times.

I don't mean the SDIO control stuff, I'm talking about the wlxxx (you know what: I still don't know the *exact* specifics of which chip is being used in Pandora, sorry about this generality) controller registers for setting up the link, opening the radio, turning on/off CRC on packets, and so on ..

registers on wifi module - more likely, the module is basically undocumented black box and no one knows how to program it (except TI; this is "normal" for wifi modules though), but the driver works on other similar devices.

Ah, this black-box aspect is a bit of a lame duck though, because obviously N900 know enough about the wifi module to get it working, and that knowledge is represented in the latest linux-2.6.34-rc2 codebase for wl12xx .. BTW, did you notice the rather extensive set of updates to this codebase between the linux-2.6.33 and linux-2.6.34-rc2 releases? There is some work going on there .. what version of these drivers are you working with?

registers on ARM - you think it's compiler bug? The compiler we use is known to be good and I tried a different one anyway.

I don't think its a compiler bug, but when it comes to debugging drivers it pays absolutely to make *NO* assumptions about how wonderful the compiler is, and to know explicitly what is going on with optimization, register usage and so on. GCC for ARM has, sometimes, a very massive impact on performance if you're not paying attention .. ARM is such a hot market that there is a lot going on for various SOC's at any given time, and I would really be very, very meticulate about whats going on.

Can you do me a favour and emit assembly for the wl12xxx driver with your current build system/compiler setup? Add the "-E" option for the driver build, and give it a temporary place to store the output files, then send me the output .. it could prove fruitful, especially with the cache/boundary-alignment question.

 

[torpor]

BTW, notaz, can you describe the symptoms a little more .. got debug logs and so on? The wl12xx drivers seem to have at least some debug output you can turn on, are you able to trace those messages?

 

[Exophase]

Emit assembly is -S, -E is perform preprocessing. notaz probably knew that though.

 

[lulzfish]

Traces are too far apart on the PCB, wires can't crosstalk without yelling, which disturbs the IRC queue

 

[u9i]

LOL, lulzfish, that was the best answer so far

 

[torpor]

lol, thanks Exo, thats what I meant.. glad you caught it..

 

[diegor]

Ah, this black-box aspect is a bit of a lame duck though, because obviously N900 know enough about the wifi module to get it working, and that knowledge is represented in the latest linux-2.6.34-rc2 codebase for wl12xx .. BTW, did you notice the rather extensive set of updates to this codebase between the linux-2.6.33 and linux-2.6.34-rc2 releases? There is some work going on there .. what version of these drivers are you working with?

Probably you should take a look at this:

http://git.openpandora.org/cgi-bin/gitweb.cgi?p=openpandora.oe.git;a=blob;f=recipes/pandora-system/wl1251-modules.bb;h=f53712a91419512cb155e4021c1ba802ec9d7df9;hb=HEAD

registers on ARM - you think it's compiler bug? The compiler we use is known to be good and I tried a different one anyway.

I don't think its a compiler bug, but when it comes to debugging drivers it pays absolutely to make *NO* assumptions about how wonderful the compiler is, and to know explicitly what is going on with optimization, register usage and so on. GCC for ARM has, sometimes, a very massive impact on performance if you're not paying attention .. ARM is such a hot market that there is a lot going on for various SOC's at any given time, and I would really be very, very meticulate about whats going on.

Do you think that they have 45KB/s , because driver is not optimized? Hard to believe. Usually optimization give 20-30% of speedup, not so massive.

 

[Dunny]

I don't think its a compiler bug, but when it comes to debugging drivers it pays absolutely to make *NO* assumptions about how wonderful the compiler is, and to know explicitly what is going on with optimization, register usage and so on. GCC for ARM has, sometimes, a very massive impact on performance if you're not paying attention .. ARM is such a hot market that there is a lot going on for various SOC's at any given time, and I would really be very, very meticulate about whats going on.

Do you think that they have 45KB/s , because driver is not optimized? Hard to believe. Usually optimization give 20-30% of speedup, not so massive.

I think you're missing the point somewhat. The problem is that turning optimisations on will lead to changes in the way your code is translated into assembly. In general, this isn't a problem as compiler optimisations are pretty well thought out these days (I have an object pascal compiler which really barfed my code a couple of years ago when optimisation was turned on) but as has been mentioned, it may be enough to throw something like a driver off track. When you get used to a compiler, you get a feel for what asm it will spit out for any given sequence of instructions, and if you rely on that knowledge, then optimisation can really make debugging hard at times.

D.

 

[Alec]

~ $ fortune bofh-excuses
BOFH Excuse #205:

Quantum dynamics are affecting the transistors

Sure, why not.

 

[droneb]

* Word-/page- alignment issues.

...would cause an exception and be quickly seen.

* Subtle compiler differences between what gcc for x86 (where the existing driver was developed and has received most of its testing) is expecting to have to output for the packet_buffer, and ARM?

The driver was developed for N900 by Nokia and has been shown to work on N900 and various ARM HTC devices like G1.

* Tried building the module with all optimizations off?

I see no reason to do that. Multiple compilers were tested.

* Incorrect register usage. (If I had a Pandora, this is the first thing I'd check - all registers are being used properly, in the right order, etc.)

Which registers do you have in mind?
MMC/SD on OMAP - same code drives both SD slots, also firmware upload works fine, it's not SDIO like I stated multiple times.
registers on wifi module - more likely, the module is basically undocumented black box and no one knows how to program it (except TI; this is "normal" for wifi modules though), but the driver works on other similar devices.
registers on ARM - you think it's compiler bug? The compiler we use is known to be good and I tried a different one anyway.

i was on the same predicament some years ago, well not as complex as a wifi unit, but with a RF tranceiver, the worst thing is that it returned no states information back to control. tha awfull black box model and the datasheet is completely useless

and i understand that if RF was hard, wifi must be nuts using inverse engineering, had a big headache for nearly a year and a half.

 

[notaz]

If the pbuffers for RX are not set at a 16-byte aligned address, and you tell the chipset to pass data to an address that is not so aligned, you may not see an exception, actually, but may just not get a full buffer transferred ..

Wouldn't that cause some more visible errors? I believe the driver has alignment right.

* Subtle compiler differences between what gcc for x86 (where the existing driver was developed and has received most of its testing) is expecting to have to output for the packet_buffer, and ARM?

The driver was developed for N900 by Nokia and has been shown to work on N900 and various ARM HTC devices like G1.

Yes, but how was the driver *compiled* for the N900 - which version of gcc, which switches, etc. Just because Nokia got it built and working for N900/Maemo does not mean that the same code compiles exactly the same for Pandora/OpenEmbedded ..

It's done how it's set up by Kbuild. Might try to disable optimization but really doubt it will help.

Looking aimlessly at full driver asm compiler output is a waste of time IMO.

I don't mean the SDIO control stuff, I'm talking about the wlxxx (you know what: I still don't know the *exact* specifics of which chip is being used in Pandora, sorry about this generality) controller registers for setting up the link, opening the radio, turning on/off CRC on packets, and so on ..

The only way to access that thing is over SDIO bus.

Ah, this black-box aspect is a bit of a lame duck though, because obviously N900 know enough about the wifi module to get it working, and that knowledge is represented in the latest linux-2.6.34-rc2 codebase for wl12xx ..

It's just a rewrite of TI driver, I don't think they have full specs, although they might do.

BTW, did you notice the rather extensive set of updates to this codebase between the linux-2.6.33 and linux-2.6.34-rc2 releases? There is some work going on there .. what version of these drivers are you working with?

The latest from wireless-testing. Some earlier versions were tested too.

Can you do me a favour and emit assembly for the wl12xxx driver with your current build system/compiler setup? Add the "-E" option for the driver build, and give it a temporary place to store the output files, then send me the output .. it could prove fruitful, especially with the cache/boundary-alignment question.

http://notaz.gp2x.de/tmp/cw_out.tar.bz2
lots of junk there but I guess you'll manage.

BTW, notaz, can you describe the symptoms a little more .. got debug logs and so on? The wl12xx drivers seem to have at least some debug output you can turn on, are you able to trace those messages?

The symptoms are simple - slow transfer or packet loss on RX path mostly (TX is performing better).
What debug output does interest you? Loads of things can be logged by the driver, but none of them are really useful IMO.

 

[Orion4874]

This may be completely useless but have you tried something like this: http://en.wikipedia.org/wiki/Iperf

It's open source as well.

 

[Mr.Confuzed]

Cosmic Rays similar to what some are claiming to affect Toyota's acceleration problems.

http://www.wired.com/autopia/2010/03/toyota-cosmic-rays/

Nah, if that was it, then the chip wouldn't stop transmitting.

 

[Exophase]

This might be a really stupid question/suggestion, but is it possible to log RX interrupts externally (like with a logic analyzer), possibly via an externally visible source (like timer output, you can put one out on ext right?) and count/timestamp RX interrupts in the driver?

I know this is really common sense stuff and you've probably been down this path, but I've had problems with communications because of interrupts being starved and hardware FIFOs not keeping up - especially in cases where there was barely any hardware FIFO to begin with. I mean, if something else in the kernel is killing interrupts for too long, or preempting the driver.

Okay, I'll go back to not insulting your intelligence ;p

 

[atomicthumbs]

someone send the dev team a Bus Pirate

 

[SNESFAN]

someone send the dev team a Bus Pirate

YARRRR!!!!

 

[rv6502]

on the gamecube not following the exact same initialization pattern as recommended caused a lot of errors, random packet loss, etc.
might be something similar.


on some chips if registers are not initialized in the "proper" order they enters a weird state in their internal state machine or badly synchronize the different clocks so that the latches can capture in-between data, (kind of like when an LCD is connected with a VGA cable and you can see pixel jitter).

some FIFO circuits can then miss the buffer-full condition sporadically and overwrite data in the ring buffer

differences between the wifi clock and the bus clock might cause random issues.

(just throwing some ideas in)

 

[torpor]

I realize you're doing this on a volunteer basis notaz, and that there may be some frustration in having to explain yourself over and over - I'm sure the OP team have questions too - but in the interest of preparing to help with this situation once the hardware arrives, I hope you can pitch in with some details on this thread and we can get, at least for now, extra eyeballs on the problem ..


I just want to clear something up - which version of the driver are you basing this analysis on - the Android sources ported to 2.6.27, or something else? I am a bit confused about how the Android drivers came into this scenario and what the current test-base is .. Because as I see it there are the following 'versions' of the wl12xx drivers available:

* Original binary-blob only TI module (useless unless you boot their kernel, and then its only useful to verify operational ability of the hardware: so, not *totally* useless for end-users, but certainly it would put MWestons mind to rest if this driver worked 'okay'..)

* Android quick-hack drivers

* wl12xx drivers which appeared from 2.6.33

* newer, updated drivers (which I see you're contributing to, actually, notaz .. interesting because I thought PandoraOS was currently 'frozen stuck' at 2.6.27) in 2.6.34-rc2 ..

The symptoms are simple - slow transfer or packet loss on RX path mostly (TX is performing better).
What debug output does interest you? Loads of things can be logged by the driver, but none of them are really useful IMO.

Can you describe the test procedure here - are you doing ftp transfers or http, or do you have a netcat or arping test rig set up to move up from raw packets, ARP, UDP, TCP, etc? If it would help I could rig up some test apps that send and receive raw packets from the Pandora to a Linux host and check things like CRC values, retry counts, etc. Or we could use the test tools from the portfwd package (udp_snd/udp_rcv) with a bit more inspection of the results to see if we can determine if the packet losses are happening due to CRC errors, retry/retransmission, frequency collision, etc.

The wl12xx driver can output the following variables to debugfs, and I would like to know, pretty much, everything it reports before, during, and after a transfer test using UDP:

internal_desc_overflow, out_of_mem, hdr_overflow, hw_stuck, dropped, fcs_err, xfr_hint_trig, path_reset,
reset_counter, rx_requested, rx_errors, tx_requested, tx_errors, cmd_cmplt, fiqs, rx_headers, rx_mem_overflow,
rx_rdys, irqs, tx_procs, decrypt_done, dma0_done, dma1_done, tx_exch_complete, commands, rx_procs, hw_pm_mode_changes, host_acknowledges, pci_pm, wakeups, low_rssi, addr_key_count, default_key_count,
key_not_found, decrypt_fail, packets, interrupt, ps_enter, elp_enter, missing_bcns, wake_on_host, wake_on_timer_exp,
tx_with_ps, tx_without_ps, rcvd_beacons, power_save_off, enable_ps, disable_ps, fix_tsf_ps, s_spread, rcvd_awake_beacons, rx_pkts, calc_failure, encrypt_fail, internal_desc_overflow, out_of_mem, hdr_overflow,
hw_stuck, dropped, fcs_err, xfr_hint_trig, path_reset, reset_counter, rx_requested, rx_errors, tx_requested,
tx_errors, cmd_cmplt, fiqs, rx_headers, rx_mem_overflow, rx_rdys, irqs, tx_procs, decrypt_done, dma0_done,
dma1_done, tx_exch_complete, commands, rx_procs, hw_pm_mode_changes, host_acknowledges, pci_pm, wakeups,
low_rssi, addr_key_count, default_key_count, key_not_found, decrypt_fail, packets, interrupt, ps_enter,
elp_enter, missing_bcns, wake_on_host, wake_on_timer_exp, tx_with_ps, tx_without_ps, rcvd_beacons, power_save_off,
enable_ps, disable_ps, fix_tsf_ps, s_spread rcvd_awake_beacons, rx_pkts, calc_failure, encrypt_fail ..

I'm hoping you're using a version of the wl12xx drivers that has support for this instrumentation .. if not, well .. yeah. I think its important to know what driver you're using, actually.

Oh .. and by the way, I do have a Bus Pirate, and sure I'd be happy to use it to work on this issue once my Pandoras arrive ..

 

[jens.l]

Can this conversation help?

http://markmail.org/message/nwpu2eddjtw3g2ai

MfG
Jens