The Wifi Problem Will Be Found To Be Caused By:

[Eight Bit]

Funny how even though I don't get over 90% what is written, I find it quite interesting stuff to read

 

[notaz]

I realize you're doing this on a volunteer basis notaz, and that there may be some frustration in having to explain yourself over and over - I'm sure the OP team have questions too - but in the interest of preparing to help with this situation once the hardware arrives, I hope you can pitch in with some details on this thread and we can get, at least for now, extra eyeballs on the problem ..

No problem with that, also making this public so that somebody else could help if they wish (and for possible future contributions for improving it).

I just want to clear something up - which version of the driver are you basing this analysis on - the Android sources ported to 2.6.27, or something else? I am a bit confused about how the Android drivers came into this scenario and what the current test-base is .. Because as I see it there are the following 'versions' of the wl12xx drivers available:

* Original binary-blob only TI module (useless unless you boot their kernel, and then its only useful to verify operational ability of the hardware: so, not *totally* useless for end-users, but certainly it would put MWestons mind to rest if this driver worked 'okay'..)

There was never such thing as binary blob driver (unless you count firmware that runs on chip, but it's always closed as on probably every other wifi device), that idea came from speculation on these boards I think. There was only driver source under NDA that MWeston got. Initially he had problems passing that to me as TI wanted company to sign the NDA, and I don't have one (I couldn't use my day job company for this for obvious reasons).

However then the

* Android quick-hack drivers

..came out which after some comparison appeared to be mostly the same code as NDA code that MWeston had, only with BSD headers slapped on them. I could then get them to work with similar results we have today (slow speed, packet loss and such). That driver has a problem of not supporting Linux wireless stack and needs custom tools to start, connect, etc. It was then decided that hardware works ok and there were some issues with the driver that nobody wanted to look at as that code is not of the best quality.

* wl12xx drivers which appeared from 2.6.33

When these came out DjWillis took them and got working. They appeared much better due to proper Linux wireless stack support. However he was getting some SDIO errors due to a problem I recently fixed, and the false "wifi does not work because of SDIO" claim was born.

* newer, updated drivers (which I see you're contributing to, actually, notaz .. interesting because I thought PandoraOS was currently 'frozen stuck' at 2.6.27) in 2.6.34-rc2 ..

The plan is to only use 2.6.27 for the first release and move to newer kernels with firmware upgrades. This is the driver I currently use, I mostly use compat-wireless backport to .27, but I've tested it on 2.6.34-rc1 too.

The symptoms are simple - slow transfer or packet loss on RX path mostly (TX is performing better).
What debug output does interest you? Loads of things can be logged by the driver, but none of them are really useful IMO.

Can you describe the test procedure here - are you doing ftp transfers or http, or do you have a netcat or arping test rig set up to move up from raw packets, ARP, UDP, TCP, etc?

Simple high level wget test is enough for me.

If it would help I could rig up some test apps that send and receive raw packets from the Pandora to a Linux host and check things like CRC values, retry counts, etc. Or we could use the test tools from the portfwd package (udp_snd/udp_rcv) with a bit more inspection of the results to see if we can determine if the packet losses are happening due to CRC errors, retry/retransmission, frequency collision, etc.

Not sure how that will help, but if you give me exact test cases I have no problem running them.

The wl12xx driver can output the following variables to debugfs, and I would like to know, pretty much, everything it reports before, during, and after a transfer test using UDP:

No useful information there, all error counts are 0, I can give you exact output if wish but specify a test to run in that case.

 

[hdonk]

Funny how even though I don't get over 90% what is written, I find it quite interesting stuff to read

It's quite simple - there's lots of ways to transfer data between peripherals (in this case the SD bus) and the memory that the CPU (OMAP3530) uses. One of these is for the CPU to go directly to the peripheral and pull the data across bit, byte or word at a time (depending on bus width). Another is to use DMA - Direct Memory Access. With DMA the SD bus controller writes the data you are wanting straight in to a block of main memory, without the CPU being used. This means you get the full benefit of the memory interface - with things like block or page transfers, full memory bus width transfers, etc...
However when doing DMA you have to be very careful about how things are set up - such as how the CPU sees the memory that the peripheral is transferring to. Fer instance, if the memory is marked as cached by the CPU, when your device driver goes to access the block, it may think it is in cache already, and pull that version of data in, rather that the version that has been recently DMA'd in to the main memory. Also the peripheral is likely to be very tightly constrained in how it can write in to the memory, for instance only starting on word aligned boundaries.
All in all it means that accessing the peripheral by bit banging is a lot easier, but very very CPU intensive and slow. DMA is much much faster and more CPU efficient, but can be tricky to manage.

Reading quickly through that message exchange, it looks like they were finding that the WiFi device was working ok bit banging, and when they tried DMAs, they ended up having to get the alignment & DMA transfer length exactly right for it to work without errors.

Apologies for any mistakes in my explanation B)

 

[torpor]

No useful information there, all error counts are 0, I can give you exact output if wish but specify a test to run in that case.

Well, dump all the values for these debug vars prior to the test, then do a netcat between the test pandora and a linux PC, transferring 1024 bytes of data, and while the transfer is occurring, dump these vars as often as possible, then after the transfer is done, dump the vars again .. wait a minute .. then dump the vars again. This will give us some clues as to what point in the driver things may be getting hung up - remember, we're not alone with having to debug this sort of issue, and the presence of all of these vars in the debugfs can be very, very benficial to getting an idea of what behaviour is triggering things.

For the netcat, use UDP mode and dump 1k of /dev/null over the link. Then it might actually be useful to do it again, but dump 1k of /dev/random instead. During both tests, collect all data as above.

I'd write you a testsuite script for this, but I am at work right now and can't really do that .. but I think you get where I'm coming from here - the more data you can glean from debugfs during both of these netcat tests (null and random data), the better ..

 

[Eight Bit]

It's quite simple - there's lots of ways to transfer data between peripherals (in this case the SD bus) and the memory that the CPU (OMAP3530) uses. One of these is for the CPU to go directly to the peripheral and pull the data across bit, byte or word at a time (depending on bus width). Another is to use DMA - Direct Memory Access. With DMA the SD bus controller writes the data you are wanting straight in to a block of main memory, without the CPU being used. This means you get the full benefit of the memory interface - with things like block or page transfers, full memory bus width transfers, etc...
However when doing DMA you have to be very careful about how things are set up - such as how the CPU sees the memory that the peripheral is transferring to. Fer instance, if the memory is marked as cached by the CPU, when your device driver goes to access the block, it may think it is in cache already, and pull that version of data in, rather that the version that has been recently DMA'd in to the main memory. Also the peripheral is likely to be very tightly constrained in how it can write in to the memory, for instance only starting on word aligned boundaries.
All in all it means that accessing the peripheral by bit banging is a lot easier, but very very CPU intensive and slow. DMA is much much faster and more CPU efficient, but can be tricky to manage.

Reading quickly through that message exchange, it looks like they were finding that the WiFi device was working ok bit banging, and when they tried DMAs, they ended up having to get the alignment & DMA transfer length exactly right for it to work without errors.

Apologies for any mistakes in my explanation

Wouldn't that mean that there is no Hardware issue? As I understand from your explenation the chip is working properly but very CPU intensive but when switched to DMA to streamline the process things are screwed up?

 

[hdonk]

Wouldn't that mean that there is no Hardware issue? As I understand from your explenation the chip is working properly but very CPU intensive but when switched to DMA to streamline the process things are screwed up?

It depends - if the problem being discussed on the mailing list referred to earlier is the same as on the Pandora, then no hardware problem - just a software one. We just need to wait and see what notaz, torpor and the others find. From past experiences debugging this kind of problem, the solution could pop out in the next day, or take months - if it's a software problem. From the information notaz has been posting here though, it is very plausibly software.
However before blaming the hardware you have to be absolutely sure you have covered all the possible software options. Working with a binary blob based firmware system makes that incredibly hard however. It's bad enough when you've got the full datasheets for both ends and support from the chip manufacturers on the phone. When the bus is as simple as the SD bus, the hardware (in terms of the circuit design) is the least likely to be the problem. A layout issue is plausible when shifting data around at fast enough clock rates, but very very hard to diagnose. Also when working with board layouts as complicated as the Pandora's, you can't usually just go in and hack tracks on the PCB to isolate hardware problems. Unless you are very lucky or foresaw a problem with the particular IC!

 

[notaz]

I'd write you a testsuite script for this, but I am at work right now and can't really do that .. but I think you get where I'm coming from here - the more data you can glean from debugfs during both of these netcat tests (null and random data), the better ..

Same here, sitting at work.
There is not much chance to dump debugfs during 1k transfer though, it will take more time taking debugfs dump than to perform the transfer.

It depends - if the problem being discussed on the mailing list referred to earlier is the same as on the Pandora, then no hardware problem - just a software one. We just need to wait and see what notaz, torpor and the others find.

The DMA problem discussed there was preventing driver to work completely, and SDIO irq problem was causing slowness. Neither applies here - DMA works as the driver works, SDIO irq is not used at all as we have GPIO line for this.

 

[hdonk]

The DMA problem discussed there was preventing driver to work completely, and SDIO irq problem was causing slowness. Neither applies here - DMA works as the driver works, SDIO irq is not used at all as we have GPIO line for this.

In that case torpor seems to have the right idea - more data on where the problem is occurring according to the debug stats would help with the speculations If the low data rate is actually caused by error'd packets rather than a DMA transfer fault though... I can see why the hardware is under suspicion.

 

[Alec]

For the netcat, use UDP mode and dump 1k of /dev/null over the link. Then it might actually be useful to do it again, but dump 1k of /dev/random instead. During both tests, collect all data as above.

I think you meant /dev/zero and /dev/urandom because that wouldn't work very well...

 

[eyeball]

Have you tried switching it off and then back on again?

 

[Bosbeetle]

Have you tried switching it off and then back on again?

:lol:

 

[SNESFAN]

"well... did you reboot it?"
"yea"
"how many times?"
"three man, you you always tell me to do it three times"

- web dude

 

[torpor]

Yes, 1k may be too little data to test with .. unless things are very slow, which is what I thought was the case (40kbit/s?), and I would be happy if this were scripted anyway which is why I suggested netcat, which can run the whole dump of debugfs through the script ..

No, it doesn't matter if its /dev/urandom or /dev/random - just *some* random data, either way, compared to a full-zero's test. The entropy doesn't matter - what matters is that its just different to a block of 000's ..

Anyway, maybe it would be good to put your test Pandora on the 'net and we can work on this in the evening at home.

BTW: GPIO is being used. Hmm.. Sure this isn't getting noise from other peripherals? I'm not fully familiar with the peripheral usage of GPIO on Pandora, but if GPIO is being used instead of a real IRQ, well then .. a bit of soul-searching around GPIO might be in order. Just an idea, don't take it seriously, but with debugging drivers its usually a good idea not to ignore ideas on face value, and to verify with data collection and inspection, instead ..

 

[Alec]

No, it doesn't matter if its /dev/urandom or /dev/random - just *some* random data, either way, compared to a full-zero's test. The entropy doesn't matter - what matters is that its just different to a block of 000's ..

Assuming you actually have enough entropy. Comapre:

$ dd if=/dev/urandom bs=1kB count=1 of=/dev/null
1+0 records in
1+0 records out
1000 bytes (1.0 kB) copied, 0.000635833 s, 1.6 MB/s
$ dd if=/dev/random bs=1kB count=1 of=/dev/null
0+1 records in
0+1 records out
128 bytes (128 B) copied, 0.000229783 s, 557 kB/s

Not to mention that /dev/random produces data at a rate that is probably slower than 40 kb/s if you actually get it to produce more...

 

[torpor]

The degree of entropy doesn't matter .. all that matters is that we have a source of random data with which to automate the testing.

 

[Alec]

The degree of entropy doesn't matter .. all that matters is that we have a source of random data with which to automate the testing.

Sorry if there's a misunderstanding, but that's exactly why I am saying that using /dev/random for that is bad idea - you're not going to get much data out of it! If all you need is a bunch of garbage data, use the fast pseudo-random /dev/urandom instead.

 

[WizardStan]

Sorry if there's a misunderstanding, but that's exactly why I am saying that using /dev/random for that is bad idea - you're not going to get much data out of it! If all you need is a bunch of garbage data, use the fast pseudo-random /dev/urandom instead.

You could dump a sufficient amount of random into an actual file, and then stream that file. Done this way, you can get highly random data that is also confirmable and repeatable.

 

[torpor]

Either way, the thing that matters is that a proper test harness is set up to use to get details about the performance of the driver .. I'll have a look at helping out with this later this evening.

 

[sryidc]

i predict the WiFi not working is a april fools joke. quote me when im right.

 

[Alec]

Either way, the thing that matters is that a proper test harness is set up to use to get details about the performance of the driver .. I'll have a look at helping out with this later this evening.

Exactly. I was just pointing out so that no one is wondering why only a few bytes are being transferred. I remember stumbling on that years ago