Hello,
I am using the CC3000 + MSP430FRAM Kit and I have written an application which allows to transfer arbitrary amounts of data from and to the CC3000 via WiFi to do throughput benchmarks.
To my surprise, when adding a consistency check which checks the data actually transferred, I found that the data transferred from the CC3000 to my PC is out of order. This is unexpected since TCP streams are by definition reliable and all data always is transferred complete and always in order. In fact TCP does not have the concept of packets which can get lost. It is just a byte stream.
My test application sends a lot of segments of size N bytes. I tested this with N= 16 to 512. This 'packet' size is of course specific to my application and does not have anything do to with TCP. The problem appears for all segment sizes.
The symptom is as follows:
From the CC3000 I send the following byte stream (here in hex):
00 00 00 00 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
10 00 00 00 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
20 00 00 00 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
30 00 00 00 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
40 00 00 00 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
50 00 00 00 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
60 00 00 00 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
70 00 00 00 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
64 61 74 61 20 6F 6B 0A
This data is sent in segemnts of size 16 (which should be irrelevant information with TCP). As you can see, the first byte increase by 16 (segment size) for each segment which is intended and allows to cross check the data on the receiver side. All other bytes in the segments are just increasing bytes starting with 4 on position 4 and so on.
The data above is generate by a for loop, and at the end is the string 'data ok\n'.
I receive (dumping the content of the actual data returned by recv(), the splitting points do not matter):
recv: 00 00 00 00 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F (len=16)
recv: 10 00 00 00 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F (len=16)
recv: 20 00 00 00 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F (len=16)
recv: 30 00 00 00 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F (len=16)
recv: 60 00 00 00 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F (len=16)
recv: 70 00 00 00 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F (len=16)
recv: 64 61 74 61 20 6F 6B 0A (len=8)
recv: 40 00 00 00 04 05 06 07 (len=8)
recv: 08 09 0A 0B 0C 0D 0E 0F (len=8)
recv: 50 00 00 00 04 05 06 07 (len=8)
recv: 08 09 0A 0B 0C 0D 0E 0F (len=8)
As you can see all data is OK, except that the 40 and 50 segments are missing in the stream (remember, this is NOT datagrams or packets, this is supposed to be a reliable byte stream). The 40 and 50 segments get finally transmitted at the 'end' of the total transmission. What that means is that my algorithm feeds data into the CC3000 as fast as it can (of course) and only once I stop for 100ms or so the help up packets get transmitted out of order.
None of the called functions return an error code (neither on the PC nor on the CC3000) and I call hci_unsolicited_event_handler() before and after each send() call. I do not call select() in any way. I call send() back to back.
This is broken and must never happen on a TCP connection.
Now my question: Is there a new CC3000 firmware which fixes this problem? I'd be happy to betatest also. Is there a workaround?
The only workaround I found so far is to wait for about 100ms after _each_ send() call. This gives the CC3000 enough time to send the segments before I push a new one in and so they never get out of order. However, this is not feasible for my target application since this limits the data rate to below 1kB/s which is not feasible.
I used 'tcpdump' on the PC and found that the TCP/IP packets already carry the out-of-order data. There are not unusual ACK or retransmits going on. The TCP/IP packets come in in-order and already carry the out-of-order data.
This happens with two different access points, with and without WPA2 encrption. It also happens for short transmissions and for long ones: For long ones also only 1-2 segments get 'dropped' and get appended at the very and, with potentially many megabytes being transmitted inbetween. The total data being reordered is always small, less than 1.5kb. It probably gets buffered erroneously in the CC3000.
This happens also for bigger segments like 512 bytes, also for segments of arbitrary size like 417, 113 etc. Always whole segments will get 'dropped' and inserted at the end.
The actual drop/insert pattern depends on the segments size and the total transfer size. It does not occur when only transferring 4 segments or less. But a pattern is always 95% reproducible. This means when I repeat the test described above in 19 of 20 runs (at least) it will always delay the segments 40 and 50, in the same way.
It feels like the CC3000 is not using a FIFO for the packets but a stack, last in first out. When the data is produce slow enough then the FIFO and the stack are of size 1 in which case they are identical. :-)
I am using the CC3000 host driver from the latest Basic WiFi example.
Is there a workaround to fix this such that the transmission is fast (without waiting for 100ms between calls to send())? Any help on this is appreciated!
Cheers,
Johannes