[RESOLVED] Send binary data without headers

[kebo]

Hey all,
This is not really a programming question, but I think there are some smart people here that can help me.

I need to send binary data over a serial port (usb, ethernet, or whatever) and I need to include header information so the receiver knows what the data is and how to handle it. Normally I would convert the data to ASCII before sending it (i.e to send "A" which has ASCII value = 0x41 I would send "4" & "1") then convert back to binary at the receiver. Doing this allows all ASCII values that are not "0" - "9" and "A" - "F" to be used as header data. The trade off is that you end up sending 2X number of bytes. Another way to do this would be to send a fixed length header before the data. This though doesn't really work well with my situation.

What I am trying to create is an Explorino application that will allow me to view and manipulate the file system on my Arduino's SD card. This requires possibly sending files that could be megabytes in size making ASCII conversion as mentioned above not taste so good give the limited Arduino resources. I would also need to send a file path first so a packet could have 2 pieces of information; the file path and the file. I could do this with a fixed length header, but I would have to make sure the file path fit in the header. If I were sending the stream to a device with more resources this would not be an issue, but to make this application useful, I need to keep the receive data buffer as small as possible (64 or 128 bytes).


Arduino aside, has anyone implemented a binary packet with headers other than sending ASCII text or sending fixed length headers?

Yhanks for looking
kevin

[PlausiblyDamp]

Hey all,
I need to send binary data over a serial port (usb, ethernet, or whatever) and I need to include header information so the receiver knows what the data is and how to handle it. Normally I would convert the data to ASCII before sending it (i.e to send "A" which has ASCII value = 0x41 I would send "4" & "1") then convert back to binary at the receiver. Doing this allows all ASCII values that are not "0" - "9" and "A" - "F" to be used as header data. The trade off is that you end up sending 2X number of bytes. Another way to do this would be to send a fixed length header before the data. This though doesn't really work well with my situation.

Perhaps I am missing something important here having never sent data to an Arduino device but if you want to send the binary value 0x41 why wouldn't you just send a byte containing the value 0x41, what does the conversion to ASCII and back to binary data achieve?

Sorry if I am asking a stupid question, it could be my inexperience of the target environment showing.

[kebo]

Just to be clear, this has nothing to do with Arduino specifically. The same situation could arise sending serial data between any 2 platforms. I only mention Arduino to bring up the limitations of my particular situation.

By sending data as ASCII, the receiver can act when specific characters arrive. For example, if I am streaming data, I might use a specific byte (maybe 0x21 asc("!")) to mark the beginning of a packet and another byte (maybe 0x7E asc("~")) to mark the end. Once the receiver sees either of those bytes, it can either begin to collect the packet data, or it can go process the data. If the binary data was sent as true binary data however and if that data contained 0x21 the receiver would see the byte as the end of the packet and go to work processing it without having received everything that was sent. Using ASCII conversion guarantees the data will be sent using only "0" - "9" and "A" - "F" leaving all other characters to be used as headers.

Essentially, sending ASCII data is a way to separate header values from the true data.

edit...
Just to further elaborate the point, this is one of my typical comm packets to one of my devices...

<preamble><address><function1><data (0-9, A-F only)> <terminator>

This particular device has about 40-50 different values for the function each of which causes a different action and different interpretation of the data on the receiving end.

[PlausiblyDamp]

Just to be clear, this has nothing to do with Arduino specifically. The same situation could arise sending serial data between any 2 platforms. I only mention Arduino to bring up the limitations of my particular situation.

By sending data as ASCII, the receiver can act when specific characters arrive. For example, if I am streaming data, I might use a specific byte (maybe 0x21 asc("!")) to mark the beginning of a packet and another byte (maybe 0x7E asc("~")) to mark the end. Once the receiver sees either of those bytes, it can either begin to collect the packet data, or it can go process the data. If the binary data was sent as true binary data however and if that data contained 0x21 the receiver would see the byte as the end of the packet and go to work processing it without having received everything that was sent. Using ASCII conversion guarantees the data will be sent using only "0" - "9" and "A" - "F" leaving all other characters to be used as headers.

Essentially, sending ASCII data is a way to separate header values from the true data.

edit...
Just to further elaborate the point, this is one of my typical comm packets to one of my devices...

<preamble><address><function1><data (0-9, A-F only)> <terminator>

This particular device has about 40-50 different values for the function each of which causes a different action and different interpretation of the data on the receiving end.

Ah, I get the problem now. Personally I would pick a byte that typically has a very low occurrence in a typical byte stream and make that the terminator byte. The only issue is if you do expect this character to crop up in the byte stream then you would need some way to escape this character - similar to how C based languages prefix a / before certain characters in a string.

If for example you decided that the @ symbol (0x40) was a rare character you could choose the byte 0x40 as your terminator then as long as this character didn't occur in the original byte stream finding it would indicate the end and there would be no issues or additional overheads.

e.g.

0x01 0x30 0x50 0x99 0x40

would be a complete sequence of the bytes 0x01 0x30 0x50 0x99 terminsted by the 0x40

If this character did occur e.g.

0x01 0x30 0x50 0x99 0x40 0x23

then chose another byte as an escape character, for the sake of argument (as it is also what C etc use) I will select \ (0x5C) as the escape character therefore the byte sequence 0x5C would be an escape character so if you had the sequence 0x5C 0x40 it would need to be treated as if a single 0x40 byte was present and not a termination character.

The encoded byte stream would now be

0x01 0x30 0x50 0x99 0x5C 0x40 0x23 0x40

so when decoding if you spot 0x5C followed by a 0x40 you know to simply treat it as a single 0x40

If you want to actually include the 0x5C byte then simply double it up so a 0x5C 0x5C would be treated as a single 0x5C when decoding.

e.g. an original sequence of

0x01 0x30 0x5C 0x99

would be encoded as

0x01 0x30 0x5C 0x5C 0x99 0x40

This way you only ever incur a single additional byte if you have either a 0x40 or a 0x5C anywhere in the byte stream.

[kebo]

I have considered multi-byte protocols, but not in the data byte stream. There is simply not enough horse power in an Arduino to look at every byte of a 1 mega file for an escape char. It would take too long (minutes I would guess.) Once the beginning of the file is known, it needs to stream directly to the SD card 256 bytes at a time.

I could imagine a scheme where a GUID was used to encapsulate the header data, then another GUID used for the terminator. Seems a bit dodgy though because there is no guarantee the GUID's wouldn't appear in the data as unlikely as it may be.

[PlausiblyDamp]

If you are going to be streaming to the card in fixed sized blocks of 256 bytes could you not use a simple prefix - say allocate the first 4 or 8 bytes to indicate the number of blocks you are sending? You could still implement an escaped terminator but this would only be needed in the final block.

[passel]

I usually would indicate the length of the data to follow in the header. You could still use a terminating value at the end of the data just to serve as a framing check (assuming you don't bother with a CRC checksum of the data for better validation).
Or, you could use something like Trivial FTP does, which uses UDP. It transfer fixed packets of 512 bytes and it knows when it is done when a packet of less than 512 bytes is sent. If the data happens to end on a 512 byte boundary, then a final packet of 0 bytes is sent to signal the end of the data.

Of course, that is based on UDP which sends packets, rather than TCP which would be a stream and the size of the received data in any transaction can be an arbitrary size.

[kebo]

I didn't think that I could send the binary data length in the header. Sounds pretty simple now. Got some work to do.
thanks guys (or gals whichever you prefer).