[RESOLVED] Yes, another 'NEWBIE' question: TOPIC: ANSI region + WINSOCK

[flyguille]

I know, that if I uses strings to comunicate through mswinsck, somehow the ANSI table configured in Windows has something to do.

But the question is, why?. And exactly HOW?

What I understand from years of programming in VB is, it is UTF16, which is the string encoding, so, in VB each char of a string is 16bits, not 8, but, when I uses functions like CHR$() , ASC() it only handles 8bit LSB portion. And for my VB program concern the MSB 8bits is just zero, unhandled... right?

So, if i do some encription in the string value, like, mid$()= chr$(asc(mid$()) xor ChangingXoringFactor) or so, ANSI has something to do here? or it is ANSI safe in those formulas?


then, where is used the ANSI and exactly for what?, when sck.send String ??? why?.


I know it must communicate as byte array if want region-less proof. Just that jumping from strings parameter to byte array plus FAST it will be hard to do, and (allow concatenation within the array in/out buffering and so. It is like too much trouble but doable, not quite fast sure overhead.

[dilettante]

The "why" is because VB6 String values are normally UCS-16 but most TCP/IP protocols originally used ASCII. Conversion to ANSI is valid for all legal ASCII characters from 0 through 127. ANSI is not "Extended ASCII" though, so anything that maps to 128 to 255 is off the table. Don't do it.

Since then UTF-8 is often used, but VB6 has no direct UTF-8 support. To get that you have to do the conversions using an API call, ADODB.Stream, or a 3rd party library. The result is binary data as far as VB6 is concerned, so when sending/receiving you'd use Byte arrays.

If you want to work across locales UTF-8 is probably your best bet. The overhead is negligible, and necessary in any case. Almost no protocol sends UCS-2 or UTF-16 over the wire.

When implementing actual binary protocols you should never apply any character encoding.

[dilettante]

See the classic The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!).

[DEXWERX]

first off, you should use the Wide versions of Asc/Chr - AscW/ChrW

also I'm glad dil posted that link. You pretty much can't get away with any sort of text programming with out a basic understanding of unicode and encodings.

[flyguille]

See the classic The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!).

I were aware of most of it, but how he teaches it, is amazing, well, more entertainment than reading boring specs of UTF16.

[flyguille]

I am willing for a quick fix.... my files are already manifested, it is possible to force Windows or VB6 to run it with codepage 1252 always regardless of what win config it has?.

[flyguille]

first off, you should use the Wide versions of Asc/Chr - AscW/ChrW

also I'm glad dil posted that link. You pretty much can't get away with any sort of text programming with out a basic understanding of unicode and encodings.

About the kripto thing, which is different from "encoding"... if in a string it has only values from 0x0000 up to 0x00ff only, unusing the high byte, and it used chr$() and asc() only, and it xors with 8 bits value only, the output will be allways 8bits... I never saw a corruption by using those things. Atleast no character convertion happens there, no double character encoding, nothing wrong can happen, only the waste of memory....

but then, the sock, is said it want to convert the UTF16 string into ANSI table, for 8 bits transmission. ok. With the 1252 codepage I never saw a problem with that. And I guarantee it correspond for all 256 number the identical binary number. I transmitted files with it and HASH was perfect.

But now, the other part receive it, lets say a japanese, and it uses another ansi table..... QUESTION if he is receiving 8bits transmission, so ANSI, but he interprets it with that other ansi table, how come it can corrup? some 8bits numbers can be illegal and mapped to a question mark or something like that?.

[dilettante]

You can call SetThreadLocale() but there are pitfalls.

Just fix your program instead. Don't used String variables for binary data.

[flyguille]

You can call SetThreadLocale() but there are pitfalls.

Just fix your program instead. Don't used String variables for binary data.

Well, it is not that easy. Now, if I see the big picture of it.

Lets, said, my program gets popular, by now it is sold all over latin american. so all countries using CP 1252.

But what will be happening?

lets, said I then really REALLY want to fix it, it is not just the SOCKET transmission be ANSI senstive-less. But also all Inputs, parameters, filenames, include the text transmitted through chatrooms must be "sanitized" or supported full UTF16.

Right now, I wonders if a malicious inputed text, writing in the chat, can broke all communication.

Right now I am not sure.

But if it become a security / stability problem, I will be forced to fix it.


Now, thinking.... the only sub-set protocol which can spread over all servers is the chat builtin the system, others protocols will affect only the local server-node connection but not spread to others clients.




The good thing, is all protocols are encrypted, and packed, so it is like in some point, each char is XORED, so, ASC() -> CHR$() , so that makes impossible to transmit, let said a 32bit UTF (2x 16), because if it is a security problem, most will be , a single char of 32bits, offsetting the size informed in the header, where in the header specified the parameter length...

so basically, is unable to transmit special international languages than the ANSI selected in the nodes. But it won't **** the synchronization of the buffers. OK.!, no security problem. Just no japanese, no chinese, no korean, no cyric, but EEUU and SPANISH are ok, all "Western" ok.

Nop, not double side character encoded can ruin it, thanks to kripto.



mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm........... psssssssss.

first it reads the LEN() then follow the string data, then it packs, then it kript, if single kripto protocol..... then the pack header hmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm, the question is , it kripts before or after the pack! if the whole command header is wrong it will unsynchronize, dropping the connection in the next try to process a new command, or it can dead silently, ping pong protocols won't work anymore..... and it will reset and reconnect.

HOLY ****! what dissaster it can be!.


URGENTLY some CHR$((ASC() AND 255)) !!!!

and I though I were ok for christmass.

[Schmidt]

Well, it is not that easy.

But it *is*...
Even the VB6-Winsock-Control supports sending and receiving of plain (non-interpreted or transformed) ByteArrays.

And that's all you need to know at the transfer-layer.

What remains is, to write Converter-Routines for both directions - as e.g. to transfer VB-Strings properly:
- B = VBStringToUTF8(S)
- BComp = CompressBytes(B)
- BEnc = EncryptBytes(B)
- send ByteArray BEnc over socket
... transmission ...
- receive ByteArray BEnc from socket
- BDec = Decrypt(BEnc)
- BUnComp = Decompress(BDec)
- S = UTF8ToVBString(BUnComp )

Once you work with ByteArrays, the whole thing becomes especially easy for Files (or Images):
- B = ReadFileContentToBytes(FileName)
... rest of the send-process as already described above (including Compression, followed by Encryption)...
... transmission ...
... rest of the receiving part as already described above (including DeCryption, followed by Decompression)...
- WriteBytesToFile ( BUnComp, FileName )

So, as already stated - always work in "ByteArray-Space" at the socket-level (with proper Helper-Routines) -
and you will have an "internationalized" Layer that will not choke when system-locales differ between machines.

Besides, ByteArrays are a necessity, as soon as Encryption and Compression are needed as well.

Olaf

[DEXWERX]

This is kind of like Coutsj' threads trying to understand Unicode. Way over complicating the issue.

[flyguille]

This is kind of like Coutsj' threads trying to understand Unicode. Way over complicating the issue.

Well, is not that easy, exists already layer over layer over layer.

By example for the community chat, I don't uses normal TEXBOXES or RichTextBoxes, It uses special coded usercontrol class, which rely in a string buffer. And it can contain things like binary coded color codes, and emoticons coded in text. Etc., it is handled as object. So, moving inside the buffer the class does verification for knowing the start of each object, being a simple ANSI letter also handled as an object that can has a start and end position.

But all that is done, in a simple string buffer (fixed size, auto expanded size per chunk size (so avoiding frequently mallocs), so, a string buffer uses start/end POINTERs instead of passing string values between functions and procedures, for speed reasons.

But all that pointers hadling, pointing to a string, is just based on the rule: EACH text character = ONE POSITION fetchable using MID$(buf, POSITION, 1) is this rule ok?

if so, means MID$() can return up to a 32bit characted wide?.

QUESTION:

Or, is suppose to read two postions if detected the UTF16 > char is extended and then to read the second position?....is to say.... or like MID$(buf,pos, 2) but using only the first position and only the second position optionally ?

[couttsj]

I know it is convenient to use strings to manipulate data, but encrypted data uses the entire 8 bit byte (ANSI). If you are careful not to let VB create any strings for you, it is possible to use string manipulation, but there are a handful of characters above 127 (&H7F) that are considered to be Unicode characters, and VB will change those in the background without your knowledge. It is an extreme pain in the butt to use byte arrays because you will have to write all your own routines for things that were so easy to do using strings, but there are just too many potential pitfalls using strings. And there are things that are not easy to do with byte arrays, like creating an array of byte arrays.

J.A. Coutts

[dilettante]

Just make yourself a "byte string builder" like this BytesString class.

Demo:

Option Explicit

Private Sub Form_Load()
    Dim BytesVal() As Byte
    Dim StringVal As String

    With New ByteString
        'ANSI:
        BytesVal = StrConv("Test Bytes, ", vbFromUnicode)
        .Cat BytesVal
        .Cat StrConv("Test String", vbFromUnicode)
        Print """"; StrConv(.Value, vbUnicode); """"; " ("; .Length; ")"
    
        .Clear

        'Unicode:
        .Cat "Hello "
        .Cat "World!"
        StringVal = .Mid(12)
        Print """"; StringVal; """"; " ("; Len(StringVal); ")"
    End With
End Sub

[flyguille]

I know it is convenient to use strings to manipulate data, but encrypted data uses the entire 8 bit byte (ANSI). If you are careful not to let VB create any strings for you, it is possible to use string manipulation, but there are a handful of characters above 127 (&H7F) that are considered to be Unicode characters, and VB will change those in the background without your knowledge. It is an extreme pain in the butt to use byte arrays because you will have to write all your own routines for things that were so easy to do using strings, but there are just too many potential pitfalls using strings. And there are things that are not easy to do with byte arrays, like creating an array of byte arrays.

J.A. Coutts

Happens that most data source are just data entry, from textboxes, so in the natural way of programming, is natural that data information is handled in strings.

Then stored in files, or DBs., if it were UTF8, some code would be the EXTENSION code, to allow chars numbers bigger than >256.

But again, atleast with the 1252, Western codepage.... it is not DATALOSS about coding encrypt information just XORING the LOW byte of each character, the 256 numbers are writable with chr$(), and the 256 numbers are readable back with ASC(), so, there is not possibility of data loss or corruption about writing an enkription algorithm about just MID ()=CHR$(ASC(MID$() xor formula)) it is perfect.

This won't touch the high byte, and if using weird japanese char, well, will be encrypt just half.


Also I can certificate, that also is not dataloss about sending that string, WRITING a file with that string as BINARY OPEN "filename" as binary , PUT ,, string ..... GET ,, string

it reads the same as writed, no data loss there.... , I readed that VB6 converts too in file i/O? well, atleast with the 1252 I can certificate there is no data-loss.....

But, I don't uses really the upper byte. And I don't remember to see double size files!, I thinks it is just saving the low byte of each character...

Now, I don't know if all others codepages will allow to do the same thing, to use the 256 values without dataloss. ¿Anyone tested that on file i/O?

now, we are not giving this way the support to MULTILANGUAGE, just that it won't fail on my market region.

So, I ends with lots unanswer questions...., not related to sck.send

[flyguille]

ok, something escape my knowledge, somebody told, VB6 uses UCS-2 .... for my knowledge UCS2=UTF16.... but, i now checking UCS2 rules... found

"UCS-2 is a fixed-width encoding; each encoded code point will take exactly 2 bytes. As a SMS message is transmitted in 140 octets, a message which is encoded in UCS-2 has a maximum of 70 characters (really, code points): (140*8) / (2*8) = 70."

So, is true to say, there is no possibility in VB6 of 32bits special chars encoded in UTF16...

So, this simplifies everything I was worried about.

So, the only thing to fix is really, now , communications with sockets, and storage in files.... because if saving string as binary data in a file, what does really VB? saves only 8bits per char?

[Schmidt]

Happens that most data source are just data entry, from textboxes, so in the natural way of programming, is natural that data information is handled in strings.

No - non-text-data should only temporarily be represented in Strings, instead it should (wherever possible) be handled in "native types" at the storage- or transfer-layers.
Text- or other Input-Boxes or -Ctls then only rendering these native DataTypes in a locale-aware fashion.

E.g. for a floating-point-number, the native storage and transfer-format would be e.g. an IEE 754 Double.
A locale-aware conversion-function, to render such a Double in a VB-TextBox could be: Text1.Text = CStr(TheDouble)
A locale-aware back-conversion from the VB-TextBox-Content could be: TheDouble = CDbl("0" & Text1.Text)

But again, atleast with the 1252, Western codepage.... it is not DATALOSS about coding encrypt information just XORING the LOW byte of each character, ...

CodePages are a thing of the past (don't use them in an App which targets an international audience).
And again: any decent Encryption- or Decryption-Routines should only accept ByteArray-Parameters - never Strings (for Content and Passwords/Keys).

WRITING a file with that string as BINARY OPEN "filename" as binary , PUT ,, string ..... GET ,, string
it reads the same as writed, no data loss there.... ,

That's also wrong information - since a Put Fnr, , VBStringVar and Get FNr, , VBStringVar will always perform an ANSI-conversion.
And that's not what you want in a "cross-country" Application - if you don't want any conversions with VB-FileFunctions, use ByteArrays.

So, I ends with lots unanswer questions...., not related to sck.send

For your "international purposes" - Winsock.send should only be provided with ByteArrays, to be on the safe side.
I already gave you hints in a previous posting, how to convert VB-Strings into such ByteArrays and vice versa.

... for my knowledge UCS2=UTF16.... but, i now checking UCS2 rules... found

"UCS-2 is a fixed-width encoding; each encoded code point will take exactly 2 bytes. As a SMS message is transmitted in 140 octets, a message which is encoded in UCS-2 has a maximum of 70 characters (really, code points): (140*8) / (2*8) = 70."

So, is true to say, there is no possibility in VB6 of 32bits special chars encoded in UTF16...

No, not at all.
VB-Strings can hold, combine or concat - as well as finally render UTF16-Content perfectly fine:

Option Explicit

Private Declare Function TextOutW& Lib "gdi32" (ByVal hDC&, ByVal x&, ByVal y&, ByVal lpString&, ByVal nCount&)

Private Sub Form_Load()
  AutoRedraw = True
  FontName = "Arial"
  FontSize = 18
  ScaleMode = vbPixels
End Sub

Private Sub Form_Click()
  Cls
  ReDim UTF16(0 To 3) As Byte

  'e.g. an emoticon: https://apps.timwhitlock.info/unicode/inspect/hex/1F601
  UTF16(0) = &H3D: UTF16(1) = &HD8: UTF16(2) = &H1: UTF16(3) = &HDE
  
'  'or another CodePoint from a different range: http://www.fileformat.info/info/unicode/char/24F5C/index.htm
'  UTF16(0) = &H53: UTF16(1) = &HD8: UTF16(2) = &H5C: UTF16(3) = &HDF
  
  Dim S  As String
      S = UTF16 'convert the above 4 Bytes without any conversions into a VB-String
 
  Print " Len(S):", Len(S)
  Print " LenB(S):", LenB(S)
  Print " UTF16Char:",: TextOutW hDC, CurrentX, CurrentY, StrPtr(S), Len(S)
End Sub

So, the only thing to fix is really, now , communications with sockets, and storage in files.... because if saving string as binary data in a file, what does really VB? saves only 8bits per char?

As already said:
- safe (unchanging) transfers over sockets
- safe (unchanging) File-Read/Writes
- safe (unsurprising) Encryption/Decryption
- safe (unsurprising) Compression/Decompression
...should all be done with ByteArrays (ByteArray-Arguments in the appropriate Function-Calls).

That's all you need to remeber as "a rule".

The rest is only: "How to write proper conversion-functions" -
and that's the only thing we should really talk about.

Olaf

[flyguille]

just to close this topic.

If I want the string passed exactly as it is in 16bits per chat.... is to say the string not converted to anything else.

it will be on transmission

dim b() as bytes

b() = StrConv(StringBuffer.get, vbWide)
sck.send b()

and then for receiving it.

din b() as byte

sock.receive b()
StringBuffer.add = StrConv(b(), vbWide)

that is all?

[passel]

You should be able to copy strings to byte arrays and byte arrays to strings directly, I believe. This will preserve the two-bytes per character in the byte array.
dim myString as String
dim b() as byte

b = myString

sck.send b


or

sock.receive b

myString = b

[flyguille]

You should be able to copy strings to byte arrays and byte arrays to strings directly, I believe. This will preserve the two-bytes per character in the byte array.
dim myString as String
dim b() as byte

b = myString

sck.send b


or

sock.receive b

myString = b

nice, I will test it. Never did that.

[flyguille]

A final question. As I needs to do a smooth transition from old versions to new version.

Is there a way to detect if a it is receiving a string (old version) vs a byte array (new version)?, I needs that as a way to detect if it is talking to an old version or new version... so, kind of do a server version for transition which works two ways.

So, thinking to check if all second byte is zero, most probably is new version...., flag the connetion as new version, there is another way?

[flyguille]

I hope this works, fingers crossed!

Private Sub UpLinkSocket_DataArrival(Index As Integer, ByVal bytesTotal As Long)
Dim Temp As String
Dim b() As Byte, x As Integer

' Protections for DoS attacks.
' Here is not interpreted the received data, allowing that way to maintain balanced service for all customers. Avoiding a single customer to capture 100% the server attention.
On Error GoTo Err

If bytesTotal Then
    ' Require to check the versión which the customer is using, before to process the data received.
    UpLinkSocket(Index).GetData b(), vbArray, MAX_BUFFER_SIZE
ReTest:
    Select Case VersionSlot(Index - 1)
        Case Is = 0
            ' Undetermined version yet.
            ' We expects a large username-only ID command as the first command, just before the kriptology enables for this channel as kriptology keys are custom per each customer.
            VersionSlot(Index - 1) = 2
            If UBound(b()) >= 9 Then
                For x = 1 To 9 Step 2
                    If b(x) <> 0 Then
                        VersionSlot(Index - 1) = 1
                    End If
                Next x
            End If
            GoTo ReTest
        Case Is = 1
            ' Old Version = VT1.
            ' Hope to equivalent of the original sentence: UpLinkSocket(Index).GetData Temp, vbString, MAX_BUFFER_SIZE
            Temp = StrConv(b(), vbUnicode)
        Case Is = 2
            ' New Version = VT2.
            Temp = b()
    End Select
    
    ' Add it to the buffer.
    If InputBuffer(Index - 1).BufLen + Len(Temp) <= MAX_BUFFER_SIZE Then
        InputBuffer(Index - 1).BufAdd Temp
    Else
        ' Buffer overflow.
        ' Must to drop this connection.
        If LoggedSlot(Index - 1) Then
            ' It is from a known customer.
            Call AdmLogging(CustomerNameSlot(Index - 1), "Too many data transmmited (input buffer overflow) dropping connection.")
        Else
            ' It is an anonymous.
            Call AdmLogging("Anonymous connection from " + SlotRemoteIP(Index - 1), "Too many data transmmited (input buffer overflow) dropping connection.")
        End If
        Call DropSlot(Index - 1)
        Exit Sub
    End If
End If

Err:
Resume Err2
Err2:

' It updates the ping time out software layer protocol.
LastUsedSlot(Index - 1) = CurrentTime

End Sub

[flyguille]

Ok, guys, new main server, connected to old customers, no problem!.

Tomorrow, I will try to connect a new customer's server to the main server.



Happy to internationalize it!.

step by step!

[Schmidt]

So, thinking to check if all second byte is zero, most probably is new version....,

Basically right, but please do not check *all* second Bytes -
(because your Bytes are now based on an Unicode-String, which can contain WChars which have both Bytes set):

Take for example the Euro-Sign;

Private Sub Form_Load()
Dim S As String
    S = "€"
Dim B() As Byte
    B = S: Debug.Print B(0), B(1)
End Sub

Prints out:

 172           32

Though if your "Message-Buffer-Protocol" starts with a few "Header-Chars" in the low ANSI-range, as e.g.:

Message-Length: 12345 
vbCrLf
<Your-Data>

Then it would be entirely sufficent, to check just the first two Bytes (for 77, 0) ... (or perhaps only the second Byte for 0).

If B(1) = 0 Then 'it's the new protocol
  S = B
Else 'the old Protocol (requiring conversion from the received ANSI-Bytes)
  S = StrConv(B, vbUnicode)
End If

Before you introduce that, I'd like to make a suggestion which makes it far less likely to ever change the Base-Protocol again:

Why not doing it properly - and transfer a "serialized Container, which can take up basically any type without conversion"?
What I'm talking about, is a (free-standing, without any DB-connection) ADO-Recordset:
- here you can define Fields for (Unicode-)Strings, Doubles, Dates, Longs - and also ByteArray-Blobs (for Image- or FileContent).
- create such an Rs
- add one or more Records to it and fill the Fields
- serialize the Rs into a ByteArray
- send that (with 4 preceding bytes before the "Rs-ByteArray", to describe the length of the serialized Rs-Buffer)
- maybe add (at the end of the whole Buffer) a few additional "unique Splitter-Bytes" (8 of them should be sufficient to "search and resync again" in case you got lost in a contiguous ByteStream at the receiving end).

At the receiving end you would do the above in reverse.
With that in place, you can enhance your protocol without breaking it
(simply adding new Fields to the Rs).

Olaf

[flyguille]

Basically right, but please do not check *all* second Bytes -
(because your Bytes are now based on an Unicode-String, which can contain WChars which have both Bytes set):

Take for example the Euro-Sign;

Private Sub Form_Load()
Dim S As String
    S = "€"
Dim B() As Byte
    B = S: Debug.Print B(0), B(1)
End Sub

Prints out:

 172           32

Though if your "Message-Buffer-Protocol" starts with a few "Header-Chars" in the low ANSI-range, as e.g.:

Message-Length: 12345 
vbCrLf
<Your-Data>

Then it would be entirely sufficent, to check just the first two Bytes (for 77, 0) ... (or perhaps only the second Byte for 0).

If B(1) = 0 Then 'it's the new protocol
  S = B
Else 'the old Protocol (requiring conversion from the received ANSI-Bytes)
  S = StrConv(B, vbUnicode)
End If

Before you introduce that, I'd like to make a suggestion which makes it far less likely to ever change the Base-Protocol again:

Why not doing it properly - and transfer a "serialized Container, which can take up basically any type without conversion"?
What I'm talking about, is a (free-standing, without any DB-connection) ADO-Recordset:
- here you can define Fields for (Unicode-)Strings, Doubles, Dates, Longs - and also ByteArray-Blobs (for Image- or FileContent).
- create such an Rs
- add one or more Records to it and fill the Fields
- serialize the Rs into a ByteArray
- send that (with 4 preceding bytes before the "Rs-ByteArray", to describe the length of the serialized Rs-Buffer)
- maybe add (at the end of the whole Buffer) a few additional "unique Splitter-Bytes" (8 of them should be sufficient to "search and resync again" in case you got lost in a contiguous ByteStream at the receiving end).

At the receiving end you would do the above in reverse.
With that in place, you can enhance your protocol without breaking it
(simply adding new Fields to the Rs).

Olaf

yes, I care about to check only the header which is internal and won't use the second byte. So only tested the first 5 characters and that is enough, if it 16bits, it will test only sync headers, if it is 8bits, it will test part of the sync header plus the LOGIN command ID plus another sync bytes, impossible to get human data entry on that first command that short test, the only one command in plain text, and once it knows which version is, it will remember, no re-test.

This method is perfectly capable of transmitting binary files, video, on just strings, was perfectly capable before discovering that a japanese won't be connecting because the ANSI codepage, and I am assuming that VB6 doesn't any conversion internally when handling strings.

The only worriying for me was the internationalization. Is to say a japanase, or chinese be able to connect, the protocol is already multi-purpose multi-capable.

Ok, now it will double the traffic, that sucks, but I am buying a DELL XEON 1225v5 for it plus others tasks , currently using a A64 s939 with 2GB in a win server 2003 puaaaaaaaaajjjjjj. and it can do 50 customers already from all latin american, atleast 2000 PCs as average serviced sinmultaneosly in two leyers of servers!!!!.

[flyguille]

Another thing I am worrying for internationalization is that it does the same for file i/O, by exmaple I am not using microsoft databaes for it, I am using kripted binary files as databases, it is really fast.

But worrying that if not codepage 1252, reading the file =! what was saved.... if using only the ANSI 0-255, it can't fail or will be safe about writing/reading file i/O?

like

open as binary: get / put #x,, StringVar that kind of code? right?


Thinking better, saving as 8bits, will be in the same problem, not be able to store east language characters..... sooo, more work to do. Step by step.

[Schmidt]

Another thing I am worrying for internationalization is that it does the same for file i/O,
But worrying that if not codepage 1252, reading the file =! what was saved....

There is an Unicode-Encoding which (for most "Latin Languages") is nearly as efficient as an ANSI.encoding;
UTF8.

Write Encoder- and Decoder-functions for it:
Function VBStringToUTF8(S As String) As Byte()
and
Function UTF8ToVBString(B() As Byte) As String

and read/write your files "As Binary" transferring only these UTF8-ByteArrays.

A forum-search will show you several existing Implementations for the above.

An easier way would be, to use a Database of course (there exist some, which support
strong encryption).

Olaf

[flyguille]

There is an Uicode-Encoding which (for most "Latin Languages") is nearly as efficient as an ANSI.encoding;
UTF8.

Write Encoder- and Decoder-functions for it:
Function VBStringToUTF8(S As String) As Byte()
and
Function UTF8ToVBString(B() As Byte) As String

and read/write your files "As Binary" transferring only these UTF8-ByteArrays.

A forum-search will show you several existing Implementations for the above.

An easier way would be, to use a Database of course (there exist some, which support
strong encryption).

Olaf

most probably I will just double the size, 16bits, to file, and that will be all, caring to automatically convert existing databases, is the way to go. After that, then it is just matter if the CHAT controls (text input , text output), which simulates the old mIRC days, but very cute, if it can handle the buffer too in 16bits... for the rest, names in normal native textboxes, etc., it will already handle it correctly I assume that.

[flyguille]

well the new customer's server connects to the main server, 16bits now, so it is a close topic, thanks guys for the help.

[flyguille]

no, it falls in troubles.... simple b() = string, didn't copy it as it is. ooor

a chr$(128) by example, is translated in the VBString as another code than 0x80 0x00...,

so a simple chr$() and then asc() is converting forward and back from usc2 to ansi code?.....??? that is it?

the problem is the detection if it is 8bit or 16bits trnasmission.

and the outer header is encoding in chars the lenght of the package, there is not guaranteee the lenght, so, if it is >=128 the first package, bye bye detection.

my idea of smooth transition is just utopic.

[Schmidt]

no, it falls in troubles.... simple b() = string, didn't copy it as it is. ooor

a chr$(128) by example, is translated in the VBString as another code than 0x80 0x00...,

so a simple chr$() and then asc() is converting forward and back from usc2 to ansi code?.....??? that is it?

Did you not read, what I wrote in #24?
I've explained it there already...

the problem is the detection if it is 8bit or 16bits trnasmission.

and the outer header is encoding in chars the lenght of the package, there is not guaranteee the lenght, so, if it is >=128 the first package, bye bye detection.

If you mean with "length-encoding in Chars", that your length-descriptor is using simple String-Digits -
e.g. produced by a concat like:

  S = LenB(S) & ":" & S

... or something like that, then you are in no trouble, because the WChars of Number-Digits are in the low ANSI-range.
If you want to make sure, you can simply check any Char you are interested in per:
AscW("1") for example - if it is below 256, then you will definitely see a Zero at the "uneven" Byte-Indexes.

Olaf

[flyguille]

Nah, I handled the buffer in binary way. I am a programmer from the era of MSX-BASIC (the competitor of C64) then machine-code, later assembler, so basically I used the VB language as the old days, now I discover that chr$() or asc() is not what used to be.

Anyway, I will solve the detection the hard way. To process all the package and check which one unpacks correctly after the dekription.



btw: I hope chr$() asc() is doing conversion just with a lookup table, otherway I am wasting lots of cpu ticks everywhere. Well, not everywhere but kripto is intense usage of that.

[flyguille]

no, I can't reuse the same protocol, one thing is to allow it to transport eastern character, but another thing is to let the west <-> east communicate.

Because if a simple chr$(129) by example is map to a given 16bits value but then in the other wide with different region that 16bits value is not map back to that original chr$(129), in way unpack, dekription works, it won't work. at all.

So there is two goals.

As somebody said, I needs to handle everything in byte array, change the way it packs and kript, this will mean no smooth transition for customers. I hate it!.

[flyguille]

Is there any help from VB6 for concatenating byte arrays?

like

redim preserve b( 0 to ubound(b)+addedsize)

then

for x=1 to added_length
b(write cursor) = ascw(mid$(raw,x,1)) mod 256

b(write cursor+1) = ascw(mid$(raw,x,1)) / 256
next x

...............................

slow....


or

redim preserver

varptr(b())

then

copymem .... uffffffffffffffffff..

Anyone already has a byte array based buffer? with .ADD , .DELETE .CLEAR .MID() methods.?

[flyguille]

NEVER MIND, I already got it in the video transmitter, just not in a nice enclosed Class... have work to do!.