I'm at my wits end! Need help converting bytes to a string.

[Peter Porter]

I've been trying for over a week, but this code is not converting anything! This suppose to give me the names of the desktop items from SysListView32.

SendMessage(vHandle, LVM_GETITEMTEXT, j, vPointer)
Dim vBuffer As Byte() = New Byte(4095) {}
ReadProcessMemory(vProcess, vPointer, Marshal.UnsafeAddrOfPinnedArrayElement(vBuffer, 0), vBuffer.Length, vNumberOfBytesRead)
Dim vText As String = Encoding.Unicode.GetString(vBuffer, 0, CInt(vNumberOfBytesRead))

But I easily get desktop item's icons xy locations with this:

SendMessage(vHandle, LVM_GETITEMPOSITION, j, vPointer.ToInt32())
Dim vPoint As Point() = New Point(0) {}
ReadProcessMemory(vProcess, vPointer, Marshal.UnsafeAddrOfPinnedArrayElement(vPoint, 0), Marshal.SizeOf(GetType(Point)), vNumberOfBytesRead)
Dim IconLocation As String = vPoint(0).ToString()

I've posted various versions of the code here, but the vText string is always empty. The only thing that populates my listview are the item's xy locations. I seriously need help!

[jmcilhinney]

Are you on a 64-bit system and, if so, what is the target platform for your project? A quick search seems to suggest that a 32-bit app on a 64-bit system may have an issue doing this. I'm not sure whether that would be your issue but it's the first thing I saw that seems like it might explain it.

[Peter Porter]

Yes, I've have a 64bit system. My target is 32bit and 64bit systems.

[jmcilhinney]

My target is 32bit and 64bit systems.

I'm not asking what you personally want to run your app on. I'm asking what the Target Platform is set to in the project properties. If it's x86 then that means that it will run in a 32-bit process on 64-bit systems, so that could be your issue. It it's 'Any CPU' and the 'Prefer 32-bit' box is checked then the same. It may be the case that you need to run in a 64-bit process on 64-bit systems to do what you want to do. In that case, you'd need to either target x86 and x64 separately and test x86 on a 32-bit system, or else target 'Any CPU' with 'Prefer 32-bit' not checked. I'm certainly not guaranteeing this as a solution but it's a possibility. The fact that your second code snippet works makes me doubtful though.

[Peter Porter]

I've changed my app's configuration to "Any CPU", and the results are asian characters for each desktop item. I guess my system has major issues. LOL!

[jmcilhinney]

It may be that you are using the wrong encoding. Try using alternatives to Unicode to see whether any of those work.

[Peter Porter]

Yep... It was the encoding! I switched it to UTF7.

Now would this fix that worked for my system make my app incompatible with other systems? Not sure where the asian characters came from.

Update: I see UTF8 also works. Which should I use?

[Sitten Spynne]

The whole thing is a mess, really. I don't mean "you are writing bad code", but what you are doing is really fidgety and prone to failure, since you're poking at undocumented bits of Windows.

.NET has a very specific definition of "a String". That includes how it's laid out in memory, and how the bytes correspond to characters.

C and the Windows API have at least three different definitions of "a String". The Encoding is just one piece of it. It might be a null-terminated string or any other one of the "standard" representations of a String in C.

But I do have an idea. Call it a psychic guess. It could be wrong as heck.

"Unicode" is the stupidest name MS could've picked for the encoding you're using. It's UTF-16. Since MS is an American company, they decided they could take the word for "any text encoding that can represent the agreed-upon international standard character set" and apply it to "a specific encoding". UTF-16 is the Windows default encoding for string data since about the Win2k era.

So it was a pretty good choice to pick that encoding for grabbing "string data" out of the memory space of Windows. "I get an empty string" sort of implies a really common problem with UTF-16 when it's not expected.

The most common kind of string data in C is a null-terminated char[]. That's why the variables tend to start with "sz", that's "string, null-terminated". The String "A" in ASCII, represented as bytes, would be:

0x65, 0x00

So even though there's 1 character in the string, it takes 2 bytes to represent it because null-terminated strings always end with a 0 byte. 0 is not a valid ASCII character in sane contexts so it's a great terminator.

Now, UTF-16 was created in a world with ideas like, "Hey, what if Japanese people wanted to use a computer?" It has to represent ALL of Unicode, which means 1 byte for characters won't cut it. It uses at least two bytes per character. That means even when we're using English text that sticks to plain old ASCII, we get two bytes. That String "A" in UTF-16 might look like:

0x00 0x65 0x00

Now we have a problem. "A" is two bytes in UTF-16. It should be 0x00 and 0x65. If you're handing that off to code that expects "null-terminated ASCII string", it'll see the first 0x00 byte and think, "Oh, there's an empty string". Disaster.

But I don't think that's happening. You're telling .NET you're using UTF-16, so it should be OK with a leading 0x00. But "reading memory" can be tricky.

See, some machines have different ideas about what byte goes first when storing a multi-byte value. This always goes back to hardware information related to the CPU. On some machines, the "little end" of the string goes first. "Little Endian", they call it. On those systems, the null-terminated String "A" would look like:

0x65 0x00 0x00

That might confuse the heck out of .NET. One of the more bonkers things about .NET is it has its OWN concept of byte order, and it doesn't give a flip about the CPU. So maybe it gets these bytes, but it's expecting the other byte order.

Converting bytes to strings can "go wrong" in two ways. One is "you have some bytes that aren't valid in the encoding". If you get that lucky, you get an exception. When you're not lucky, there are "insane" characters but no "invalid" characters. So maybe in my String, 0x65 0x00 corresponds to some weirdo character. I get 1 weird character back when I expected "A".

That sounds like getting a lot of Asian characters, if you ask me.

So to me, the obvious question is, "Are the bytes you're getting something that seems like it can even correspond to a valid String?" The algorithm seems questionable, to be frank.

If I wanted to read an item's name on the desktop, I wouldn't make my buffer size 4096. MAX_PATH is 255, so there's no way a desktop item's name can be much longer than 230 or so characters. 4096 is, frankly, crazy.

But that aside: you are asking for the Bytes at a location in memory. You are assuming that is a UTF-16 String with the .NET byte order. Have you had a peek at the bytes to see if they look reasonable?

It's sort of like you stuck your hand into a hole, hoping to find treasure inside. There could be snakes. Or knives. Or scorpions.

So have you put a debugger on this code, looked at the bytes, and compared them to a UTF-16 listing? If there's nothing in the data you pull that looks remotely like a sane string, I think the answer is: "You got your offset wrong, Windows puts stuff in different places in memory than you have been told."

In short: you're sticking your hand in a bag and pulling out random bytes. They could be in any order, any encoding, or not even the right bytes. So have a look at the raw data and try to decide if you're even in the right zip code.

[Niya]

.NET has a very specific definition of "a String". That includes how it's laid out in memory, and how the bytes correspond to characters.

In general I'd agree what he is doing is risky. But fortunately in this case we know what kind of string it is. It's documented as LPSTR. Been a while since I did stuff like this so I can't recall what the conventions are for LPSTR strings but it should be easy to look up. He can easily adapt his code to correctly read the String as long as the documentation is accurate.

Update: I see UTF8 also works. Which should I use?

Use UTF8. If memory serves, an LPSTR is an ANSI format string, which UTF-8 covers correctly. UTF-7 could possibly cause problems if the string uses character codes above 128.

[Sitten Spynne]

Actually, it's "a pointer to an LPSTR", so reading that memory should involve a level of indirection: you have to read the address, then read the string at that address. The presence of cchTextMax makes me think it may be storing the buffer directly in the structure, though. It's hard to tell.

Either way, could you have a look at my comments in this similar thread? It has more of his code. I'm not as confident that "ReadProcessMemory and WriteProcessMemory aren't needed" is true, but posts like this make me very suspicious we have a 32-bit vs. 64-bit layout issue that will be very difficult to address in a general manner.

I can't test it easily. I tried on my way out of work yesterday and you have to get the handle to the Desktop differently in Windows 10. Such is the journey when doing undocumented things.

[DEXWERX]

That String "A" in UTF-16 might look like:

0x00 0x65 0x00

FYI, a null terminator for a wide string is 2 bytes.

https://msdn.microsoft.com/en-us/lib...(v=vs.85).aspx

[Sitten Spynne]

See? This is why I normally don't comment on API threads >.<

This would all be easier if OP would look at the bytes he has in the debugger to decide if they even look sane for an LVITEM.