I haven't the foggiest on the SendMessage but I did find this while reading up on LVM_SORTITEMS

The comparison function must return a negative value if the first item should precede the second, a positive value if the first item should follow the second, or zero if the two items are equivalent

I noticed you returned 0, 1, and 2

Will that help?