Okay, so let me see if I have this problem right...

OP is trying to keep selstart and actual byte position of file equal to each other...

Okay, if and only if each new line is equal to vbNewLine or vbCrLf meaning two bytes then could you not get the line count of current cursors position and multiply it by two?

Code:
Private Declare Function SendMessage Lib "user32" Alias "SendMessageA" (ByVal hwnd As Long, ByVal wMsg As Long, ByVal wParam As Long, lParam As Any) As Long

Private Const WM_USER = &H400
Private Const EM_EXLINEFROMCHAR = (WM_USER + 54)

Private SomeSub()
Dim CursorPosition As Long, RowCount As Long

CursorPosition = RTB.SelStart

RowCount = SendMessage(RTB.hwnd, EM_EXLINEFROMCHAR, 0, ByVal CursorPosition) + 1

MsgBox "Cursor Is On Line " & RowCount

End Sub

Good Luck