For some reason I could not think of how to do this and when I searched, I could not find any easy code that was not API related or some such silliness.

To select the item that was right clicked on, use the following:

Code:
 Private Sub ListBox1_MouseUp(ByVal sender As Object, ByVal e As System.Windows.Forms.MouseEventArgs) Handles ListBox1.MouseUp
        If e.Button = Windows.Forms.MouseButtons.Right Then
            ListBox1.SelectedItem = ListBox1.Items(ListBox1.IndexFromPoint(e.Location))
        End If
    End Sub