What you're able to do is create a Dictionary that would store the index of the item clicked in ListBox1 as the key and an array of indices of the items added in ListBox2 as the Value. In the SelectedIndexChanged event for ListBox1, you would check if the SelectedItems count is more or less than the Dictionary count. If it's more then you know that the user is wanting to add values to ListBox2. If it's less then you know that the user is removing values from ListBox2. Here is a quick (free-typed) example:
Code:
Private indices As Dictionary(Of Integer, Integer()) = New Dictionary(Of Integer, Integer())
Private Sub ListBox1_SelectedIndexChanged(ByVal sender As Object, ByVal e As EventArgs) Handles ListBox1.SelectedIndexChanged
    If indicies.Count < ListBox1.SelectedItems.Count Then
        'Add values to ListBox2
        Dim fooValues() As String = {"Foo1", "Foo2", "Foo3"}
        indicies.Add(ListBox1.SelectedIndex, Enumerable.Range(ListBox2.Items.Count, ListBox2.Items.Count + fooValues.Length).ToArray())
        ListBox2.Items.AddRange(fooValues)
    Else
        'Remove values from ListBox2
        For Each index As Integer In indices.Values(ListBox1.SelectedIndex)
            ListBox2.Items.RemoveAt(index)
        Next
        indices.Remove(ListBox1.SelectedIndex)
    End If
End Sub