Ok, I pointed the problem out. I have a ListView.ListViewItemSorter set on my control, and it is active when inserting data.
I now set it to nothing when inserting data and I set it back when finished.

I have to wait 6s to sort my listview now. Does Anyone know what algorithm MS uses in their Sort() Method, I think it's in n², so maybe it should be interesting if someone had implemented one in n.log(n).