The next best thing would be a 1D array which would sort the X values with the Index.
--------------------------------------------
IE: 4,5 4,7 2,1 2,3 3,8 would make an array that looked like
Index - Value
1 - null
2 - 1 3
3- 8
4- 5 7
---------------------------------

But that would still end up being quite slow if multiple Y values ended up being in the same index.

Anyone have some more ideas?