Loading large amount of data estimate 1,000,000 of records into a listview

[CoMMiE]

hi

Lets say in a situation where i need to load all the records in a table which is estimate to be around 1,000,000 records excluding linkage to other table

what is the best way to load all the data into the listview?

if have thought of a few ways like selecting all the records and download into a datatable, but using this way will only kill the client machine.

Does anyone knows or suggest a better way to handle this kind of situation?


Thanks!!

[cim3]

I certainly wouldn't bring that many records across to a client app. There is no way a client is going to be searching through a million records in a listview.
I would allow my clients to search the database for the records they want or closely want and then display these ina listview. The Database engine is far more powerful and therefore quicker.

[CoMMiE]

Thanks for the reply

but what if you are in a situation where you are force to load that one million record into the listview and there will be no filter or any search ?

[powdir]

if you are being 'forced' to populate a listview with 1 million records then there is something seriously wrong with the design. I cant see what purpose 1 million displayed records would serve. What about having the client execute a stored proc which would execute this data transfer at the server side?

[hellswraith]

Agreed, you should NEVER be forced to do such a thing. It is just plain stupid. Now with that said, you would definately want the retrieval to be on its own thread. So multi-thread this application.

[VictorB212]

Think of the memory that 1 million listviewitems would require. I don't think that many computers could handle it (without diving into virtual memory, which would take forever). Everyone here is almost certainly correct: the design criteria are flawed. However, one way you could implement this is making the user think that the listview contains 1 million entries, but load them as he scrolls. I believe there are controls that already do this, but I forget what the term is for this.

[Pirate]

If you still insist to do it that ugly way , then use API to populate your listview . It's makes big difference and faster too . Haven't got a clue on which API Function you should use but I thought this might help a bit .

[CoMMiE]

yes,i have also thought about loading the data while the user scrolls,for example like loading the first 100,then only start to load the rest while the user scrolls down.

i have tried to search on the web and also this forum but so far i have not found any solution on it yet


Thanks

[DevGrp]

Dude, whoever is forcing you to load 1 million records is either retarded or has a lot of time on their hands, because its gonna take a while to scroll thru all those records.

[CoMMiE]

sometimes is is not that i want to load that much of record into the listview, but in some circumstances it is just unavoidable

[hellswraith]

Exactly when is it unavoidable? There is NO circumstance that you can't avoid it someway or another. If there is, I want to hear it. Please do tell.

[BodwadUK]

A list view has a maximum value of integer doesnt it???

It cant load 1,000,000 records, try breaking it down by adding next previous buttons and splitting it into 1000's or something

[powdir]

a listview (or any other control) would fall apart if it was expected to contain 1mill items. The CLR would have 1 mill object references to manage and the user would be overwhelmed.

Paging is one answer but that would require a search, maybe by sequence number, or an alpha-index column to bring down the data in manageable chunks, but CoMMie states that there should be NO filter or search.

The MOST i have ever seen a listview contain is some 60000 items and even then it was ungainly and derided by the users. The app was reworked.

IF data transfer is your aim, let the DB handle it because 1mill records is not for the eyes of us mere mortals

[Pirate]

This is just insanity .

[nemaroller]

but what if you are in a situation where you are force to load that one million record into the listview and there will be no filter or any search ?

Look at it like this... when you search for CAT in Google, do you get a listview of 23,000,000 web pages to click?

No... you get the first 10... then you click for the next page of results...

In a similiar way, you can simply use an indexer and retrieve 100 or 500 or ahem, 1000 records at a time. In no circumstance, should you return 1 million records...

I don't know what database your communicating with but many support paging, and you should definitely implement it because you HAVE NO other choice. Failing support of paging, have a variable hold the last record retrieved, and use that as the basis for the next 1000 or so records you will return.

[CoMMiE]

well

thanks for all the reply and suggestion it has been really helpfull, i guess the best way to go now is to do paging whithout loading all the records.

[MrGTI]

Agreed.