now to make this faster?

**mrdutchie** · Feb 11th, 2004, 05:46 PM

Ok, I have a testfile with 40.000 lines

I want to add all of these in a control.

I tried Listview and it's very slow.. takes about 50 seconds to load
I did use the event listview1.beginupdate and .endupdate
so that speeds it up a little bit to 14 seconds , but still to slow..

Here is the source, and I hope someone can help me.. cause I see the newsreaders doing it within 1-2 seconds

Dim s As Array
Dim w As String
Dim f As String = "C:\newsgroups.txt"
Dim stream_reader As New IO.StreamReader(f)
Dim he As Long
Dim tmpImportProgress As Double
Dim splitlist(2) As String
Dim aas As String
Dim z As Long
Dim zz As Long

z = GetTotalRows()
he = z
z = z - 1

s = (Split(stream_reader.ReadToEnd, vbCrLf))

stream_reader.Close()
Dim c1, c2, c3 As String
Dim a() As String = s(0).Split(Chr(10))
ListView1.BeginUpdate()
Application.DoEvents()

For x = 1 To a.Length - 2
splitlist = a(x).Split("<PKA>")
c1 = splitlist(0)
c2 = Mid(splitlist(1), 5, Len(splitlist(1)) - 4)
c3 = Mid(splitlist(2), 5, Len(splitlist(2)) - 4)

Dim listItem1 As New ListViewItem(splitlist(0))
listItem1.SubItems.Add(c3)
listItem1.SubItems.Add(c2)
listItem1.SubItems.Add(c2 - c3)

fakeListView.Items.Add(listItem1)
ListView1.Items.Add(listItem1)
tmpImportProgress += (100 / he)
clsImportProgress = CInt(tmpImportProgress)
RaiseEvent ImportProgress()
Next

Application.DoEvents()
For zz = 0 To he - 1 '9
Dim listItem As New ListViewItem(fakeListView.Items(zz).SubItems(0).Text)
listItem.SubItems.Add(fakeListView.Items(zz).SubItems(1).Text)
listItem.SubItems.Add(fakeListView.Items(zz).SubItems(2).Text)
listItem.SubItems.Add(fakeListView.Items(zz).SubItems(2).Text - fakeListView.Items(zz).SubItems(1).Text)
ListView1.Items.Add(listItem)
Next
ListView1.EndUpdate()

and here is a part from the newsgroups.txt
basicly <PKA> is for seperating to different columns.

So

Newsgroup Endnumber Startnumber

24hoursupport.helpdesk<PKA>922050<PKA>739010
3b<PKA>5174<PKA>4224
3b.config<PKA>29105<PKA>27724
3b.misc<PKA>28002<PKA>26610
3b.tech<PKA>12728<PKA>11546
3b.test<PKA>95298<PKA>88012

Thanks so much

**nswan** · Feb 11th, 2004, 06:07 PM

are you sure you really need to load the listview with 40,000 rows? That's an awful lot of rows for a user to be looking through!

**Kasracer** · Feb 11th, 2004, 06:48 PM

If I were you, I'd add an option to allow the user to connect to the newsgroup server they specified and download the list of all newsgroups so each time it is loaded it doesn't waste their bandwidth (this is what I assumed you did). Then I would take the listbox and I would serialize the class (I think you can). That way, when the program exists for the first time, it'll basically copy the class into a binary file. Simply serializing it back in, I would imagine would be alot faster than reading it in from a textfile and loading it into a listbox.

If you post the full code you use (you may not want to and some people, like me, may or may not have the time to look though it, especially when you don't use the [code][/code] tags), we may be able to make it alot quicker just by changing some simple syntax.

Originally posted by nswan
are you sure you really need to load the listview with 40,000 rows? That's an awful lot of rows for a user to be looking through!

He is making a newsgroup browsing program, so I assume you never used newsgroups? Most, if not all newsgroup program lists the thousands and thousands of groups in one large listbox.

**mrdutchie** · Feb 11th, 2004, 06:54 PM

Well I downloaded the groups earlier..
that's that C:\newsgroups.txt file
I am just reading from it.

I did tried in a listview and only showing 10 items.
and then when you go up or down it Adds one or removed 1.
that works, but I still need to load all 40.000 rows in a Fakecontrol
and that's where the problem is.

**nswan** · Feb 11th, 2004, 06:58 PM

can't you read the text file into a dataset?

**mrdutchie** · Feb 11th, 2004, 07:04 PM

Tried that... but the 1st 1000 are fast but then it starts crawling...
and takes over 8 minutes

**Edneeis** · Feb 11th, 2004, 07:16 PM

Why not just read chunks of data from the file as needed? Then you don't have to every read in more than they want to see.

You would have to read x amount of bytes from the file then convert them to data or items and show them then remember the location you last read in the file. Then if you need more data just start reading from the last location on in x amount again.

I think the term is LazyLoading when you only load on demand.

Also binary serialization MAY be faster then storing the items as text and parsing them. Where did the format of the text file come from? If you can change it then you could switch to serialization.

**Kasracer** · Feb 11th, 2004, 07:22 PM

Why not load everything with a thread? Then your application won't be frozen and a user can scroll through your list as the file is being read into it. They may not even notice it isn't done loading 30 seconds later.

Or, if you wanted to, you could create a DLL in C# or C++ .NET, off load everything with a thread in the DLL and have the DLL load up the listbox. From what I've seen, it looks like C# has faster IO capabilities (I still don't know why that is) so just using C# may make it faster, or if you're adventurous, you could use unmanaged and managed C++ which will probably be even quicker.

I still think serilization may be a bit quicker than loading from a textbox though.

If you want, you could even ask the XNews developers how they do it with XNews, because when I open XNews, it fills in the listbox instantanious with more than 50,000 newsgroups (or I think it's about 50k, never counted).

**mrdutchie** · Feb 11th, 2004, 10:22 PM

Yes, but then how do they manage it, that you can still Sort those fields within a second..
don't think that work, if you load only a part of it..

Originally posted by Edneeis
Why not just read chunks of data from the file as needed? Then you don't have to every read in more than they want to see.

You would have to read x amount of bytes from the file then convert them to data or items and show them then remember the location you last read in the file. Then if you need more data just start reading from the last location on in x amount again.

I think the term is LazyLoading when you only load on demand.

Also binary serialization MAY be faster then storing the items as text and parsing them. Where did the format of the text file come from? If you can change it then you could switch to serialization.

**VSNightblade** · Feb 12th, 2004, 04:48 AM

Errr... why the hell do you want 40.000 lines in a listbox? Of course thats slow =\

Maybe you can split them up into multiple files and allow the user to select what they want to see. I cant imagine anyone checking out 40k items like its no big deal.

**Kasracer** · Feb 12th, 2004, 03:20 PM

I am checking out my XNews files and it appears that XNews saves all the servers in one textfile. Each server is seperates by a '!' and a return so it looks like

Code:

server.news.one!
server.news.two!
server.news.three!

XNews reads the entire file in on load and puts it into the listbox in a pretty much instantanious matter (I have 67797 servers in my XNews text file and it loads them instantly).

My guess is that XNews is either written in C or C++ and uses the Win32 API. I don't think VB.NET can even come close to the speed.

My advice would be to do a similar thing like XNews has done in saving things and reading them in. Then make a DLL that has a function which populates the listbox in a seperate thread. The function in the DLL accepts the listbox by reference to cut on the overhead and the DLL would be written in managed/unmanaged C++ (Mix both. Use unmanaged for IO and managed for throwing the items into the listbox).

If you don't know C++, you could try C# (which supposedly has faster IO access). If you don't know either, then you may want to try serialization of the listbox.

I don't think you really have many options here, VB isn't really made to be very fast in execution time but in development time.

Originally posted by VSNightblade
Errr... why the hell do you want 40.000 lines in a listbox? Of course thats slow =\

Maybe you can split them up into multiple files and allow the user to select what they want to see. I cant imagine anyone checking out 40k items like its no big deal.

Like mentioned earlier, this is a newsgroup program and the good programs list all 40k+ groups in one large listbox. XNews can do it instantly, as can most others so it really shouldn't be very slow, if at all.

If XNews didn't list all my servers in one list, I'd go freakin crazy.

**Edneeis** · Feb 12th, 2004, 03:39 PM

Post this 40K line text file so there is test data.

kasracer I think it's funny that in like every other post you meantion how C# is faster at IO operations, even though the difference is miniscule. I don't mean that in any sarcastic way either I actually do find it amusing and I'd be let down if you stopped. Either that I just happen to read about 5 posts where you mentioned it.

**Kasracer** · Feb 12th, 2004, 03:55 PM

Originally posted by Edneeis
kasracer I think it's funny that in like every other post you meantion how C# is faster at IO operations, even though the difference is miniscule. I don't mean that in any sarcastic way either I actually do find it amusing and I'd be let down if you stopped. Either that I just happen to read about 5 posts where you mentioned it.

I kept repeating myself for some reason in this thread, but it is true.

**BenFinkel** · Feb 12th, 2004, 03:58 PM

Dutchie,

I'd read them into an Access database file. Once it's in a database your speed issues should disappear.

Does anyone disagree with me on that one? .NET data controls should allow you to read many more than 40,000 records without any speed issues.

--Ben

**Edneeis** · Feb 12th, 2004, 04:45 PM

I disagree with you. Also the Listview (that is what he said he is using) doesn't support databinding. The main bootleneck is the control itself or adding items to it. This is where the hang comes into play especially since it must be done on the UI/main thread. I'd be curious to see if switching to a datagrid would speed things up, but I don't really think it would.

I'm sure the IO itself is not the issue as in a previous thread he mentioned that he first parsed the file to an array and read it in and that creating the array went rather quickly. Its the control and adding in the heirarchy of objects to make the item that is slow and unforunately unavoidable. Switching to C++ or using APIs to fill the Listview may help in showing the items but then you will probably less the OOP of .NET in having actual ListViewItem objects.

Also switching to another control to display MAY speed things up. Try the datagrid or listbox.

**perryk** · Apr 21st, 2004, 08:11 AM

Still fighting with it...
Sorry for the late response, but her it the file
I Zipped it to make is a lot smaller
hmmm even zipped it's still 500K so I can't upload it.

Paste this in a textfile and repeat it a few times..
then you have the same effect.

alt.binaries.vcd.dvd-rip 89265 89248
alt.binaries.movies.dvd-r 393475 393476
alt.binaries.dvd2svcd 8021772 7973304
a.b.dvd2svcd 12 13
fido7.ru.dvd 26752 26571
rec.video.dvd.advocacy 14680 14666
alt.video.dvdr 54416 54271
aus.dvd 314267 313739
alt.dvd.video 15225 15196
rec.video.dvd.tech 51390 51272
alt.video.dvd.authoring 28871 28628
fido7.su.hardw.dvd 3192 3193
fj.comp.dev.dvd 720 721
rec.video.dvd.misc 33344 33305
es.rec.video.dvd 43053 42987
alt.video.dvd 721472 720846
hr.rec.dvd 48817 48474
de.comp.hardware.laufwerke.cd+dvd 25995 25948
intel.technology.dvd 518 519
dk.marked.privat.tv+video.dvd 40430 40320
rec.video.dvd.marketplace 28187 28146
dk.edb.hardware.drev.dvd 28369 28149
alt.video.dvd.complain 5623 5610
pt.rec.media.dvd 6961 6950
alt.video.dvd.software 93424 93162
dk.edb.hardware.drev.cd+dvd 9782 9778
alt.video.dvd.friends-of-joe 2427 2415
alt.video.dvd.non-anamorphic 1116 1116
alt.video.dvd.tech 93587 93385
maus.technik.dvd 2884 2879
creative.products.dvd.encore 25865 25860
rec.video.dvd.titles 84182 84120
fr.rec.son-image.dvd-video.titres 75141 75042
uk.media.dvd 637782 636474
nl.media.dvd 69003 68890
alt.media.dvd.cracked 17057 17018
rec.video.dvd.players 78509 78429
creative.products.dvd.misc 3525 3524
alt.binaries.cd.image.playstation2.dvdiso 18434997 18215119
alt.binaries.dvd 35543568 35005928
alt.binaries.dvd.french 13875100 13733304
alt.binaries.dvd.french.d 37409 37051
alt.binaries.dvd.erotica 7554464 7465326
alt.binaries.dvd.english 1970 1971
alt.binaries.dvd2svcd.repost 358830 358815
alt.binaries.dvd.repost 2333056 2295712
alt.binaries.dvdr 49643977 49074361
alt.binaries.dvd-r 5666973 5609489
alt.binaries.dvd.anime 7349464 7179648
alt.binaries.dvd.music 1242418 1205277
alt.binaries.german.dvd 2 3
alt.binaries.dvd.french.repost 283717 278565
alt.binaries.dvd.music.d 1014 1010
alt.binaries.dvd.german 394948 375830
alt.binaries.dvd.asian 213321 192462
alt.binaries.nl.ftd.dvd 66 67
alt.binaries.dvd.image 895847 810087
alt.binaries.nospam.multimedia.erotica 2259329 2243666
alt.binaries.nospam.multimedia.facials 1096840 1084315
alt.binaries.pictures.multimedia.erotica 95800 95509
telekabel.software.multimedia.sounds 7015 7016
asus.support.english.multimedia 2002 2001
alt.binaries.multimedia.cartoons.looneytunes 1678246 1670889
tnn.multimedia.cdrom 391 392
clinet.list.freebsd-multimedia 1483 1484
alt.binaries.multimedia.debbie-gibson 98948 98925
alt.binaries.multimedia.xena-herc 175841 175134
microsoft.public.usasalesinfo.developer.multimedia 571 563
alt.binaries.multimedia.futurama 2134746 2129138
alt.binaries.multimedia.help 6399 6373
telekabel.software.multimedia 1344 1345
tw.bbs.comp.multimedia 63512 58424
alt.binaries.multimedia 48964861 48535565
microsoft.public.windows.inetexplorer.ie5.programming.multimedia 640 452
alt.binaries.multimedia.erotica.tickling 284121 283698
alt.multimedia 550 540
alt.binaries.multimedia.jessica-alba 1052883 1052471
alt.binaries.vcd 21736256 21701752
alt.binaries.multimedia.erotica.anime 1764214 1760953
alt.binaries.multimedia.erotica.interracial 1119759 1113729
alt.multimedia.tk 467652 461765
cn.bbs.comp.multimedia 5954 5771
alt.binaries.multimedia.erotica.transsexuals 1411989 1384360
jyu.multimedia 286 282
alt.binaries.multimedia.japanese 7899294 7853265
alt.binaries.multimedia.andromeda 1141054 1137190
fr.comp.graphisme.multimedia 35850 29970
microsoft.public.fr.devmultimedia 1471 1097
alt.binaries.multimedia.erotica.discussion 12203 12185
alt.binaries.multimedia.pantyhose 610658 602774
alt.binaries.multimedia.disney 1527042 1466397
comp.sys.amiga.multimedia 1561 1391
fido7.ru.unix.multimedia 975 865
alt.multimedia.emblaze 1743 1642
alt.binaries.multimedia.chris-morris 18248 18228
alt.binaries.multimedia.buffy-v-slayer.repost 2265933 2248340
alt.binaries.multimedia.erotica.rm 125732 125521
alt.binaries.multimedia.vintage-film 7262381 7196161
alt.binaries.multimedia.naturism 317501 315840
alt.binaries.multimedia.erotica.lesbians 1900782 1890792
alt.comp.multimedia 451 440
alt.binaries.vcd.repost 10311588 10303067
alt.binaries.multimedia.erotica.d 287755 287382
francom.multimedia 1632 1557
alt.binaries.multimedia.utilities 4317422 4297507
alt.binaries.multimedia.erotica.voyeurism 867526 853085
alt.binaries.multimedia.buffy-v-slayer 3356998 3309167
alt.binaries.multimedia.erotica.male 5292433 5259920
alt.binaries.multimedia.comedy 12462931 12365909
microsoft.public.win98.multimedia.directx5 3006 2583
alt.binaries.multimedia.queer-as-folk 792783 788658
alt.multimedia.tk.terri-disisto 8920 8806
alt.binaries.multimedia.flonk 164558 164529
alt.binaries.multimedia.cartoons.repost 2078683 2048846
alt.binaries.erotica.vcd 14217513 14179366
alt.binaries.multimedia.babylon5.repost 261461 254095
microsoft.public.multimedia.directx.danimation.programming 1734 1711
alt.binaries.multimedia.darkangel 1101044 1100972
microsoft.public.pocketpc.multimedia 10198 10124
alt.binaries.multimedia.erotica. 1215537 1215470
microsoft.public.multimedia.directx.dshow.activemovie 2103 1849
microsoft.public.windowsme.multimedia 9145 9064
microsoft.public.platformsdk.multimedia 2215 2191
sol.lists.freebsd.multimedia 5519 5203
alt.binaries.multimedia.erotica.asian 11297458 11261623
microsoft.public.multimedia 858 851
alt.binaries.vcd.xxx 5025360 5009187
alt.binaries.multimedia.cartoons 15652142 15513400
alt.binaries.multimedia.sailor-moon 653889 649630
alt.binaries.multimedia.smallville 1273367 1268893
alt.binaries.multimedia.erotic.playboy 1034010 1024372
muc.lists.freebsd.multimedia 4677 4020
alt.binaries.multimedia.startrek 3340411 3271796
alt.binaries.multimedia.erotica.black 3134077 3125408

Thread: now to make this faster?

Thread Tools

Display

now to make this faster?

Posting Permissions