hey all
The objective is to read a massive textfile (200+mb)
each line is a "entry"
then output the totals of each entry
rob
rob
rhaps
tim
output
rob =2
rhaps =1
tim =1
the code i came up with is here
c# Code:
System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch(); Dictionary<String,int> Count = new Dictionary<string,int>(); string path = @"C:\Documents and Settings\Rob\Desktop\test.txt"; int num = 0; sw.Reset(); sw.Start(); using (StreamReader sr = new StreamReader(path)) { string line; while ((line = sr.ReadLine()) != null) { if (Count.ContainsKey(line)) { Count[line]++; } else { Count.Add(line, 0); } num++; } } foreach (KeyValuePair<string, int> kvp in Count) { Console.WriteLine(kvp.Key+": "+kvp.Value.ToString()); } sw.Stop(); long time = sw.ElapsedMilliseconds; Console.WriteLine("Time = " + time + " milliseconds."); Console.WriteLine("Total Rows: " + num.ToString()); Console.WriteLine("lines per ms: "+ (num / time).ToString());
I was thinking of somesort of parrellel while but im not sure how to do that or if it even exists and if it did exist would it work with the stream reader
the other idea was to load it as a DB and then run a sql command to get the count or something but im not sure if that would be faster or not?
any ideas are greatly appreshated
thanks




Reply With Quote