List Array performing slowly

Hi All,

I'm loading in some data from an Oracle DB and finding it's reaaaaaally slow. It's about 65,000 customers I'm loading in and it takes about 30 mins to run.

I've tried a few different array types and settled on List (I read that the strongly typed collection was one of the best), but it's bloody awful.

Any ideas what I'm doing wrong?

Code:

---

    class Customer : IComparable
    {
        //create a constructor
        public Customer()
        {
 
        }
 
        public string Code;
        public string Name;
        public string GenericPriceListName;
        public string GenericPriceListType;
        public string PriceList;
        public string DiscountList;
        public string ExceptionList;
        public string PromotionList;
 
 
        // implement IComparable interface
        public int CompareTo(object obj)
        {
            if (obj is Customer)
            {
                return this.Code.CompareTo((obj as Customer).Code);  // compare user names
            }
            throw new ArgumentException("Object is not a Customer");
        }
 
    }
   
 class Program
 {
 
        private static List<Customer> GetCustomerList(OleDbConnection myOleDbConnection)
        {
            OleDbCommand myOleDbCommand;
            OleDbDataReader myOleDbDataReader;
            List<Customer> listOfCustomers = new List<Customer>();

            myOleDbCommand = myOleDbConnection.CreateCommand();

            StringBuilder sqlStatement = new StringBuilder();
            sqlStatement.Append("Select * from Customers");


            myOleDbCommand.CommandText = sqlStatement.ToString();


            myOleDbDataReader = myOleDbCommand.ExecuteReader();


           

            while (myOleDbDataReader.Read())
            {

                Customer currentCustomer = new Customer();
                currentCustomer.Code = myOleDbDataReader.GetString(0);
                currentCustomer.Name = myOleDbDataReader.GetString(1);
                currentCustomer.GenericPriceListName = myOleDbDataReader.GetString(2);
                currentCustomer.GenericPriceListType = myOleDbDataReader.GetString(3);

                listOfCustomers.Add(currentCustomer);

                Console.WriteLine(string.Format("Extracting account {0}", currentCustomer.Name));
            }

            myOleDbDataReader.Close();
            myOleDbCommand.Dispose();


            return listOfCustomers;
        }
}

---

I ran a test of your code, though not against an Oracble DB (I used a quick and dirty SQLite database) and my test completed in less than 3 seconds for all 65,000 customers.

Now, the problem could be your DB backend, but I doubt it, it's more likely the Console.WriteLine statement which is your bottleneck.

If it's there for debugging, comment it out for a true test of performance.
If it's there for progress reporting, setup something a little more efficient, like using a Timestamp to only output the progress every few seconds, instead of every record. UI updating is very expensive, even to the console.

Regards,

- Aaron.

Quote:

---

Originally Posted by Flustor

Any ideas what I'm doing wrong?

---

Quote:

---

Originally Posted by Flustor

It's about 65,000 customers I'm loading in

---

Why do you need to load 65000 customers at once?

If it's for human consumption, there's no way to handle that amount of information at once, try paging techniques to only be loading a few at a time.
If it's for machine consumption, do the work in batches, or maybe try doing it in the database if possible?