Results 1 to 3 of 3

Thread: Handling Large Amounts of Temporary Data

  1. #1

    Thread Starter
    Hyperactive Member
    Join Date
    Sep 2004
    Location
    Manchester
    Posts
    266

    Post Handling Large Amounts of Temporary Data

    I'm looking to find the most efficient way to handle (what I could consider) to be large amounts of temporary data in VB.NET.

    I'm looking to store 300,000 strings of data, each string can have up to around 500 characters within it.
    If my calculations are correct (they probably are not!), this would require RAM of around 143 megabytes (worst case scenario). I'm a little uneasy about storing this amount of data in RAM - would you be concerned?

    Is it perhaps better to store this as perm data in a database and make queries to that database when/if I need to query the data further?
    What would you generally work with as a best practice limit on temporary data storage?

    Thanks in advance!
    Jordan

  2. #2
    PowerPoster
    Join Date
    Mar 2002
    Location
    UK
    Posts
    4,780

    Re: Handling Large Amounts of Temporary Data

    A 32bit process can use up to 2GB before you get into trouble (somewhere before actually), so is 200mb going to kill it? No, not likely, I have used way more than that in certain apps. The issue will be what you are planning on doing with it, does it all need to be used at any time, is this for a partial selection, for a lookup?

  3. #3
    Smooth Moperator techgnome's Avatar
    Join Date
    May 2002
    Posts
    34,532

    Re: Handling Large Amounts of Temporary Data

    I agree... it really depends on what you're planning to do with it... I deal with that kind of volume of that data on a daily basis.... actually it's a mere piffle compared to the over all amount I deal with, but I never need all of it, or deal with all of it all at once like that... usually I'll only load just the data I need.

    -tg
    * I don't respond to private (PM) requests for help. It's not conducive to the general learning of others.*
    * I also don't respond to friend requests. Save a few bits and don't bother. I'll just end up rejecting anyways.*
    * How to get EFFECTIVE help: The Hitchhiker's Guide to Getting Help at VBF - Removing eels from your hovercraft *
    * How to Use Parameters * Create Disconnected ADO Recordset Clones * Set your VB6 ActiveX Compatibility * Get rid of those pesky VB Line Numbers * I swear I saved my data, where'd it run off to??? *

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width