The best way to save many inputs ??

[moyalt]

Hello , members.

I have written a vb program sampling inputs from data logger , for now i am saving this inputs into a *.mdb data base file - the fields are value(of the inout i sample),date,time.
my problem is that this data base grows quikly its about 275MB for a day of sampling. do you know better way for sampling this input to a data base that will take less space
this program have to work all year!!

My second question is on a progaram or way to take this inputs and make with them reports and graph in a very fast way?

[HunterMcCray]

Originally posted by moyalt
Hello , members.

I have written a vb program sampling inputs from data logger , for now i am saving this inputs into a *.mdb data base file - the fields are value(of the inout i sample),date,time.
my problem is that this data base grows quikly its about 275MB for a day of sampling. do you know better way for sampling this input to a data base that will take less space
this program have to work all year!!

My second question is on a progaram or way to take this inputs and make with them reports and graph in a very fast way?

The problem you are facing is common to real-time data sampling. Without some knowledge of your data source the answer is difficult. Here are a few approaches:

Are you interested only in a change of events? IE if you are measuring temperature, and your measuring device is capable of + or - one degree and you can sample 5000 times a second, you only need a database record for every one or two degree temp change even if they are three weeks apart.

If you are measuring something that falls within certain limits like say ocean wave heights where there is a certain subjective nature to the measurement then you only are concerned with an average wave height within a certain time period.

If you are measuring a periodic function like a sin wave then you need only post deviations from this function. Remember that functions were the solution to data that seemed to be ordered but changed with time. Ultimately you should be searching for some function to represent your data changes with time. (Ever hear of Chaos Theory?)

If you are measuring the clorine content in your tap water then again you only need to note changes in data.

If you are trying to watch the NYSE then you have a few hundred companies of interest, but their value changes every few seconds. Pick a time period, say one hour. Create data fields "High", "Low", "Begin", "End", "Hour", "Date". Design your frontend so that for any given hour you store only these values. From these you can glean far more information than you ever could looking at each data change during the course of the day. After a few days have past then roll the hours up into a daily format.

If your subject is not proprietary, and you would like me to help you set the parameters for deciding on how to save your data I will be more than happy to review it. Either email me ([email protected]) or post more information about your data here.

Hope This Helps,

Hunter

[moyalt]

HELLO HUNTER,
Firsti wanne thank you for your caring and help.
My samples of data base are samples that have to enter the database file every time , i'll explain :

The datalogger system I'm connected to is a control system of gas turbine motor , the datalogger systm show the inputs coming from the sensors of the gas turbine in real time , but the sampling rate of my program (which is max by the way) is 70 inputs (sensors) for half a second - in 1 second i sample this 70 sensors twice.

this database file has to keep all the sampling for future analyzing - graphs... the database has to include this next fields : exact time , date , sensor name , sensor value .

if you could help me i will be more than greatfull.

TOMER.

[HunterMcCray]

Originally posted by moyalt
HELLO HUNTER,
Firsti wanne thank you for your caring and help.
My samples of data base are samples that have to enter the database file every time , i'll explain :

The datalogger system I'm connected to is a control system of gas turbine motor , the datalogger systm show the inputs coming from the sensors of the gas turbine in real time , but the sampling rate of my program (which is max by the way) is 70 inputs (sensors) for half a second - in 1 second i sample this 70 sensors twice.

this database file has to keep all the sampling for future analyzing - graphs... the database has to include this next fields : exact time , date , sensor name , sensor value .

if you could help me i will be more than greatfull.

TOMER.

OK, First off, Gas Turbines fascinate me. I have done some engine testing (Two Cycle Model Air Plane Engines) First let's look carefully at your data structures, and let's spend a bunch of time thinking before we begin saving tons of raw data. You should probably forget the idea of a database in favor of a sequential read/write file system. You should probably consider using the actual file record number as a clock reference, and you should probably think carefully about how to minimze each record's file size. Here are my thoughts: If you are running 24 hours a day seven days a week then you are going to need massive storage capability, but I am going to assume that you are limiting your trials to fixed periods, even if they are a day or two in length. If you reserve the first n bytes of a random access file as a "Header" and your data alway begins at say byte 1024 or 2048 and in the header you have the exact time that the trial began as well as any other parameters that you need to save with it, and you sample data at exactly 70 samples per second then any given sample's relative time to the header's time is a function of its relative position in the file. (This is the exact way .WAV files are constructed, except they have two channels and 44,100 samples per second). Secondly you need to determine the smallest possible resolution that will correctly convey your sample. ie will a byte do it, or do you need an integer or larger value? If you are measuring RPM and the normal operating RPM range is 40,000 to 70,000 then do you need to measure values below 40,000 or above 70,000? If not then you can subtract 40,000 from your actual value and use an integer to represent your data value...If you want to compress your data value even more then you can determine the maximum acelleration/decelleration of your turbine and then record the beginning RPM in the header and then only record ds/dt, probably a byte in the preceeding example. The down side to these types of compression is the amount of work that you need to do to create a front-end, the upside is that you can compress tons of data into a very small memory map without loss of resolution. A great example would be MP3 Files VS WAV files.
I will think on this some more, but it would help if you could fill in some more of the details on the nature of the testing, the length of the trials, the number and type of the data sets, and the resolution that is required. E-Mail me at [email protected] if you do not want this information posted on the board, otherwise post your answers to this thread and perhaps others can fill in areas that I miss.

Hope that it helps,

Hunter