Best ways to transfer alot of data to .txt .xls etc

[giants00]

When you have a-lot of data that gets output across a network to excel, .txt files etc., and I mean a-lot. I'm just wondering if it is better to store the data and then output it all at once after testing is finished. Or is it better to output data across the network after each test, as the test progresses, the test is automating electronics through gpib? Just as a rough example a single test would contain enough data to fill a 40X1K array. A full test would multiply that by up to 10K times.

So would the sequence be better which way:
Run Test
Collect Data
Output Data
Run Next Test

or

Run Test
Collect Data
Store Data
Run Next Test
Finished Testing
Output all data

[jmcilhinney]

It would depend on circumstances but you might be better off writing the output after each individual test. You could perform the test and write the output on different threads, thereby potentially reducing the overall time to complete the entire process.

[IanS]

I do a lot of this kind of thing and I prefer to store everything in memory until I have a complete result set.

For example a full result set might contain 10 different tests all stored in one file and the file must not exist unless it contains a full result set. If I was saving at each stage and the computer or network crashed then I'd have an incomplete result set. not good.

My favorite method is to save the text to an invisible textbox. During debugging I can make the box visible so that I can observe it but the release version has the box hidden. The reason I use the textbox is that it has very quick save method built in without me needing to worry about file handles etc.

[IanS]

Let me go in to a little more detail. A project I did a few years ago was a Brake Test Rig (Used by manufacturers of brakes for Semi trailers etc.

A single brake goes through a number of different tests generating maybe a few hundred lines of data and the test rig might test 500 brakes a day.

I'd collect all the various test results for ONE brake, maybe 100k of data, and save that before starting on the next the next brake. In other words I'm saving whole meaningful result sets.

Once you've collected enough data that is meaningful on its own then save it. But don't save a few small test results if they don't mean anything without the rest - if you see what I mean.

[jmcilhinney]

My favorite method is to save the text to an invisible textbox. During debugging I can make the box visible so that I can observe it but the release version has the box hidden. The reason I use the textbox is that it has very quick save method built in without me needing to worry about file handles etc.

That's really not good. If you want to view data while debugging then you should be calling Debug.WriteLine. That will display in the Output window by default but you can redirect it and the code will simply not be compiled into a Release build. If you want to save then you call File.WriteAllText.

[KGComputers]

In our company, we handle large volumes of xml files everday..If we test all xml files from different servers/networks, I agree with writing output/testing on different threads.
And each resource/method, we have a test class for it.

[IanS]

That's really not good. If you want to view data while debugging then you should be calling Debug.WriteLine. That will display in the Output window by default but you can redirect it and the code will simply not be compiled into a Release build. If you want to save then you call File.WriteAllText.

Sorry, I didn't mean debugging "In the Debugger /IDE".

I meant 'After deployment' when I'm setting up, servicing or calibrating the system on the target hardware I like to observe the data being captured. I can expose the textbox and hide it again before handing back to the user.

[jmcilhinney]

Sorry, I didn't mean debugging "In the Debugger /IDE".

I meant 'After deployment' when I'm setting up, servicing or calibrating the system on the target hardware I like to observe the data being captured. I can expose the textbox and hide it again before handing back to the user.

That would be what tracing is for.

[IanS]

Now I can learn something - what's 'Tracing' ?

[jmcilhinney]

Now I can learn something - what's 'Tracing' ?

As always, look first and ask questions later.

http://social.msdn.microsoft.com/Sea...20tracing&ac=3

If you already have the keyword(s) then you have what you need to search.