Portable Data - Advice Needed

[Slabs1960]

I do not need this right now, but is sitting in the back of my mind.

The application I an writing is desktop based. The targeted user needs this to be portable. They will be using the application on more than one PC. The application has a 'base' database from star catalogues. They can add, edit or modify the data as per their requirements. This must be installed on the initial install.

When installed the app will have the initial database installed with it. They will then add data as per their requirements. This will probably done on their desktop PC's. They will then want to transfer the data to a laptop/ tablet for field work. Log their data and transfer back to the desktop system.

What is a nice easy and elegant way to do this?

[Shaggy Hiker]

Your database will want to have GUID primary keys so that a record can be transferred from one DB to another without having the primary keys conflict with existing data. Beyond that, I'd be looking at compressed XML files. The reason I'd be using XML, is that moving data from one database to another is particularly easy using XML. This is because both datasets and datatables have WriteXML and ReadXML methods. Therefore, they can be written to, and read from, a text file with a single line of code. It doesn't get easier than that. The files may or may not be excessively large, though, which is why I suggest the compression. This increases the size of the the code for writing and reading, but only to half a dozen lines, or so.

At that point, you can just pass the files around by whatever means you feel like. If you have a web server, you could run a service to upload and download them, or you could just pass them around on flash drives.

I've got a program that works this way. The field systems are generally quite remote. When they are ready, they hit a button that packages some part of their data (or all of it) into zipped XML files. These are then emailed to a coordinator when they get the chance. At worst, they could put them on a flash drive and hand carry them. The coordinator then imports the data into their local database. At some point, they can then upload the data to a central database. That upload process works the same way, but it's using a web service for the transfers because, while the field computers can be quite remote, the coordinators never are.

EDIT: I should also add that I do something like this using JSON in a different application. JSON has become more widespread than XML, and if you are working with non-.NET and non-Windows systems, JSON will likely be easier. As long as you are talking about .NET applications running on Windows systems, which you probably are, since you are talking about databases on the mobile computers, then XML will be easier because of the great ease of moving datatables to and from XML.

[Slabs1960]

Thanks.

My data is already in XML format. The data is in a XML DataSet. The files are small and should not be bigger than 1 or 2 MB. Is it possible to have the backed up data (either a user specified directory, or preferably a system generated backup) installed with the deployment and installation?

[Shaggy Hiker]

Yes, an installer can do what you describe, but you may not want to. That would work as long as the files don't change, which seems unlikely.

[Slabs1960]

Yes, an installer can do what you describe, but you may not want to. That would work as long as the files don't change, which seems unlikely.

I have a database with a Astronomical Catalogue. This has a list of Galaxy's etc. This catalogue has about 150 records in it. The idea is that on first install, on a PC this, this catalogue would be installed. The user would then have a base from which to work (this data would not change). The user could then add his own items over time. He should then be able to backup his modified catalogue and restore on on another PC.

Any sample code or method out there that I could use to implement this?

[Shaggy Hiker]

It's that bit about modifying and restoring that makes me think that you want to do this as a two step process rather than just on install. I probably could give you some sample code, I'm just not quite sure what code would benefit you, yet. My thinking is that you'd just have something in the program that would import an .XML file (you could even give it your own file type to indicate that it is an XML file usable by the program). You'd also want to be able to store it, but since a dataset has that functionality built in, there isn't much to show about that.

[Sitten Spynne]

There's going to be a lot of different approaches because this isn't a "tutorial" kind of problem, there's lots of ways that would work.

One way would be to treat the "original" data as read-only. An installer could install it, or the program could download it, or the user could download it... it really doesn't matter. This is what a user gets at the start. Your program doesn't touch it. If the user wants to restore it, they just have to do whatever they did at installation: a repair install, a re-download, etc.

The devil's in the details for the user data. To me the hard part is to decide between, "I open my specified file at startup" or "I wait for the user to tell me which file to open". Both have pros and cons, and the implementation's different for either.

If you go for "I open at startup", you're taking some control away from the user. In this case, I wouldn't have "File>Open..." or even "File>Save...", I'd auto-save every few minutes *and* when the program closes. The user would indicate a directory in the Options dialog where they want their files saved, and your program would choose the file names. When they want to back up their work, you tell them to copy every file in the directory they chose. When they want to restore it on a new machine, you tell them to place those files in the directory they've chosen on the new machine. You can sort of automate this with buttons, but whatever.

If you go for "I wait for the user to tell me", you need a "File>Open..." and "File>Save..." mechanism. At any time, the user might choose to save their work, you let them do so with a file picker. When they start the program, they choose "File>Open..." to indicate what data they want. Moving to a new machine is as simple as telling them to copy whatever files they wanted to save. In this case, they could even work from a USB drive.

There's not really much to show, the bulk of the work is basic save/load file code and there's a million tutorials for that. The hard part is figuring out which bits to glue together.

[Slabs1960]

Thank for the advice. You guys have given me something to think about.

My instinct would be to take the user control away. With user control you tend to get the support call - "Where has all my data gone?". My thought was to have a backup directory somewhere in the install directory. Whenever the user makes changes and saves, a backup is made. If something happens it will be a simple choice of "restore backup". No messing with file paths. etc.

Your thoughts?

[jdc20181]

You could also, do it easy and dirty and do it via online, but each pc would need to have a connection to internet for it work.

This is honestly the easiest way, unless you create a network, like my old school did where all the application data is saved, and on each computer etc.

[Sitten Spynne]

Technically you can't put user data in the install directory, especially if that directory happens to be under "Program Files". If you try to write files there these days, at best your program throws an exception and at worst a virtualized location is created.

If you're making data that isn't user-generated, the recommended place to put it is in a folder under AppData. That can vary on different machines, but this snippet will always lead the way:

Function GetAppDataDirectory() As String
    Dim appDataPath = Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData)
    Return appDataPath
End Function

Users sometimes don't know where that is, but you can add a button that either tells them or opens the folder for them.

[Slabs1960]

Technically you can't put user data in the install directory, especially if that directory happens to be under "Program Files". If you try to write files there these days, at best your program throws an exception and at worst a virtualized location is created.

If you're making data that isn't user-generated, the recommended place to put it is in a folder under AppData. That can vary on different machines, but this snippet will always lead the way:

Function GetAppDataDirectory() As String
    Dim appDataPath = Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData)
    Return appDataPath
End Function

Users sometimes don't know where that is, but you can add a button that either tells them or opens the folder for them.

Thanks.

I will test it. I am planning to add a "Backup to Removable Media" in any case to make it easy for the user.