Daylight Saving nightmare

This isn't really a VB6 issue, but I need to overcome it for an incremental backup I run (weekly) to various flash drives. Quick background:

My backup program is designed to support three different flash drives of three different sizes (because that's what I personally have): 2 GB, 32 GB and 128 GB. During the folder recursion, I include logic to NoRecurse some paths, exclude some subfolders, and pick and choose which flash drive(s) any given folder tree is sent to. I've stripped out this logic for the code posted here.

I just now noticed that all of a sudden my larger (80 GB) backup shows every single file as out of sync. Obviously I didn't change all 17,000+ files in the past few hours; I see the reason is because Daylight Saving just kicked in around four hours ago.

Here's a stripped down version of the recursion logic I'm using:

vb Code:

Private Type FILETIME
    dwLowDateTime As Long
    dwHighDateTime As Long
End Type
 
Private Type SYSTEMTIME
    wYear As Integer
    wMonth As Integer
    wDayOfWeek As Integer
    wDay As Integer
    wHour As Integer
    wMinute As Integer
    wSecond As Integer
    wMilliseconds As Long
End Type
 
Private Const MAX_PATH  As Long = 260
 
Private Type WIN32_FIND_DATA
    dwFileAttributes As Long
    ftCreationTime As FILETIME
    ftLastAccessTime As FILETIME
    ftLastWriteTime As FILETIME
    nFileSizeHigh As Long
    nFileSizeLow As Long
    dwReserved0 As Long
    dwReserved1 As Long
    cFileName As String * MAX_PATH
    cAlternate As String * 14
End Type
 
Private Declare Function FindClose Lib "kernel32" (ByVal hFindFile As Long) As Long
Private Declare Function FindFirstFile Lib "kernel32" Alias "FindFirstFileA" (ByVal lpFileName As String, lpFindFileData As WIN32_FIND_DATA) As Long
Private Declare Function FindNextFile Lib "kernel32" Alias "FindNextFileA" (ByVal hFindFile As Long, lpFindFileData As WIN32_FIND_DATA) As Long
Private Declare Function FileTimeToLocalFileTime Lib "kernel32" (lpFileTime As FILETIME, lpLocalFileTime As FILETIME) As Long
Private Declare Function FileTimeToSystemTime Lib "kernel32" (lpFileTime As FILETIME, lpSystemTime As SYSTEMTIME) As Long
 
' My data structure, as opposed to API call structures
Private Type FileType
    Group As Long
    Folder As String
    FileName As String
    FileDate As Date
    FileSize As Currency
    Flags As Long
End Type
 
Private Sub AddFolder(ByVal pstrFolder As String)
    Const MAXDWORD = &HFFFF
    Const INVALID_HANDLE_VALUE = -1
    Dim strPath As String
    Dim strFile As String
    Dim lngHandle As Long
    Dim intSearching As Integer
    Dim typFile As FileType
    Dim typFind As WIN32_FIND_DATA
 
    intSearching = 1
    lngHandle = FindFirstFile(pstrFolder & "\*.*", typFind)
    Do While lngHandle <> INVALID_HANDLE_VALUE And intSearching <> 0
        strFile = Left$(typFind.cFileName, InStr(typFind.cFileName, vbNullChar) - 1)
        If (typFind.dwFileAttributes And vbDirectory) = vbDirectory Then
            If Left$(strFile, 1) <> "." Then AddFolder pstrFolder & "\" & strFile
        Else
            Select Case strFile
                Case "desktop.ini"
                Case Else
                    With typFile
                        .Group = mlngGroup
                        .Folder = Mid$(pstrFolder, Len(mstrPrefix) + 1)
                        .FileName = strFile
                        .FileDate = StructureToDate(typFind.ftLastWriteTime)
                        .FileSize = CCur((typFind.nFileSizeHigh * MAXDWORD) + typFind.nFileSizeLow)
                    End With
                    AddFile typFile
            End Select
        End If
        intSearching = FindNextFile(lngHandle, typFind)
    Loop
    lngHandle = FindClose(lngHandle)
End Sub
 
Private Function StructureToDate(ptyp As FILETIME) As Date
    Dim typLocal As FILETIME
    Dim typSystem As SYSTEMTIME
 
    If FileTimeToLocalFileTime(ptyp, typLocal) = 1 Then
        If FileTimeToSystemTime(typLocal, typSystem) = 1 Then
            With typSystem
                StructureToDate = DateSerial(.wYear, .wMonth, .wDay) + TimeSerial(.wHour, .wMinute, .wSecond)
            End With
        End If
    End If
End Function
 
' This is the actual function I use, which includes references and data structures not included in this post
Private Sub AddFile(ptypFile As FileType, plngSource As Long)
    Dim lngMax As Long
    Dim i As Long
    
    With mtypDrive(plngSource)
        .Current = .Current + 1
        If .Current > .Files Then
            .Files = (.Files * 5) \ 4
            ReDim Preserve .File(1 To .Files)
        End If
        .File(.Current) = ptypFile
    End With
End Sub

This code correctly returns all file dates exactly like explorer shows them. Unfortunately, explorer seems to be randomly making up dates for my old haven't-touched-them-in-years files like my wallpapers.

Click the thumbnail for a comparison of the files on my hard drive vs the files I manually copied over to my Sansa Extreme 128 GB flash drive (bought on Amazon, link) a couple weeks ago:

http://s26.postimg.org/pt2bjvad1/Time_Stamps.jpg

Any ideas for how I can get proper, unchanging file dates? This is just for me on my personal computer, so any solution that works is a possibility.

For example, would it help to turn off system DST and just adjust the time manually? (Many of those unchanged files have crazy datetime changes on them, much more radical than a single hour off. Up to 9 days wrong!)

Most filesystems Windows can use store timestamps in UTC values. <- Wrong!

You should use API calls to work with those, and where necessary for display or input purposes you can do conversions to and from local time at the "point of use." Always store UTC timestamps within files or databases you use to track things. <- More complicated than this.

This is a pretty classic newb mistake, and a reason why off the shelf utilities usually make more sense than hand-rolled code for something so common. <- Thus I don't do this myself.

You might start at FILETIME structure and investigate from there if you want to correct your program.

Also check the CodeBank here, since code to call the related APIs including converstion to/from localized times in VB6 Date type format have been published there many times.

Googling around, this is likely due to the flash drives using exFAT instead of NTFS for the file system.

I can format the flash drive to use NTFS, which would presumably correct the synchronization issues, but this would increase wear and tear and reduce the lifespan of the flash drives. I'd rather not if I can help it.

Is there a way to convert exFAT file dates to NTFS file dates?

Quote:

---

Originally Posted by dilettante

You might start at FILETIME structure and investigate from there if you want to corerct your program.

Also check the CodeBank here, since code to call the related APIs including converstion to/from localized times in VB6 Date type format have been published there many times.

---

My code is returning the "correct" dates as far as Windows is concerned. The problem is Windows is not supplying the correct dates to begin with, as seen by my screenshot of Windows Explorer showing the same wrong dates as my code.

ExFAT uses the same format as FAT, i.e. timstamps are in local time. Bleh.

Quote:

---

Not all file systems can record creation and last access time and not all file systems record them in the same manner. For example, on NT FAT, create time has a resolution of 10 milliseconds, write time has a resolution of 2 seconds, and access time has a resolution of 1 day (really, the access date). On NTFS, access time has a resolution of 1 hour. Therefore, the GetFileTime function may not return the same file time information set using the SetFileTime function. Furthermore, FAT records times on disk in local time. However, NTFS records times on disk in UTC. For more information, see File Times.

---

It sounds like you're doomed unless you format in NTFS.

Quote:

---

Originally Posted by dilettante

Most filesystems Windows can use store timestamps in UTC values. You should use API calls to work with those, and where necessary for display or input purposes you can do conversions to and from local time at the "point of use." Always store UTC timestamps within files or databases you use to track things.

This is a pretty classic newb mistake, and a reason why off the shelf utilities usually make more sense than hand-rolled code for something so common.

---

Perhaps you only skimmed the OP? You don't seem to grasp what the issue is.

"Newb" mistake; that's nice.

Hey, I slapped myseld 3 times above. What more do you want?

This is like encryption and other issues. Not something you want to mess with until you do a lot of research and testing.

Actually, I'm thinking I can store a master list of file dates/times (and sizes) on the flash drive itself, so I won't be at the mercy of the file system. Tossing in an extra 1mb file at the end of each incremental backup is a total non-issue.

Quote:

---

Originally Posted by dilettante

Hey, I slapped myseld 3 times above. What more do you want?

This is like encryption and other issues. Not something you want to mess with until you do a lot of research and testing.

---

Heh, we were cross-posting. No harm no foul, I remember you well and fondly from my years posting here. (2007-2010ish)

The FindFile() API calls use a UDT to hold file info. Each date is held in its own copy of the following structure:

Code:

---

Private Type FILETIME
    dwLowDateTime As Long
    dwHighDateTime As Long
End Type

---

This structure is then converted to a readable date via FileTimeToLocalFileTime() and FileTimeToSystemTime() API calls.

This 64-bit structure (before any conversions) is the raw UTC datetime value I should store, yes?

I haven't needed to look at this in quite a while. I see that I created a class to help deal with it for another project, and perhaps there is something in there you might find useful. Basically it does what MSDN suggests.

Here's the old testbed/demo program using that class. Compile it so you can use drag/drop without running into a security boundary (the IDE is normally run elevated).

Once compiled run the program and either drag/drop in a file or folder or type its name. Then fetch the timestamps using multiple techniques... including my FileDateTimes class.

Attachment 136051

"Local Bad" shows the error you can get when you don't take into account the DST setting for the time the file's timstamps were set.

The "G drive" used for the run above is FAT32 flash drive.

Quote:

---

This code correctly returns all file dates exactly like explorer shows them. Unfortunately, explorer seems to be randomly making up dates for my old haven't-touched-them-in-years files like my wallpapers.

---

i note in the image that your date columns are headed , date modified and date
windows 10 has both (and several other date cloumns) and may contain different values

on testing it appears that sometimes date is the same as date created and sometimes the same as date modified, of course sometimes all 3 are the same
i have no idea if this has any relevance to your problem

I guess I'd convert to UTC timestamps and store those and use them for all comparisons. At that point VB's Date format is probably as good as anything.

Very cool utility, thanks much. I'll play around with it tomorrow. If any of its methods return the proper filedates on my flash drive, I'll just go with that.

If that doesn't work reliably for all files, I'll just save the 64-bit UTC timestamps manually in a "directory" file that lists all files on my flash drives. Fortunately, I already create directory files so that I can compare changes without having to actually plug in the flash drives, so this would be a minor tweak to what I was already doing. The key difference is that I had planned to re-scan each flash drive for each incremental backup instead of relying on the directory file, but I'm reasonably okay with just relying on the directory file. As a bonus, it'll be much faster, and less wear and tear on the flash drives.

Quote:

---

Originally Posted by dilettante

I guess I'd convert to UTC timestamps and store those and use them for all comparisons. At that point VB's Date format is probably as good as anything.

---

As long as the FILETIME structure I've been using is actually the UTC times, I should be good to go. The first article you linked (FILETIME structure) describes the exact structure and methodology I'm using, so it seems that I do have easy access to the UTC times.

The best part is I think that means I don't even have to convert dates at all. I could just compare the raw 64-bit values directly:

Code:

---

Private Function SameDate(plngSource As Long, plngDest As Long, plngDrive As Long) As Boolean
    If mtypDrive(0).File(plngSource).dwLowDateTime <> mtypDrive(plngDrive).File(plngDest).dwLowDateTime Then Exit Function
    If mtypDrive(0).File(plngSource).dwHighDateTime <> mtypDrive(plngDrive).File(plngDest).dwHighDateTime Then Exit Function
    SameDate = True
End Function

---

...which would be faster than converting the dates.

If it does the job then go for it. But be sure and test, test, test.

There seem to be subtle time-bombs awaiting us in handling file timestamps.

Quote:

---

Originally Posted by dilettante

There seem to be subtle time-bombs awaiting us in handling file timestamps.

---

Why should one handle file times, in this kind of common file operations?

Quote:

---

Any ideas for how I can get proper, unchanging file dates?

---

I would recommend using Robocopy, if one is making one to one** folder/file structure backups, works flawlessy.

** By 'one to one', i mean that files are not copied to single or multiple 'backup' files.

https://technet.microsoft.com/fi-fi/.../cc733145.aspx

If there is a utility that does what you need then I'd agree. That's why (aside from coping with the time-change myself, waking up to find I was out of coffee) I started out cranky and unhelpful I suppose. ;)

All we can assume is that the OP has special requirements an existing utility didn't deal with.

I would tend to think if an original file and its backed up sibling are exactly out of synch by 1 hour (or multiples thereof) the reason they are out of synch is very probably because;
1. Daylight saving has kicked in or out
2. Or you have crossed a time zone and updated you computer time to the prevailing local time by exactly n hours. International time zones are nearly always in hour intervals except I think in a couple of Pacific Island groups using 1/2 hour.

Quote:

---

Originally Posted by Tech99

I would recommend using Robocopy, if one is making one to one** folder/file structure backups, works flawlessy.

---

That's appealing, for sure. I was already thinking of ways to have the physical copying be done by a second program so that my main program can easily allow user interaction ("cancel") and progress updates without resorting to DoEvents. Communication between the two apps would probably just be the old SendMessage() to hidden textboxes trick. The upside is then I could splurge with complicated graphical progress displays without concern over slowing down the copy operations.

Robocopy would theoretically do everything I need: It supports mirroring, it handles skipping empty folders, it allows selected subfolders to be excluded when copying trees, it does incremental backups; it's golden. But, it will "fail" after DST triggers because it will think every file has changed.

The problem is that the operating system itself incorrectly sees the files as having been changed. The only two solutions would be to reformat my flash drives using NTFS, which I don't want to do, or maintain my own registry of file information, which I can't communicate to robocopy.

Quote:

---

Originally Posted by Magic Ink

I would tend to think if an original file and its backed up sibling are exactly out of synch by 1 hour (or multiples thereof)

---

Not neccessarily. I have seen that in some systems mismatch could be 2 seconds. To date it has not been clarified what was that origin, but sure two seconds apart were times.

Quote:

---

Originally Posted by Ellis Dee

...But, it will "fail" after DST triggers because it will think every file has changed.

The problem is that the operating system itself incorrectly sees the files as having been changed. The only two solutions would be to reformat my flash drives using NTFS, which I don't want to do, or maintain my own registry of file information, which I can't communicate to robocopy.

---

Wrong, it does not fail.

You just have to use/set proper parameters - the good old archive flag, when doing incremental backups.

from MSDN...

Quote:

---

/a

Copies only files for which the Archive attribute is set.

---

Actually, in addition to the archive attribute (which looks quite promising) there's also this in the RoboCopy help:

/FFT :: assume FAT File Times (2-second granularity).
/DST :: compensate for one-hour DST time differences.

Interesting...



But more on the archive attribute. Is it reliable? As in, is it maintained by Windows itself, getting set from any change, or do I need to rely on individual apps setting it for the files they modify?

A quick googling seems to suggest that the the archive bit is set by the OS, and is only ever changed (unset) by backup programs. This sounds perfect, as I'll only be using the one backup routine.

RoboCopy has the handy /M parameter:

/M :: copy only files with the Archive attribute and reset it.

Is there a way to make robocopy silent and have it send progress reports back to my program, I wonder?

Actually, cursory testing is immediately showing RoboCopy may have rights issues, such that I need to run it as administrator to copy files (while unsetting the archive bit) from \ProgramData.

Quote:

---

Originally Posted by Ellis Dee

Is there a way to make robocopy silent and have it send progress reports back to my program, I wonder?

---

Sure there is, one could execute it without command window - fex. as scheduled job or shellexecute with proper parameter and for reporting back, one might want to use /LOG parameter.

Quote:

---

Originally Posted by Ellis Dee

Actually, cursory testing is immediately showing RoboCopy may have rights issues, such that I need to run it as administrator to copy files (while unsetting the archive bit) from \ProgramData.

---

Try '/ zb' parameter...

Quote:

---

/zb

Uses Restart mode. If access is denied, this option uses Backup mode.

---

Hmmm, as I fiddle with RoboCopy, I'm not in love with some of the compromises involved. It essentially ends up as a glorified bat file, which isn't the direction I was hoping to go. And having to restart the computer is a big minus.


I just realized, however, that I don't need to compare timestamps with the flash drive files. I store the timestamps of when I do each backup. I can just copy all source files (files on the hard drive) with a modified date after the last backup. I could even toss out the size comparisons.

Can anyone think of examples that would trip up this approach? Any way files get modified without their modified date getting updated?

Quote:

---

Originally Posted by Ellis Dee

..And having to restart the computer is a big minus.

---

Why? There is no need to restart, at least i have newer experienced restart need with robocopy.

Yes, sure there is no point/need to compare timestamps - modified date neither - archive flag comparison is enough in incremental backups.

Quote:

---

Originally Posted by Tech99

Why? There is no need to restart, at least i have newer experienced restart need with robocopy.

---

Oh, I misunderstood. I saw "restart mode" and thought "restart computer."

Quote:

---

Yes, sure there is no point/need to compare timestamps - modified date neither - archive flag comparison is enough in incremental backups.

---

No need to bother with archive flags; date modified for the source files is sufficient to identify what files have changed since the last backup. Plus I don't have to change file attributes, which can cause rights issues depending on where you're copying from. And I don't have to worry about conflicts with other backup programs, like if I decide to test out some freeware backup alternatives.

Quote:

---

Originally Posted by Ellis Dee

Actually, in addition to the archive attribute (which looks quite promising) there's also this in the RoboCopy help:

/FFT :: assume FAT File Times (2-second granularity).

---

Yes - so it seems, KB article 127830 clarifies that.
https://support.microsoft.com/en-us/kb/127830

Quote:

---

Originally Posted by Ellis Dee

No need to bother with archive flags; date modified for the source files is sufficient to identify what files have changed since the last backup. Plus I don't have to change file attributes, which can cause rights issues depending on where you're copying from. And I don't have to worry about conflicts with other backup programs, like if I decide to test out some freeware backup alternatives.

---

You absolutely don't have to modify archive bit as archive bit is set when file is created or modified - i would trust that - newer failed, when making incremental backups.

There is also a flag not to reset archive bit, if you are using another backup program.

Quote:

---

/a
Copies only files for which the Archive attribute is set.

/m
Copies only files for which the Archive attribute is set, and resets the Archive attribute

---

I see no way that the archive bit is better than, or even as good as, comparing date of last backup to the dates of the source files.

Quote:

---

Originally Posted by Tech99

You absolutely don't have to modify archive bit as archive bit is set when file is created or modified

---

You absolutely do have to modify the archive bit, obviously. You have to unset it on files you back up. That's the whole point of what you're recommending. (RoboCopy failed in my first test because it didn't have privileges to unset the archive bit on files in ProgramData. Which is weird, but whatever. Running as administrator solved that.)

Unsetting the archive bit requires write privileges. Reading last modified date requires no write privileges. This is a very minor advantage, but it is still an advantage.

Quote:

---

Originally Posted by Ellis Dee

You absolutely do have to modify the archive bit, obviously. You have to unset it on files you back up. That's the whole point of what you're recommending. (RoboCopy failed in my first test because it didn't have privileges to unset the archive bit on files in ProgramData. Which is weird, but whatever. Running as administrator solved that.)

---

Modify archive bit to backup? No no and once more no...

File which has no archive bit set, is already backed up - in other words file is not modified since last backup. So - when doing incremental backups there is absolutely no need, to modify file attributes.

aand... neither when doing full backup as one could use '/MIR' switch.

Quote:

---

Originally Posted by Tech99

Modify archive bit to backup? No no and once more no...

---

Yes, yes, and once more yes.

File which has archive bit set, is not already backed up - in other words file is modified since last backup. So - when doing incremental backups there is absolutely a huge, critical, undebatable need to UNSET the archive bit AFTER you copy the file.

RoboCopy does this with the following switch:

/M :: copy only files with the Archive attribute and reset it.

This is the switch that failed when I initially tried to robocopy files in ProgramData. The fix is to run as administrator.

Your entire suggestion relies on resetting the archive attribute on files as you back them up. I honestly can't figure out why you don't seem to understand how your own suggestion works.

If you really mean that you never need to change the archive bit, then what utility does the archive bit offer? It will always stay set, and thus you'll always get complete backups, never incremental.

Hmm... it seems that you didn't figure what i meant - and how archive bit/flag works?

You don't have to manually manipulate archive bits. Robocopy clears archive bit after copy is made - then when/if you modify file by saving it, the archive bit is set again - and then incremental backup using robocopy copies file again.

So, there absolutely is no need, to manipulate archive bits (unset or set manually).

...aand what comes to incremental backup using robocopy, then the '/MIR' switch is enough, it copies modified files only - making exact mirror copy from source folder structure. So even when there are files without archive bit set, robocopy copies those files also;
- if destination file differ from source file
- if file is missing from destination

Quote:

---

Originally Posted by Tech99

Robocopy clears archive bit after copy is made

---

It seems you're not comprehending Ellis' posts? This is exactly what Ellis keeps repeating.
He's saying Robocopy is not going to be clearing any archive bits without the right permissions.

Quote:

---

Originally Posted by DEXWERX

It seems you're not comprehending Ellis' posts? This is exactly what Ellis keeps repeating.
He's saying Robocopy is not going to be clearing any archive bits without the right permissions.

---

Yeah, but that is not robocopy process fault, but insufficient permissions. Happens regardless of program used, if permissions are not adequate.

Quote:

---

No need to bother with archive flags; date modified for the source files is sufficient to identify what files have changed since the last backup. Plus I don't have to change file attributes, which can cause rights issues depending on where you're copying from. And I don't have to worry about conflicts with other backup programs, like if I decide to test out some freeware backup alternatives.

---

From this sentence, one can give impression (i certainly did), that his/her intention was not to reset archive bit - when files are copied. That is the point i am trying to figure, why - because imho there is no need to prevent archive flag from clear/reset.

Anyhow... Back to square one, with file timestamps. I would recommend studying/trying Sync framework, much easier and one could use cloud storage.

https://msdn.microsoft.com/en-us/lib...=sql.110).aspx

Synctoy is kind of robocopy with gui.
https://www.microsoft.com/en-us/down....aspx?id=15155

https://social.microsoft.com/Forums/...?forum=synctoy

One word about permissions. Create user and give it backup operator rights, then junction point folders, programdata etc. can be handled without admin level permissions.

https://technet.microsoft.com/en-us/...(v=ws.10).aspx

Quote:

---

Originally Posted by Tech99

Yeah, but that is not robocopy process fault, but insufficient permissions. Happens regardless of program used, if permissions are not adequate.

---

Okay, I figured out where we got our wires crossed.

You're saying my program doesn't need to change the archive bit, robocopy handles it.
I'm saying that if my program calls robocopy, I consider robocopy part of "my program." Anything robocopy does or fails to do is, by association, something "my program" does or fails to do.

No harm no foul.

It was really just a throw-away comment that robocopy (and by association, my program, if my program calls robocopy) needs write permissions to unset the archive bits.

I now realize that the archive bit is inadequate to handle my needs anyway. My backup program is designed to independently manage three separate backups for the same files. Namely, my three different flash drives. Here's a sample screenshot of when I first run the program: (click images for full size)

http://s26.postimg.org/ux7ddapmd/Backup.jpg

The backup code isn't technically written yet (but great progress so far; almost finished!) but I did a manual copy of most of my files to the Extreme a couple weeks ago, then manually set its last backup date to a a day before the most recent files I could find on it. Those totals are accurate.

When I plug in a flash drive, it gets autodetected. Here's what it looks like when I plug in the Ultra:

http://s26.postimg.org/7x0q0yrsl/Backup_Ultra.jpg

Now I would click "scan" to refresh the files list by reading the flash drive directly. This files list gets saved to the hard drive (for each flash drive) so the program can remember what files they hold, but I don't run the actual backup off those lists. They're just for informational purposes, like how it shows how many files have changed for the Extreme even though it's not plugged in.

After a scan, I would click backup to copy just the newer files, then eject to safely remove the flash drive. (I had to resort to a third-party freeware utility for Safely Remove because my Safely Remove code doesn't seem to work anymore.)

If I used the archive bit for the backup to the Ultra, the next time I back up to the Extreme would be incomplete. So I have to just go by Last Modified Date for each file on the hard drive.

Specifying which folders to include in each backup is stored in a simple ini file I can easily maintain with notepad. I considered writing a user interface for settings, and still might, but ultimately decided that it isn't necessary.

I definitely find this project interesting and fun to work on. With so many files involved, it's gotten me to really think about writing efficient code, which is satisfying in its own right.

The main processing code reads all the files from a given drive into a UDT array, sorts the UDT array (by folder then by file), then compares it to a similar UDT array of the master list from the hard drive to find out the list of files that have been added, changed, or deleted. Total processing time is down to around ~0.6 seconds for the Extreme. (Reading 17,000+ files from a flash drive and comparing all 17,000 against a master list of 15,000+ files, all in under a second. Nice!)

Based on your process explanation, i think that it is somewhat flawed - but anyway good luck with that. Still robocopy could be used - be it one or several backups from same source - and that is a fact. No need to write a 'specific program' to handle backup task - even when there is a need to 'opt in' files to backup. Powershell script can handle that.

Hardware changes detection is quite straightforward, one just needs to handle Win32_VolumeChangeEvent.

For unmounting one could use powershell WMI script or fex. using devcon.exe utility can perform usb device unmounting.

https://blogs.technet.microsoft.com/...ets-connected/

https://msdn.microsoft.com/en-us/lib...(v=vs.85).aspx