This isn't really a VB6 issue, but I need to overcome it for an incremental backup I run (weekly) to various flash drives. Quick background:
My backup program is designed to support three different flash drives of three different sizes (because that's what I personally have): 2 GB, 32 GB and 128 GB. During the folder recursion, I include logic to NoRecurse some paths, exclude some subfolders, and pick and choose which flash drive(s) any given folder tree is sent to. I've stripped out this logic for the code posted here.
I just now noticed that all of a sudden my larger (80 GB) backup shows every single file as out of sync. Obviously I didn't change all 17,000+ files in the past few hours; I see the reason is because Daylight Saving just kicked in around four hours ago.
Here's a stripped down version of the recursion logic I'm using:
vb Code:
Private Type FILETIME
dwLowDateTime As Long
dwHighDateTime As Long
End Type
Private Type SYSTEMTIME
wYear As Integer
wMonth As Integer
wDayOfWeek As Integer
wDay As Integer
wHour As Integer
wMinute As Integer
wSecond As Integer
wMilliseconds As Long
End Type
Private Const MAX_PATH As Long = 260
Private Type WIN32_FIND_DATA
dwFileAttributes As Long
ftCreationTime As FILETIME
ftLastAccessTime As FILETIME
ftLastWriteTime As FILETIME
nFileSizeHigh As Long
nFileSizeLow As Long
dwReserved0 As Long
dwReserved1 As Long
cFileName As String * MAX_PATH
cAlternate As String * 14
End Type
Private Declare Function FindClose Lib "kernel32" (ByVal hFindFile As Long) As Long
Private Declare Function FindFirstFile Lib "kernel32" Alias "FindFirstFileA" (ByVal lpFileName As String, lpFindFileData As WIN32_FIND_DATA) As Long
Private Declare Function FindNextFile Lib "kernel32" Alias "FindNextFileA" (ByVal hFindFile As Long, lpFindFileData As WIN32_FIND_DATA) As Long
Private Declare Function FileTimeToLocalFileTime Lib "kernel32" (lpFileTime As FILETIME, lpLocalFileTime As FILETIME) As Long
Private Declare Function FileTimeToSystemTime Lib "kernel32" (lpFileTime As FILETIME, lpSystemTime As SYSTEMTIME) As Long
' My data structure, as opposed to API call structures
' This is the actual function I use, which includes references and data structures not included in this post
Private Sub AddFile(ptypFile As FileType, plngSource As Long)
Dim lngMax As Long
Dim i As Long
With mtypDrive(plngSource)
.Current = .Current + 1
If .Current > .Files Then
.Files = (.Files * 5) \ 4
ReDim Preserve .File(1 To .Files)
End If
.File(.Current) = ptypFile
End With
End Sub
This code correctly returns all file dates exactly like explorer shows them. Unfortunately, explorer seems to be randomly making up dates for my old haven't-touched-them-in-years files like my wallpapers.
Click the thumbnail for a comparison of the files on my hard drive vs the files I manually copied over to my Sansa Extreme 128 GB flash drive (bought on Amazon, link) a couple weeks ago:
Any ideas for how I can get proper, unchanging file dates? This is just for me on my personal computer, so any solution that works is a possibility.
For example, would it help to turn off system DST and just adjust the time manually? (Many of those unchanged files have crazy datetime changes on them, much more radical than a single hour off. Up to 9 days wrong!)
Last edited by Ellis Dee; Mar 13th, 2016 at 06:18 AM.
Most filesystems Windows can use store timestamps in UTC values. <- Wrong!
You should use API calls to work with those, and where necessary for display or input purposes you can do conversions to and from local time at the "point of use." Always store UTC timestamps within files or databases you use to track things. <- More complicated than this.
This is a pretty classic newb mistake, and a reason why off the shelf utilities usually make more sense than hand-rolled code for something so common. <- Thus I don't do this myself.
Last edited by dilettante; Mar 13th, 2016 at 06:32 AM.
Reason: one mistake after another
You might start at FILETIME structure and investigate from there if you want to correct your program.
Also check the CodeBank here, since code to call the related APIs including converstion to/from localized times in VB6 Date type format have been published there many times.
Googling around, this is likely due to the flash drives using exFAT instead of NTFS for the file system.
I can format the flash drive to use NTFS, which would presumably correct the synchronization issues, but this would increase wear and tear and reduce the lifespan of the flash drives. I'd rather not if I can help it.
Is there a way to convert exFAT file dates to NTFS file dates?
You might start at FILETIME structure and investigate from there if you want to corerct your program.
Also check the CodeBank here, since code to call the related APIs including converstion to/from localized times in VB6 Date type format have been published there many times.
My code is returning the "correct" dates as far as Windows is concerned. The problem is Windows is not supplying the correct dates to begin with, as seen by my screenshot of Windows Explorer showing the same wrong dates as my code.
ExFAT uses the same format as FAT, i.e. timstamps are in local time. Bleh.
Not all file systems can record creation and last access time and not all file systems record them in the same manner. For example, on NT FAT, create time has a resolution of 10 milliseconds, write time has a resolution of 2 seconds, and access time has a resolution of 1 day (really, the access date). On NTFS, access time has a resolution of 1 hour. Therefore, the GetFileTime function may not return the same file time information set using the SetFileTime function. Furthermore, FAT records times on disk in local time. However, NTFS records times on disk in UTC. For more information, see File Times.
It sounds like you're doomed unless you format in NTFS.
Most filesystems Windows can use store timestamps in UTC values. You should use API calls to work with those, and where necessary for display or input purposes you can do conversions to and from local time at the "point of use." Always store UTC timestamps within files or databases you use to track things.
This is a pretty classic newb mistake, and a reason why off the shelf utilities usually make more sense than hand-rolled code for something so common.
Perhaps you only skimmed the OP? You don't seem to grasp what the issue is.
Actually, I'm thinking I can store a master list of file dates/times (and sizes) on the flash drive itself, so I won't be at the mercy of the file system. Tossing in an extra 1mb file at the end of each incremental backup is a total non-issue.
I haven't needed to look at this in quite a while. I see that I created a class to help deal with it for another project, and perhaps there is something in there you might find useful. Basically it does what MSDN suggests.
Here's the old testbed/demo program using that class. Compile it so you can use drag/drop without running into a security boundary (the IDE is normally run elevated).
Once compiled run the program and either drag/drop in a file or folder or type its name. Then fetch the timestamps using multiple techniques... including my FileDateTimes class.
"Local Bad" shows the error you can get when you don't take into account the DST setting for the time the file's timstamps were set.
The "G drive" used for the run above is FAT32 flash drive.
Last edited by dilettante; Mar 13th, 2016 at 07:02 AM.
This code correctly returns all file dates exactly like explorer shows them. Unfortunately, explorer seems to be randomly making up dates for my old haven't-touched-them-in-years files like my wallpapers.
i note in the image that your date columns are headed , date modified and date
windows 10 has both (and several other date cloumns) and may contain different values
on testing it appears that sometimes date is the same as date created and sometimes the same as date modified, of course sometimes all 3 are the same
i have no idea if this has any relevance to your problem
i do my best to test code works before i post it, but sometimes am unable to do so for some reason, and usually say so if this is the case. Note code snippets posted are just that and do not include error handling that is required in real world applications, but avoid On Error Resume Next
dim all variables as required as often i have done so elsewhere in my code but only posted the relevant part
come back and mark your original post as resolved if your problem is fixed
pete
I guess I'd convert to UTC timestamps and store those and use them for all comparisons. At that point VB's Date format is probably as good as anything.
Very cool utility, thanks much. I'll play around with it tomorrow. If any of its methods return the proper filedates on my flash drive, I'll just go with that.
If that doesn't work reliably for all files, I'll just save the 64-bit UTC timestamps manually in a "directory" file that lists all files on my flash drives. Fortunately, I already create directory files so that I can compare changes without having to actually plug in the flash drives, so this would be a minor tweak to what I was already doing. The key difference is that I had planned to re-scan each flash drive for each incremental backup instead of relying on the directory file, but I'm reasonably okay with just relying on the directory file. As a bonus, it'll be much faster, and less wear and tear on the flash drives.
Originally Posted by dilettante
I guess I'd convert to UTC timestamps and store those and use them for all comparisons. At that point VB's Date format is probably as good as anything.
As long as the FILETIME structure I've been using is actually the UTC times, I should be good to go. The first article you linked (FILETIME structure) describes the exact structure and methodology I'm using, so it seems that I do have easy access to the UTC times.
The best part is I think that means I don't even have to convert dates at all. I could just compare the raw 64-bit values directly:
Code:
Private Function SameDate(plngSource As Long, plngDest As Long, plngDrive As Long) As Boolean
If mtypDrive(0).File(plngSource).dwLowDateTime <> mtypDrive(plngDrive).File(plngDest).dwLowDateTime Then Exit Function
If mtypDrive(0).File(plngSource).dwHighDateTime <> mtypDrive(plngDrive).File(plngDest).dwHighDateTime Then Exit Function
SameDate = True
End Function
...which would be faster than converting the dates.
If there is a utility that does what you need then I'd agree. That's why (aside from coping with the time-change myself, waking up to find I was out of coffee) I started out cranky and unhelpful I suppose.
All we can assume is that the OP has special requirements an existing utility didn't deal with.
I would tend to think if an original file and its backed up sibling are exactly out of synch by 1 hour (or multiples thereof) the reason they are out of synch is very probably because;
1. Daylight saving has kicked in or out
2. Or you have crossed a time zone and updated you computer time to the prevailing local time by exactly n hours. International time zones are nearly always in hour intervals except I think in a couple of Pacific Island groups using 1/2 hour.
I would recommend using Robocopy, if one is making one to one** folder/file structure backups, works flawlessy.
That's appealing, for sure. I was already thinking of ways to have the physical copying be done by a second program so that my main program can easily allow user interaction ("cancel") and progress updates without resorting to DoEvents. Communication between the two apps would probably just be the old SendMessage() to hidden textboxes trick. The upside is then I could splurge with complicated graphical progress displays without concern over slowing down the copy operations.
Robocopy would theoretically do everything I need: It supports mirroring, it handles skipping empty folders, it allows selected subfolders to be excluded when copying trees, it does incremental backups; it's golden. But, it will "fail" after DST triggers because it will think every file has changed.
The problem is that the operating system itself incorrectly sees the files as having been changed. The only two solutions would be to reformat my flash drives using NTFS, which I don't want to do, or maintain my own registry of file information, which I can't communicate to robocopy.
I would tend to think if an original file and its backed up sibling are exactly out of synch by 1 hour (or multiples thereof)
Not neccessarily. I have seen that in some systems mismatch could be 2 seconds. To date it has not been clarified what was that origin, but sure two seconds apart were times.
...But, it will "fail" after DST triggers because it will think every file has changed.
The problem is that the operating system itself incorrectly sees the files as having been changed. The only two solutions would be to reformat my flash drives using NTFS, which I don't want to do, or maintain my own registry of file information, which I can't communicate to robocopy.
Wrong, it does not fail.
You just have to use/set proper parameters - the good old archive flag, when doing incremental backups.
from MSDN...
/a
Copies only files for which the Archive attribute is set.
Actually, in addition to the archive attribute (which looks quite promising) there's also this in the RoboCopy help:
/FFT :: assume FAT File Times (2-second granularity).
/DST :: compensate for one-hour DST time differences.
Interesting...
But more on the archive attribute. Is it reliable? As in, is it maintained by Windows itself, getting set from any change, or do I need to rely on individual apps setting it for the files they modify?
A quick googling seems to suggest that the the archive bit is set by the OS, and is only ever changed (unset) by backup programs. This sounds perfect, as I'll only be using the one backup routine.
RoboCopy has the handy /M parameter:
/M :: copy only files with the Archive attribute and reset it.
Is there a way to make robocopy silent and have it send progress reports back to my program, I wonder?
Actually, cursory testing is immediately showing RoboCopy may have rights issues, such that I need to run it as administrator to copy files (while unsetting the archive bit) from \ProgramData.
Is there a way to make robocopy silent and have it send progress reports back to my program, I wonder?
Sure there is, one could execute it without command window - fex. as scheduled job or shellexecute with proper parameter and for reporting back, one might want to use /LOG parameter.
Actually, cursory testing is immediately showing RoboCopy may have rights issues, such that I need to run it as administrator to copy files (while unsetting the archive bit) from \ProgramData.
Try '/ zb' parameter...
/zb
Uses Restart mode. If access is denied, this option uses Backup mode.
Hmmm, as I fiddle with RoboCopy, I'm not in love with some of the compromises involved. It essentially ends up as a glorified bat file, which isn't the direction I was hoping to go. And having to restart the computer is a big minus.
I just realized, however, that I don't need to compare timestamps with the flash drive files. I store the timestamps of when I do each backup. I can just copy all source files (files on the hard drive) with a modified date after the last backup. I could even toss out the size comparisons.
Can anyone think of examples that would trip up this approach? Any way files get modified without their modified date getting updated?
Why? There is no need to restart, at least i have newer experienced restart need with robocopy.
Oh, I misunderstood. I saw "restart mode" and thought "restart computer."
Yes, sure there is no point/need to compare timestamps - modified date neither - archive flag comparison is enough in incremental backups.
No need to bother with archive flags; date modified for the source files is sufficient to identify what files have changed since the last backup. Plus I don't have to change file attributes, which can cause rights issues depending on where you're copying from. And I don't have to worry about conflicts with other backup programs, like if I decide to test out some freeware backup alternatives.
No need to bother with archive flags; date modified for the source files is sufficient to identify what files have changed since the last backup. Plus I don't have to change file attributes, which can cause rights issues depending on where you're copying from. And I don't have to worry about conflicts with other backup programs, like if I decide to test out some freeware backup alternatives.
You absolutely don't have to modify archive bit as archive bit is set when file is created or modified - i would trust that - newer failed, when making incremental backups.
There is also a flag not to reset archive bit, if you are using another backup program.
/a
Copies only files for which the Archive attribute is set.
/m
Copies only files for which the Archive attribute is set, and resets the Archive attribute
Last edited by Tech99; Mar 14th, 2016 at 03:05 AM.
I see no way that the archive bit is better than, or even as good as, comparing date of last backup to the dates of the source files.
Originally Posted by Tech99
You absolutely don't have to modify archive bit as archive bit is set when file is created or modified
You absolutely do have to modify the archive bit, obviously. You have to unset it on files you back up. That's the whole point of what you're recommending. (RoboCopy failed in my first test because it didn't have privileges to unset the archive bit on files in ProgramData. Which is weird, but whatever. Running as administrator solved that.)
Unsetting the archive bit requires write privileges. Reading last modified date requires no write privileges. This is a very minor advantage, but it is still an advantage.
Last edited by Ellis Dee; Mar 14th, 2016 at 04:35 AM.
You absolutely do have to modify the archive bit, obviously. You have to unset it on files you back up. That's the whole point of what you're recommending. (RoboCopy failed in my first test because it didn't have privileges to unset the archive bit on files in ProgramData. Which is weird, but whatever. Running as administrator solved that.)
Modify archive bit to backup? No no and once more no...
File which has no archive bit set, is already backed up - in other words file is not modified since last backup. So - when doing incremental backups there is absolutely no need, to modify file attributes.
aand... neither when doing full backup as one could use '/MIR' switch.
Modify archive bit to backup? No no and once more no...
Yes, yes, and once more yes.
File which has archive bit set, is not already backed up - in other words file is modified since last backup. So - when doing incremental backups there is absolutely a huge, critical, undebatable need to UNSET the archive bit AFTER you copy the file.
RoboCopy does this with the following switch:
/M :: copy only files with the Archive attribute and reset it.
This is the switch that failed when I initially tried to robocopy files in ProgramData. The fix is to run as administrator.
Your entire suggestion relies on resetting the archive attribute on files as you back them up. I honestly can't figure out why you don't seem to understand how your own suggestion works.
If you really mean that you never need to change the archive bit, then what utility does the archive bit offer? It will always stay set, and thus you'll always get complete backups, never incremental.
Hmm... it seems that you didn't figure what i meant - and how archive bit/flag works?
You don't have to manually manipulate archive bits. Robocopy clears archive bit after copy is made - then when/if you modify file by saving it, the archive bit is set again - and then incremental backup using robocopy copies file again.
So, there absolutely is no need, to manipulate archive bits (unset or set manually).
...aand what comes to incremental backup using robocopy, then the '/MIR' switch is enough, it copies modified files only - making exact mirror copy from source folder structure. So even when there are files without archive bit set, robocopy copies those files also;
- if destination file differ from source file
- if file is missing from destination
Last edited by Tech99; Mar 14th, 2016 at 09:45 AM.
It seems you're not comprehending Ellis' posts? This is exactly what Ellis keeps repeating.
He's saying Robocopy is not going to be clearing any archive bits without the right permissions.
It seems you're not comprehending Ellis' posts? This is exactly what Ellis keeps repeating.
He's saying Robocopy is not going to be clearing any archive bits without the right permissions.
Yeah, but that is not robocopy process fault, but insufficient permissions. Happens regardless of program used, if permissions are not adequate.
No need to bother with archive flags; date modified for the source files is sufficient to identify what files have changed since the last backup. Plus I don't have to change file attributes, which can cause rights issues depending on where you're copying from. And I don't have to worry about conflicts with other backup programs, like if I decide to test out some freeware backup alternatives.
From this sentence, one can give impression (i certainly did), that his/her intention was not to reset archive bit - when files are copied. That is the point i am trying to figure, why - because imho there is no need to prevent archive flag from clear/reset.
One word about permissions. Create user and give it backup operator rights, then junction point folders, programdata etc. can be handled without admin level permissions.
Yeah, but that is not robocopy process fault, but insufficient permissions. Happens regardless of program used, if permissions are not adequate.
Okay, I figured out where we got our wires crossed.
You're saying my program doesn't need to change the archive bit, robocopy handles it.
I'm saying that if my program calls robocopy, I consider robocopy part of "my program." Anything robocopy does or fails to do is, by association, something "my program" does or fails to do.
No harm no foul.
It was really just a throw-away comment that robocopy (and by association, my program, if my program calls robocopy) needs write permissions to unset the archive bits.
I now realize that the archive bit is inadequate to handle my needs anyway. My backup program is designed to independently manage three separate backups for the same files. Namely, my three different flash drives. Here's a sample screenshot of when I first run the program: (click images for full size)
The backup code isn't technically written yet (but great progress so far; almost finished!) but I did a manual copy of most of my files to the Extreme a couple weeks ago, then manually set its last backup date to a a day before the most recent files I could find on it. Those totals are accurate.
When I plug in a flash drive, it gets autodetected. Here's what it looks like when I plug in the Ultra:
Now I would click "scan" to refresh the files list by reading the flash drive directly. This files list gets saved to the hard drive (for each flash drive) so the program can remember what files they hold, but I don't run the actual backup off those lists. They're just for informational purposes, like how it shows how many files have changed for the Extreme even though it's not plugged in.
After a scan, I would click backup to copy just the newer files, then eject to safely remove the flash drive. (I had to resort to a third-party freeware utility for Safely Remove because my Safely Remove code doesn't seem to work anymore.)
If I used the archive bit for the backup to the Ultra, the next time I back up to the Extreme would be incomplete. So I have to just go by Last Modified Date for each file on the hard drive.
Specifying which folders to include in each backup is stored in a simple ini file I can easily maintain with notepad. I considered writing a user interface for settings, and still might, but ultimately decided that it isn't necessary.
I definitely find this project interesting and fun to work on. With so many files involved, it's gotten me to really think about writing efficient code, which is satisfying in its own right.
The main processing code reads all the files from a given drive into a UDT array, sorts the UDT array (by folder then by file), then compares it to a similar UDT array of the master list from the hard drive to find out the list of files that have been added, changed, or deleted. Total processing time is down to around ~0.6 seconds for the Extreme. (Reading 17,000+ files from a flash drive and comparing all 17,000 against a master list of 15,000+ files, all in under a second. Nice!)
Last edited by Ellis Dee; Mar 15th, 2016 at 06:42 AM.
Reply With Quote
Originally Posted by dilettante
