Printable View
Yes Tech9 get with the program :confused:
-----Quote:
Code:NTSTATUS = NtOpenFile(hFind, _
FILE_LIST_DIRECTORY Or SYNCHRONIZE Or FILE_READ_ACCESS, _
obAttr, _
objIoStatus, _
3, _
FILE_DIRECTORY_FILE Or FILE_SYNCHRONOUS_IO_NONALERT)
Edit:
I believe I found the problem with the missing files. It won't list more than 144 files in a particular directory. Haven't figured out how to solve it yet; increasing the buffer size didn't help. This limit only applies on my remote share.
Relieved that someone else is seeing the same problem I am! The number 144 that you are experiencing is not the same for me - I see more than that on my remote share but it's not too far off from that. I don't recall the exact number but I'm at around 169. I'm hopeful that you will find the problem. My optimism is rekindled! :)Quote:
Originally Posted by fafalone
Yeah and the number varies based on which information I request... 144 is what I get with FILE_DIRECTORY_INFORMATION, and with FILE_FULL_DIR_INFORMATION I get 139.
Edit: So FILE_NAMES_INFORMATION seems to be returning all the files. The only problem is that this structure doesn't include FileAttributes, so to search recursively you'd have to make a separate call to PathIsDirectory.
Code:Private Type FILE_NAMES_INFORMATION
NextEntryOffset As Long
FileIndex As Long
FileNameLength As Long
FileName1 As Integer '//OFFSET= &HC
End Type
FileNamesInformation = 12
In terms of returned filenames ie. limiting count see this.
http://gate.upm.ro/os/LABs/Windows_O...io/iomgr/dir.c
Rather than calling PathIsDirectory you can stay with the NT API to determine if a path is a directory. Here's a routine I just created. It should be faster than the Win32 equivalent.Quote:
Originally Posted by fafalone
This solution would be fine if someone simply needs the list of all files/folders. I need more than that however, I need all of the file details (size, dates, attributes). I could obtain this with a follow-up call to NtQueryFullAttributesFile (or NtQueryInformationFile) but I think that the additional call to this function for each file found would negate any performance gain I obtained from using NTQueryDirectoryFile.Code:
Private Declare Function NtQueryAttributesFile Lib "ntdll.dll" (ObjectAttributes As OBJECT_ATTRIBUTES, FileBasicInfo As FILE_BASIC_INFORMATION) As Long
Public Function NTIsPathDirectory(NTPath As String) As Boolean
'//Determines if a path is a DIRECTORY
'Note 1: NTPath must be an NT path not a DOS Path (i.e. "\??\C:\" not "C:\")
'Note 2: NtQueryAttributesFile() uses as a result FILE_BASIC_INFORMATION, but it fills only FileAttributes field.
Dim obAttr As OBJECT_ATTRIBUTES
Dim fbi As FILE_BASIC_INFORMATION
Dim usUnicode As UNICODE_STRING
Dim lngNTStatus As Long
'initialize Unicode String structure
RtlInitUnicodeString usUnicode, StrPtr(NTPath)
'populate Object Attributes struct
With obAttr
.Length = LenB(obAttr)
.Attributes = OBJ_CASE_INSENSITIVE
.ObjectName = VarPtr(usUnicode)
.RootDirectory = 0
.SecurityDescriptor = 0
.SecurityQualityOfService = 0
End With
lngNTStatus = NtQueryAttributesFile(obAttr, fbi)
If lngNTStatus = STATUS_SUCCESS Then
NTIsPathDirectory = (fbi.FileAttributes And FILE_ATTRIBUTE_DIRECTORY)
Else
Debug.Print "Path: " & NTPath; ", NTStatus: " & CStr(lngNTStatus)
End If
End Function
Is this the section to which you are referring?Quote:
Originally Posted by Tech99
Fafalone's code consider all of these things I believe. Though I do have questions about the bytBuffer() size. Why is this being used as the buffer size for FindNextFile?Quote:
This service operates on a directory file specified by the FileHandle
parameter. The service returns information about files in the directory
specified by the file handle. The ReturnSingleEntry parameter specifies
that only a single entry should be returned rather than filling the buffer.
The actual number of files whose information is returned, is the smallest
of the following:
o One entry, if the ReturnSingleEntry parameter is TRUE.
o The number of files whose information fits into the specified
buffer.
o The number of files that exist.
o One entry if the optional FileName parameter is specified.
Breaking this down into its components:Code:(LenB(pDir) + CLng(260 * 2 - 3)) * CLng(&H2000)
1) byte length of the FILE_DIRECTORY_INFORMATION structure
2) MAX_PATH * 2 - 3Code:LenB(pDir)
3) 8192Code:CLng(260 * 2 - 3))
My questions are:Code:CLng(&H2000)
2) Why is 260 being used? That's a Win32 number to represent MAX_PATH. But this limitation doesn't apply in kernel mode. The actual length limit for full paths in the kernel is 32767. Does this implementation not allow for long file names? Why do we subtract 3 from MAX_PATH * 2?
3) What is the purpose of multiplying all of this by 8192?
Thanks for helping me out with this. And maybe talking this out will yield some insight which leads to a solution?
Though if the buffer size was the issue here I would expect to see a STATUS_BUFFER_TOO_SMALL error but this never occurs.
I was referring this, under NTQueryDirectoryFile and Routine Description in there...Quote:
Originally Posted by AAraya
I would not assume that STATUS_BUFFER_TOO_SMALL would be a return value, routine just fills available buffer and that's it.Quote:
The actual number of files whose information is returned, is the smallest
of the following:
o One entry, if the ReturnSingleEntry parameter is TRUE.
o The number of files whose information fits into the specified buffer.
o The number of files that exist.
o One entry if the optional FileName parameter is specified.
260 comes from the fact that the 'originally' in NTFS file system, maximum path length was 1+2+256+1 ie. [drive][:\][path][null], nowadays maximum path length is that mentioned 32K.Quote:
Originally Posted by AAraya
I wonder (now i am assuming) that when/if buffer is insufficient in size, the return value would be STATUS_BUFFER_OVERFLOW?
The 8192 shouldn't matter unless there's more than 8192 file names (that's why it's multiplied by the length of the name). I tried increasing it; it makes no difference.
I'm having a temporary memory lapse; the full path can be 32,767, but the buffer here doesn't store the full path, only the file name. Can that be longer than MAX_PATH?
Maximum length of filename part in path is 255 characters (lpMaximumComponentLength limits that), actually all individual path parts are limited to that**.
https://msdn.microsoft.com/en-us/library/aa365247.aspx
** See the Win10 excerpt from above msdn link.
Huh, didn't know that. Thanks for pointing that out!Quote:
Originally Posted by Tech99
The problem of missing files is not restricted to just remote shares. Performing an NT enumeration (no recursion into subfolders) on my Win System32 folder found 505 objects in the root but the count should be about 3900! Not even close! See if you can reproduce this result on your computers.
EDIT: Actually this was probably my doing when I was monkeying with buffer sizes... What stinks is that NTQueryDirectoryFile simply returns the number of files whose information fits into the specified buffer size with not notice passed back to me that this happened. How am I to know that my buffer was too small and that there are more files?
I check the NTStatus return codes and the IoStatus object return values but the return values never notify me of a problem. My understanding was that this was supposed to notify me if the buffer was too small and then I simply resize the buffer and try again. ???
Here's another thread which uses NTQueryDirectoryFile. It's a little different in that it's getting Pipes not files but lots of the code is the same. Might be something useful in here for anyone who stumbles on this in the future...
http://www.vbforums.com/showthread.p...-a-little-help
Well the system folder is tricky. Have you tried Wow64DisableWow64FsRedirection? I've searched paths with 5000+ files without the count even being off by 1.
Also we should probably start doing time comparisons to see if this is all even worth here. GetTickCount is terribly inadequate; I like the high resolution timing class from VBSpeed, which uses QueryPerformanceCounter.
Yes, I turn off FS redirection on 64 bit OS. Good thought but that's not the issue.Quote:
Originally Posted by fafalone
I thought the problem I was having was due to the buffer size but I've been able to reproduce the issue with a pretty massive buffer so that's not the issue.
My plan was definitely to do timings but only once I got it working properly. It doesn't matter to me how fast it is if I can't get all the files. Also, remember I'm not just getting the list of file names like you are, I'm getting all of the file details via FILE_DIRECTORY_INFORMATION. For now I'm going to step away from this and focus on other areas of my project. Do appreciate all of your help with this. Hope that the dialog and code in here helps someone else with this.
btw... does not work with Samba (unix/linux) shares, fbi.FileAttributes value is -357103104 (0xEAB70A00).Quote:
Originally Posted by AAraya
Instead of that the NtQueryFullAttributesFile might do...
Good to know. I play only in the Windows world so that doesn't really impact me but it may bite others. I don't have access to those types of shares so perhaps you can try out the NtQueryFullAttributesFile suggestion?Quote:
Originally Posted by Tech99
Well don't worry, I'm not a big fan of giving up on bizarre problems so I'll keep working on this. Just one more question, the file discrepancy on local disks, has that happened anywhere outside of C:\Windows and does it happen with the other strangely structured folders (Program Files, Program Files (x86), and Users)?
Tested that code from post 43, it 'seems to work' on local folders*, but not in shares**.
* Folders under \windows directory not necessarily report correct count, fex. sysWOW64 reports correct count, but system32 do not.
** Does not enumerate files/folders from shares correctly, fex. W28K share with over 45 folders and over 50K files, return only 1 as file count ie. enumerates only X:\sharename\. <- X:\sharename\and one dot.
** Samba share with 37 folders and 29K files, enumerates 119 files (both dot folders ie. '.' and '..' and 117 files).
Here is an assembler version (DTLight2.zip), have to analyze how this does use NTQueryDirectoryfile API, but fast it sure is - works on shares also.
http://www.masmforum.com/board/index...3124#msg143124
With the system folders, I'm getting STATUS_ACCESS_DENIED errors; that would throw off the count.
Edit: Do you think you could translate the main calls to NtQueryDirectoryFile Tech99? I can't follow it in enough detail to know what the difference is; it uses the FILE_DIRECTORY_INFORMATION but so do I.