Results 1 to 28 of 28

Thread: Extracting specific information from an IEnumerable using LINQ?

  1. #1

    Thread Starter
    Lively Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    72

    Question Extracting specific information from an IEnumerable using LINQ?

    I have an IEnumerable which contains file and directory information - example below of its contents:-

    Name:  2020-11-21_13-32-35.png
Views: 440
Size:  17.4 KB

    I want to extract specific information from this IEnumerable and am assuming LINQ is the best route

    I want a list of files and a separate list of directories in a given path - but just that level, so not including sub directories etc

    So if I pass C:\ then it just returns list of files in root of C:\ and a list of directories in root of C:\ - 1 level only. If I pass C:\Windows then it returns list of files in root of C:\Windows and a list of directories in C:\Windows - again just the level given and no lower (or higher)

    The lists just need to contain the FullName of each file / directory and nothing else

    The Attributes value contains Directory if a directory so I can use that to easily determine if a file or a directory

    But I am stuck on how I just return the FullName value - AND - how to just return one level

    Example project attached shows exactly how the IEnumerable is filled using NTFSreader link

    N.B. I want to continue using Ntfs reader and NOT use IO.Directory.GetDirectories / GetFiles as NTFS reader is much quicker and doesn't have same issue with IO permissions as it gets information directly from MFT

    Anyone here got any thoughts please? Have spent some hours with Google and other forums but at a loss here - thanks

    Happy to use something other than LINQ if it works better, just assumed that would be best method as querying an IEnumerable
    Attached Files Attached Files
    Last edited by wingers; Nov 22nd, 2020 at 08:33 AM.

  2. #2
    Frenzied Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    1,335

    Re: Extracting specific information from an IEnumerable using LINQ?

    What was wrong with the code you were given in SO? You said it didn't work, but you didn't say how it failed.

  3. #3

    Thread Starter
    Lively Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    72

    Re: Extracting specific information from an IEnumerable using LINQ?

    I think I did put why it failed - "it is missing some directories where Attributes contains more than just Directory e.g. C:\ProgramData has Attributes shown as "Hidden Or Directory Or NotContentIndexed" so think the bit where it says "n.Attributes = Attributes.Directory" needs to change somehow to cover those that include .Directory rather than just ones that ONLY have .Directory"

  4. #4
    Frenzied Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    1,335

    Re: Extracting specific information from an IEnumerable using LINQ?

    OK. Well the normal pattern is to use the Enum as a mask, and compare the result to the mask itself. So instead of
    Code:
    Return n.FullName = rootDir & n.Name And n.Attributes = Attributes.Directory
    you'd use something more like:
    Code:
    AndAlso ((n.Attributes And Attributes.Directory) = Attributes.Directory)

  5. #5

    Thread Starter
    Lively Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    72

    Re: Extracting specific information from an IEnumerable using LINQ?

    So to explain further

    It correctly gets directories that just have Attributes of Directory e.g.

    Name:  1.png
Views: 267
Size:  11.5 KB

    But it doesn't get those which has multiple Attributes that include Directory e.g.

    Name:  2.png
Views: 260
Size:  12.7 KB

  6. #6

    Thread Starter
    Lively Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    72

    Re: Extracting specific information from an IEnumerable using LINQ?

    @Inferrd - sorry posted my explanation at same time as you replied - thanks I will give that a try

  7. #7

    Thread Starter
    Lively Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    72

    Re: Extracting specific information from an IEnumerable using LINQ?

    Okay from initial testing that may well have solved my problem. How would I amend below so the resulting list is in alphabetical order please?

    Code:
     Dim dirs As List(Of String) = nodes.Where(Function(n)
                              Return n.FullName = path & n.Name AndAlso ((n.Attributes And Attributes.Directory) = Attributes.Directory)
                              End Function).Select(Function(n) n.FullName).ToList
    
    Dim files As List(Of String) = nodes.Where(Function(n)
                              Return n.FullName = path & n.Name AndAlso ((n.Attributes And Attributes.Directory) <> Attributes.Directory)
                              End Function).Select(Function(n) n.FullName).ToList

  8. #8
    Frenzied Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    1,335

    Re: Extracting specific information from an IEnumerable using LINQ?

    dirs (and files) is a List(Of String). The List(Of T) Class has a Sort Method you can use (or you can use OrderBy in your LINQ before calling ToList, as you were doing in your earlier code).

    On a different note, do you find NTFSReader to be reliable? It seems to miss a lot of files/folders, and also mangle some of the path fragment names to a DOS like 8.3 scheme. First time I've played with it and I've not found any documentation, so maybe I'm doing it wrong?

  9. #9

    Thread Starter
    Lively Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    72

    Re: Extracting specific information from an IEnumerable using LINQ?

    Yes so far I have found it to be quite reliable, certainly so much quicker than using any of the System.IO methods. I spent ages testing all of the different ways of enumerating my C: drive, some methods tooks minutes to complete whereas this is a matter of seconds. From testing so far I haven't noticed any of the issues you mention. Below are some links to two great projects that use it and articles the chap wrote explaining it all - worth a look

    https://github.com/bsonnino/NtfsDirectories
    https://github.com/bsonnino/NtfsFileEnum
    https://blogs.msmvps.com/bsonnino/20...fs-structures/
    https://blogs.msmvps.com/bsonnino/20...fs-structures/

    Hope they help

    I tried using OrderBy but couldn't get the code correct, and can't find the earlier code you mentioned? I would rather do it with OrderBy if possible

  10. #10
    Frenzied Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    1,335

    Re: Extracting specific information from an IEnumerable using LINQ?

    OK. Thanks for the links. Will check them out later.

    You're actually using the OrderBy in your GetFileNodes method to order the IEnumerable(Of INode) returned by the NtfsReader:
    Code:
    .OrderBy(Function(n) n.FullName).ToList()
    So the list of INode is already ordered alphabetically by the full path name, and so the files and dirs Lists should also be already ordered alphabetically.

    If you want to further use OrderBy on a List(Of String) then you'd just use:
    Code:
    .OrderBy(Function(n) n)
    although you might want to add some logic to deal with ignoring character casing order.

  11. #11

    Thread Starter
    Lively Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    72

    Re: Extracting specific information from an IEnumerable using LINQ?

    No problem at all, have spent weeks working with it now and those project and the author were very helpful

    Ah yes, sorry forgot I had used it in example project

    Oddly enough even though the original list of INode was sorted alphabetically the files and dirs Lists were NOT coming out in alphabetical order

    Thanks for that

    As it happened it wasn't sort order which was causing the problem I had, it was actually because I was replacing a method that had used Directory.GetDirectories and Directory.GetFiles so this automatically knew to assume the "path" variable need to end with "" whereas the LINQ method didn't so I had to check for and add a "" to the end of the "path" variable and it then worked fine

    I just did below before the Dim dirs As List(Of String) / Dim files As List(Of String) lines

    Code:
       If path.EndsWith("\") = False Then path = path & "\"
    But if there is a way of doing this automatically in the LINQ statement then even better - but it is at least now working as I needed

    Thank you for your help, was not getting very far on SO and I prefer being able to reply fully in posts rather than the SO method of short comments

  12. #12
    Frenzied Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    1,335

    Re: Extracting specific information from an IEnumerable using LINQ?

    Aye, you can't beat a good forum

  13. #13

    Thread Starter
    Lively Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    72

    Re: Extracting specific information from an IEnumerable using LINQ?



    To explain fully (for future reading) my last comment

    The below worked fine when "path" was "C:\" or "C:\Windows\" but doesn't work if it is "C:\Windows" e.g. if no \ on end

    Code:
    Dim dirs As List(Of String) = nodes.Where(Function(n)
                                                                          Return n.FullName = path & n.Name AndAlso ((n.Attributes And Attributes.Directory) = Attributes.Directory)
                                                                      End Function).Select(Function(n) n.FullName).OrderBy(Function(n) n).ToList
    
    Dim files As List(Of String) = nodes.Where(Function(n)
                                                                           Return n.FullName = path & n.Name AndAlso ((n.Attributes And Attributes.Directory) <> Attributes.Directory)
                                                                       End Function).Select(Function(n) n.FullName).OrderBy(Function(n) n).ToList

  14. #14
    Frenzied Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    1,335

    Re: Extracting specific information from an IEnumerable using LINQ?

    Quote Originally Posted by wingers View Post
    To explain fully (for future reading) my last comment

    The below worked fine when "path" was "C:\" or "C:\Windows\" but doesn't work if it is "C:\Windows" e.g. if no \ on end
    Yeah, I was a bit puzzled there.

    With a little refactoring, you can use the Path.Combine Method from the Path Class to build your paths and sort that out for you automatically inside the LINQ:
    Code:
    Dim yourSearchPath = "G:\Windows"
    
    
    Dim dirs As List(Of String) = nodes.Where(Function(n) n.FullName = Path.Combine(yourSearchPath, n.Name) AndAlso
                                              (n.Attributes And Attributes.Directory) = Attributes.Directory).
                                              Select(Function(n) n.FullName).
                                              ToList
    
    
    Dim files As List(Of String) = nodes.Where(Function(n) n.FullName = Path.Combine(yourSearchPath, n.Name) AndAlso
                                               (n.Attributes And Attributes.Directory) <> Attributes.Directory).
                                               Select(Function(n) n.FullName).
                                               ToList
    Note that because Path is the name of a Class, I'd normally avoid using path as a variable name, so I've renamed your path variable to yourSearchPath, but you should use whatever name makes sense to you (and those reading your code ;-).
    Last edited by Inferrd; Nov 22nd, 2020 at 01:47 PM.

  15. #15

    Thread Starter
    Lively Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    72

    Re: Extracting specific information from an IEnumerable using LINQ?

    Thanks

    Doing it that way seems to be much slower than just adding \ to end of path if not already got one

    Running it with your new LINQ code took 21 seconds to run and populate my TreeList, running it the other way took 11 seconds

    I can only assume something to do with the Path.Combine - but thank you as is handy to know both methods

  16. #16
    Frenzied Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    1,335

    Re: Extracting specific information from an IEnumerable using LINQ?

    Yeah, it will be slower as that method is called for every node in the list, whereas adding the \ to the end of the search path is only done the one time. That said, I wouldn't have expected an extra 11 seconds! Using the Path.Combine, it takes me about 250 milliseconds to search through about 200,000 nodes, twice.

  17. #17

    Thread Starter
    Lively Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    72

    Re: Extracting specific information from an IEnumerable using LINQ?

    I think it is because it is looping through that sub many times in my code to then add folder structure to a tree

  18. #18
    Frenzied Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    1,335

    Re: Extracting specific information from an IEnumerable using LINQ?

    Quote Originally Posted by wingers View Post
    I think it is because it is looping through that sub many times in my code to then add folder structure to a tree
    Do you mean that you are fetching the dirs and files lists for many search paths? If so how many?

    I notice that the Path.Combine approach does not scale well. For 100 search paths, I'm seeing times of about 10 seconds for your append the \ method, and about 26 seconds for the Path.Combine method. Kinda worse than I was expecting!

    However, you might be able to speed things up, depending on what you are doing.

    I created a LookUp from the nodes, and that took about 9 milliseconds for the same 100 search paths.

    (Populating TreeViews might slow things down, though. Not tried.)

  19. #19

    Thread Starter
    Lively Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    72

    Re: Extracting specific information from an IEnumerable using LINQ?

    I am using a third party control - DevExpress TreeList as more customizable for my needs.

    It is loading the levels of the filesystem "on demand" so to speak using what DevExpress called Virtual Mode - so yes I get all the node information using NTFSreader, and then get first levels of folders/files so it can create tree, it then loops though same sub x number of times depending on how many folders it found in root (or chosen start location) to retrieve next level of folder/file information ready to show when you expand the tree.

    Works quite well and using the Dev control I can customize the look to suit my requirements - which in this case is an equivalent of TreeSize so sorted by total size of directories (including sub folders / files) and then a bar graph to illustrate sizes

    Name:  tree.jpg
Views: 228
Size:  39.9 KB
    Last edited by wingers; Nov 22nd, 2020 at 03:48 PM.

  20. #20
    Frenzied Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    1,335

    Re: Extracting specific information from an IEnumerable using LINQ?

    Looks good, even with the evil this forum does to images (resizes them to max 600px wide).

    You must be recursing through all the folders on the drive to get the sizes? Is that what takes the time?

  21. #21

    Thread Starter
    Lively Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    72

    Re: Extracting specific information from an IEnumerable using LINQ?

    Yes it displays the tree and then in the background it calculates the sizes of each folder (and its contents) from the information held in the nodes IEnumerable

    Code:
    totalsum = nodes.Where(Function(n) n.FullName.StartsWith(d.FullName & "\")).Sum(Function(n) n.Size)
    It then gets size of parent folder and then the percentage for each folder is shown relevant to size of parent

    Other information such as last access, last write time, file count, dir count is also gathered in background and added to grid as and when it gets results. That way it doesn't affect the loading of the tree nodes etc

  22. #22
    Frenzied Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    1,335

    Re: Extracting specific information from an IEnumerable using LINQ?

    Sounds like you've got it sussed

  23. #23

    Thread Starter
    Lively Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    72

    Re: Extracting specific information from an IEnumerable using LINQ?

    I hope so, has taken some time, and getting this bit working was hopefully the last major hurdle

  24. #24

    Thread Starter
    Lively Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    72

    Re: Extracting specific information from an IEnumerable using LINQ?

    Quick related question

    I am struggling (for some reason) to be able to retrieve a specifc value from the IEnumerable when matching another value

    In english based on image in first post - I want LastAccessTime when FullName equals the value I provide

    Basically want to perform a lookup in the IEnumerable to get values for specific items

    Thought it would be something like this but not working

    Code:
    Dim lastwritetime = nodes_test.Where(Function(n) n.FullName.Equals(dr.FullName)).Select(Function(a) a.LastChangeTime)

  25. #25
    Frenzied Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    1,335

    Re: Extracting specific information from an IEnumerable using LINQ?

    Quote Originally Posted by wingers View Post
    In english based on image in first post - I want LastAccessTime when FullName equals the value I provide
    Try something like:
    Code:
    Dim lastChangeTime As Date? = nodes.FirstOrDefault(Function(n) n.FullName.Equals(mypath))?.LastChangeTime
    Note the 2 question marks. lastChangeTime is a Nullable Date, and will be set to Nothing if the path you provide isn't found.

  26. #26

    Thread Starter
    Lively Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    72

    Re: Extracting specific information from an IEnumerable using LINQ?

    That doesn't seem to work as just returning Nothing

    Name:  3.jpg
Views: 162
Size:  4.9 KB

  27. #27
    Frenzied Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    1,335

    Re: Extracting specific information from an IEnumerable using LINQ?

    Quote Originally Posted by wingers View Post
    That doesn't seem to work as just returning Nothing
    I've tested it and it does work. If it's returning Nothing, then the path you are looking for doesn't exist in the node collection. You're using .Equals(filename) in your code. Make sure you are passing a full path, not just a file name.

  28. #28

    Thread Starter
    Lively Member
    Join Date
    Jul 2011
    Location
    UK
    Posts
    72

    Re: Extracting specific information from an IEnumerable using LINQ?

    Ignore me - glitch my side, should learn to read properly!!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width