Results 1 to 32 of 32

Thread: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

  1. #1

    Thread Starter
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,652

    Lightbulb [VB6] Event Tracing for Windows - Monitoring File Activity with ETW


    Name:  evntrace.jpg
Views: 2268
Size:  41.4 KB
    VBEventTrace v2.1
    Using Event Tracing for Windows in VB6
    (Updated 2022 May 15)

    Event Tracing for Windows (ETW) is a notoriously complex and unfriendly API, but it's extremely powerful. It allows access to messages from the NT Kernel Logger, which provides a profound level of detail about activity on the system. It provides details about many types of activity, but this first project will focus on File Activity. I also plan to follow this up with a monitor for TcpIp and Udp connections.

    Given the complexity and unfriendliness that's given it the reputation of the world's worst API, why use it? You can find many projects that monitor file activity, using methods like SHChangeNotify, FindFirstChangeNotification, and monitoring open handles. But the reality is these are all high level methods that don't cover quite a bit of activity. The kernel logger shows activity coming from low level disk and file system drivers. This project started with me wanting to know what was causing idle hard drives to spin up, and none of the higher levels methods offered a clue. Programs like ProcessHacker and FileActivityView use the NT Kernel Logger as well, but I wanted two things: Better control over the process, and doing it in VB6. Why? Well, if you've seen my other projects, you know I'm excessively fond of going way beyond what VB6 was meant for both in terms of low level stuff and modern stuff.

    Intro

    This project tracks most of the FileIo events, providing a great deal of control over what events you watch and filtering them to find what you're looking for. It also looks up name and icon of the process that generated the activity (not always available). With no filtering or only light filtering, a tremendous amount of data is generated. The VB TextBox and ListView simply could not keep up with the rapid input, and all sorts of memory and display issues ensued where text and List Items disappeared. So while the project was already complicated to begin with, the only way to cope with this was to use an API-created Virtual ListView (created via API and using the LVS_OWNERDATA style so it only includes the data currently being displayed).

    How It Works
    Have a read here for an introduction to setting up a Kernel Logger with ETW, and then realize it's even *more* complicated than that article suggests, because of some VB6 specific issues, and the hell on earth involved in interpreting the data.

    Just starting the tracing session has 3 steps. You start with the EVENT_TRACE_PROPERTIES structure. Now, it's daunting enough on it's own. But when you read the article linked, you realize you have to have open bytes appended *after* the structure for Windows to copy the name into. Then the article doesn't touch on a recurring theme that was the source of a massive headache implementing it... in other languages, structures get automatically aligned along 8 byte intervals (a Byte is 1 byte, an Integer 2 bytes, a Long 4 bytes... alignment is making each Type a multiple of a certain number of bytes). Not so in VB. It took quite a bit of crashing and failures to realize this, then properly pad the structures. The code uses it's own structure for the StartTrace function that looks like this:

    Code:
    Public Type EtpKernelTrace
        tProp As EVENT_TRACE_PROPERTIES
        padding(0 To 3) As Byte
        LoggerName(0 To 31) As Byte 'LenB(KERNEL_LOGGER_NAMEW)
        padding2(0 To 3) As Byte
    End Type
    Needed to include 4 bytes of padding after the structure, then add room for the name, then make sure it's all aligned to 8 byte intervals. Now we're ready to go, with tStruct being a module-level EtpKernelTrace var:

    Code:
    With tStruct.tProp
        .Wnode.Flags = WNODE_FLAG_TRACED_GUID
        .Wnode.ClientContext = 1&
        .Wnode.tGUID = SelectedGuid
        .Wnode.BufferSize = LenB(tStruct)
        .LogFileMode = EVENT_TRACE_REAL_TIME_MODE 'We're interested in doing real time monitoring, as opposed to processing a .etl file.
        If bUseNewLogMode Then
            .LogFileMode = .LogFileMode Or EVENT_TRACE_SYSTEM_LOGGER_MODE
        End If
        'The enable flags tell the system which classes of events we want to receive data for.
        .EnableFlags = EVENT_TRACE_FLAG_DISK_IO Or EVENT_TRACE_FLAG_DISK_FILE_IO Or EVENT_TRACE_FLAG_FILE_IO_INIT Or _
                        EVENT_TRACE_FLAG_DISK_IO_INIT Or EVENT_TRACE_FLAG_FILE_IO Or EVENT_TRACE_FLAG_NO_SYSCONFIG
        .FlushTimer = 1&
        .LogFileNameOffset = 0&
        .LoggerNameOffset = LenB(tStruct.tProp) + 4 'The logger name gets appended after the structure; but the system looks in 8 byte alignments,
                                                    'so because of our padding, we tell it to start after an additional 4 bytes.
    End With
    
    'We're now ready to *begin* to start the trace. StartTrace is only 1/3rd of the way there...
    hr = StartTraceW(gTraceHandle, StrPtr(SelectedName & vbNullChar), tStruct)
    This begins to start a trace session. There's SelectedGuid and SelectedName because there's two options here. In Windows 7 and earlier, the name has to be "NT Kernel Logger", and the Guid has to be SystemTraceControlGuid. If you use that method, there can only be 1 such logger running. You have to stop other apps to run yours, and other apps will stop yours when you start them. On Windows 8 and newer, there can be several such loggers, and you supply a custom name and GUID, and inform it you want a kernel logger with the flag added with bUseNewLogMode. This project supports both methods. The EnableFlags are the event providers you want enabled. This project wants the disk and file io ones, but there's many others. Onto step 2...

    Code:
    Dim tLogfile As EVENT_TRACE_LOGFILEW
    ZeroMemory tLogfile, LenB(tLogfile)
    tLogfile.LoggerName = StrPtr(SelectedName & vbNullChar)
    tLogfile.Mode = PROCESS_TRACE_MODE_REAL_TIME Or PROCESS_TRACE_MODE_EVENT_RECORD 'Prior to Windows Vista, EventRecordCallback wasn't available.
    tLogfile.EventCallback = FARPROC(AddressOf EventRecordCallback) 'Further down, you can see the prototype for EventCallback for the older version.
    gSessionHandle = OpenTraceW(tLogfile)
    We have to tell it *again* we want to use real time mode, not a .etl log file, and at this point we supply a pointer to a callback that receives events. This project uses a newer type of callback available in Vista+, but has prototypes for the older one. Like a WndProc for subclassing, this has to be in a standard module (.bas); to put it in a class module/form/usercontrol, you'd need the kind of self-subclassing code like you find on the main form (but be careful copying/pasting that, it's been slightly modified and only works with Forms).

    The final step is a single call: To ProcessTrace. Only then will you begin receiving events. But of course, this simple call couldn't be simple. ProcessTrace doesn't return until all messages have been processed, which in a real-time trace means indefinitely until you shut it off. So if you call it, execution stops. In that thread. In other languages, spinning off a new thread to call ProcessTrace is easy. In VB, it's painful. This project makes use of The trick's VbTrickThreading project to launch a new thread for the ProcessTrace call. The downside here is that means event tracing is only possible in a compiled exe, making debugging difficult.

    Once you've called ProcessTrace, your callback begins receiving messages. We need to match them up with their provider, and then check the OpCode...

    Code:
    Public Sub EventRecordCallback(EventRecord As EVENT_RECORD)
    '...
    If IsEqualIID(EventRecord.EventHeader.ProviderId, DiskIoGuid) Then
        iCode = CLng(EventRecord.EventHeader.EventDescriptor.OpCode)
        
        'Some events use the same MOF structure and are processed similarly, so we group them together and separate
        'the codes for filtering and logging later.
        If (iCode = EVENT_TRACE_TYPE_IO_READ) Or (iCode = EVENT_TRACE_TYPE_IO_WRITE) Then
    The EVENT_RECORD structure is also a nightmare. Many different parts of it had to having alignment padding added, and it tripped me up for a good long while. Extra thanks to The trick for helping me figure out the right alignment on this part.

    From here, we're ready to process the data. The raw data is returned in MOF structures, e.g. this one for one of the Open/Create messages. There's ways to automate the processing of them, but that makes everything so far seem simple, and is the domain for a future project. For now, we manually process the raw data, which we copy from the pointer in .UserData in the event record. The documentation doesn't mention *at all* that even if you're running a 32bit application, these structures have 64bit sizes on 64bit Windows. The official documentation doesn't note which "uint32" types are pointers, and thus are 8 bytes instead of 4, so I had to go digging in some deep system files. The original 32bit structures are all included, but currently this project only works on 64bit Windows. It's possible to tell automatically via flags in the event record... perhaps in the future. EventRecord.EventHeader.Flags has flags EVENT_HEADER_FLAG_[32,64]_BIT_HEADER.

    Here what the File Open/Create structure looks like, and how we set it up:

    Code:
    Public Type FileIo_Create64 'Event IDs: 64
        IrpPtr As Currency
        FileObject As Currency
        ttid As Long
        CreateOptions As CreateOpts
        FileAttributes As FILE_ATTRIBUTES
        ShareAccess As Long
        OpenPath(MAX_PATH) As Integer
    End Type
    Fortunately VB has the Currency data type, which we also used for our event trace handles, which is 8 bytes. We can use this because there's no point where we have to interact a numeric representation of the value... it's just all raw bytes behind the scenes. Unfortunately, FileAttributes is only what's passed to the NtOpenFile API and not an actual query of the file's attributes, so is almost always 0 or FILE_ATTRIBUTES_NORMAL. We pick MAX_PATH for the size of the array, because using a fixed-size array avoids VB's internal SAFEARRAY type, which would make copying a structure from a language without it much more complicated. Converting a string of integer's to a normal string is trivial, but the real problems comes when you see what it is: files names look like \Device\HarddiskVolume1\folder\file.exe. To convert those into normal Win32 paths the project creates a map by querying each possible drive letter in the QueryDosDevice API, which returns a path like that for each drive.

    Not all events contain a file name, so the project stores a record with the FileObject, which allows us to match other operations on the same file, and get the name. The documentation says we're supposed to receive event code 0 for names... but I've never seen that message come in. Perhaps on earlier Windows versions.

    Perhaps the biggest problem in processing the data is that while there's an ProcessID and ThreadID in the event record's header, the process id is very often -1. Sometimes that information is returned in other events. This project goes through incredible lengths to correlate every with every other event in order to track down the process whenever possible. So many events will display -1 at first, and get updated later.

    There's still a lot of work to be done in process attribution, and getting info about files already open before the trace starts. I attempted to copy ProcessHacker's use of a KernelRundownLogger, but so far have not been successful. I'll be look at other methods, but if I didn't put out a Version 1, who knows how long it would be.

    Once we've captured the events, we store it in a the ActivityLog structure, which is the master data store for what's displayed on the ListView.

    Options

    You can see in the screenshot a number of options. There's the main controls for the trace; you don't really need to worry about 'Flush', it's there for completeness and shouldn't be needed. Stop is always enabled because in the event of crashes, you can stop previous sessions. You can save the trace; it saves what you see in the ListView, tab separated. There's options for which events you want to capture, whether to use the new logger method described earlier (Win8+), and the refresh interval for the ListView. The items aren't added to the ListView; they're stored in the ActivityLog structure, and the ListView is in virtual mode, so it only asks for what it's currently displaying. The refresh interval is how often it checks for new events and sets the last one as visible, creating a view that is always scrolled to the bottom but without the invisible items stored in the ListView itself, dramatically improving speed. (The greyed out option is for future work, not currently implemented)

    Very important is the filtering system, if you're looking for certain activity. Each field allows multiple entries separated with a | (bar, it also accepts broken bars found on some keyboards). There's a button that displays a message explaining the syntax and the flow... the first thing checked is whether it's from a process we're interested in based on the process options. You can use DOS wildcards in the Process name field and File name fields, but not the paths at this point... for now the paths are strictly checked on a 'Starts with...' basis. After checking the process, then it checks 'Path must match', then 'Exclude paths', then 'File name must match', finally 'Exclude file name'.

    Finally on the right there's a message log, which displays information about starting/stopping the trace, when a different function has correlated a previously unidentified process id, and any errors that arise.

    Not shown: If you right click the ListView, there's a popup menu with options to open the selected items, show the selected items in Explorer, copy selected file names, copy all file names, copy the selected lines (tab separated), copy all lines, show properties of the process, and show the process in Explorer.

    Requirements
    PLEASE TAKE NOTE. This program has atypical requirements.

    -Windows Vista or newer 64bit. Although like all VB6 apps the app itself is 32bit, it handles data structures generated by the system, and is currently only coded to handle 64bit structures. To run on 32bit Windows, use the regular MOF structures instead of the x64 ones (and change the size checks at the start of each processing routine).

    -This program can only start event tracing when compiled, due to the need for multithreading that cannot be done in a single thread.

    -The NT Kernel Logger requires additional permissions- you need to be a member of the Administrators group (but not necessarily run as admin), or be a member of the Performance Log Users group, or otherwise have permission to enable the SeSystemProfilePrivilege.

    -There are no external dependencies. However, the demo uses a manifest for Common Controls 6.0 styles, and it's advised you also use them in any other project.

    -Unicode is supported in the ListView for displaying files etc, but the filter TextBoxes are just regular VB ones, so you'd need to replace those to use Unicode in filtering.

    Windows 10 is strongly recommended. I have not had the opportunity to test this on other OSs.

    This API is *extremely* complicated and finicky, so there's bound to be bugs. Especially on other Windows versions. Let me know, I'll see what I can do.



    Project Update: Version 1.1 uses a separate activity log for the display that's synchronized in a critical section-- previously I had not realized the ProcessTrace thread was the one that sent events into the callback, meaning both threads were constantly reading from the ActivityLog structure. Very high event rates, particularly rundowns if you enabled the old logger, resulted in an access violation and app crash when one thread attempted to use the memory in use by the other. Now when the data is copied from the ActivityLog structure, it's done inside a critical section, which locks access so the other thread waits rather than crashes accessing in use memory.
    Also corrected the bug with the old logger mode setting, enabled the 'Ignore rundown events' option-- the rundown only happens with the old mode, and tried to gain back some speed by using a different way of converting the fixed-length wchar buffers with filenames to variable length strings in the activity log, and using a much faster version of Replace() for resolving Win32 paths.

    Project Update: Version 1.2 - While for whatever reason start a 2nd logger to get a rundown of open handles just wasn't working, I figured out a way to trigger it at the start, in a way that works for both the single (Win7) and shared (Win8+) logging modes. This means you're now able to see disk read/writes to files that were already open before the trace started; a lot of events were missed like this. You'll now see the $Mft and other special NTFS location read/write totals like ProcessHacker. Added an event toggle for this. Also fixed some bugs, added some defs, and enforced numbers only on the refresh interval textbox, which can be used to modify a running session now by pressing enter.

    Project Update: Version 2.0 - This version brings extensive new features, improvements and bug fixes. A major bug was updates were never synchronized, so only displayed if they occurred between syncs; this left most read/writes incorrect, and an update was the only way they ever got set at all, since they weren't initially added. On the new feature front, there's now an optimized mode for tracking only activity to the disk (rather than including cached activity). Running as administrator is no longer required *if* you're a member of the administrators group, the Performance Log Users group, or another group with permission to enable the SeSystemProfilePrivilege. There's now an option to merge similar activities; e.g. there's new columns for open and delete count, for programs repeatedly opening the same file. This will merge IO as well. After seeing a research paper suggest it, I implemented thread context switch tracking for process attribution. It's completely useless. Only accurate when the events are already attributed. Nonetheless, I left the option in for those who want to play around with it (it's in the new additional options popup when you click 'More'). Options to control when rundowns are done, and control process caching. See the full changelog for a few more.

    Version 2.1 - Fix for incorrect DirEnum pattern and infoclass; scrolling will now stay where it is when you scroll away from the bottom. When you put it back to the bottom, it will resume keeping it there.



    There's now an x64 version of this project, thanks to twinBASIC, which is impressively far along.

    [twinBASIC] x64-compatible port of Event Tracing for Windows File Activity Monitor
    Attached Files Attached Files
    Last edited by fafalone; Jul 28th, 2022 at 04:46 AM. Reason: Minor edits to documentation; no code change

  2. #2

    Thread Starter
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,652

    [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    Full changelog:


    Code:
    'Version 2.1: -If you scroll away from the bottom, it will stay where you scroll
    '              to instead of immediately going back to the bottom.
    '
    '             -Bug fix: FileIo_DirEnum pattern and InfoClass were reported wrong.
    '
    'Version 2.0: -Running as administrator is no longer strictly required. However,
    '              you must still be a member of the administators group, a member
    '              of the Performance Log Users group, or another user/group that has
    '              access to the SeSystemProfilePrivilege, which the code now calls
    '              AdjustTokenPrivileges to set.
    '
    '             -Project has been optimized to track disk read/write alone. Because
    '              of caching, FileIo events don't neccessarily trigger disk activity,
    '              so if you only wanted to watch disk activity, disabling everything
    '              except DiskIO will enable DiskIO Exclusive Mode; FileIo events not
    '              activated by the DiskIO flags will be disabled (and can't be enabled
    '              while the trace is in progress, since this is done in the inital
    '              flags), and DiskIO will properly attribute IO to Create events,
    '              where when disabled, create events use FileIO_ReadWrite, which
    '              doesn't reflect disk io because of caching.
    '              DiskIO includes open/delete events that trigger disk activity.
    '
    '             -Added option to merge certain activities by the same process on the
    '              same file. Create/open/read/write/delete can be merged. Multiple
    '              events may still exist where an initial pid was -1 and then updated.
    '              An additional option restricts this to same opcode only (combining
    '              only read/write opcodes).
    '
    '             -The default configuration is now DiskIO Exclusive Mode with merging
    '              enabled. Remember that enabling any FileIO operation takes it out
    '              of that mode, and that FileIO operations do not represent disk read
    '              and write activity due to caching.
    '
    '             -There's now an option to disable the initial rundown; you'll miss
    '              activity on open files during the trace, but idle disks won't spin
    '              up. If this option is used, the rundown may be performed at the end
    '              of the trace (always, if old logger mode, optionally with new), and
    '              the disk io logs will be scanned for missed events, and added then.
    '
    '             -Process caching now has an option to be disabled (never, always, or
    '              only when running in DiskIO Exclusive Mode), for situations where
    '              process id use may be a problem (processes rapidly being created
    '              and exiting).
    '
    '             -Context switch tracking for process attribution is now implemented,
    '              however I've found it entirely useless. It only works in cases where
    '              the pid was already returned.
    '
    '             -Log sync interval can now be adjusted.
    '
    '             -Changed how filenames are read to no longer use a fixed buffer;
    '              the MOF structure now uses String (variable length) and is set
    '              by a Fill_<type> routine in modEventTrace that first copies the
    '              data into a variable length byte array, then sets the MOF type
    '              String. This allows long file name support without using a
    '              massive fixed buffer.
    '
    '             -There's now an event for unattributed FileIo_ReadWrite events like
    '              there is for DiskIo_ReadWrite, to catch read/writes on files already
    '              open before the trace started.
    '
    '             -So few events were in the type enum I got rid of the enum and made
    '              them byte constants to improve performance. Also added constants
    '              for the FileIo opcodes.
    '
    '             -Added some more flags in the Misc column.
    '
    '             -Cleaned up some lingering potentially unsafe thread data accesses.
    '
    '             -There's reports of poor critical section performance on Win8+ with
    '              the default dynamic spin count adjustment; trying 4000 instead per
    '              MSDN recommendation as optimal; will see how it goes.
    '
    '             -Bug fix: AddActivity didn't add read/write size, so it was only ever
    '                       non-zero if an update came in, which was generally never for
    '                       FileIo since MSDN lied about FileKey being what you use to
    '                       correlate FileIo_ReadWrite events.
    '
    '             -Bug fix: While read/write totals were updated, sync between the
    '                       ActivityLog struct and threadmain copy consisted only of
    '                       adding new items, so only updates that occured between sync
    '                       calls were ever updated on the ListView.
    '                       This also impacted pid updates.
    '
    'Version 1.2: -Now able to log read/write to files open before the trace started
    '              by triggering a rundown. This comes up as 'DiskIO', which can
    '              also be filtered. If you disable all events except DiskIO, the
    '              list should be similar to what ProcessHacker does.
    '
    '             -Numbers only enforced on refresh interval textbox; can now update
    '              while running by pressing enter. (Note: All other checkboxes except
    '              'Use new logger' take effect immediately while running too, and
    '              the 'Update' button on filters is for when it's running.)
    '
    '             -Fixed a number of small bugs.
    '
    '             -Added additional declares and corrected enums for modEventTrace.
    '
    'Version 1.1: -Fixed bug where old logger option didn't worked, sped up code
    '              significantly by using a different wchar->string method and
    '              alternative to VB's Replace() function, and added thread safety
    '              to prevent crashes from too high an incoming event rate.
    '
    '             -There's now a separate activity log that the ListView displays
    '              that's synchronized to the primary one inside a critical section
    '              so the same memory isn't accessed by both threads at the same time,
    '              which caused crashes in certain circumstances, on slower systems,
    '              or very high event rates (such as the rundown with the old logger).
    Known Issues

    - If the program crashes for some reason, there's currently a bug with shutting down and restarting the previous session. Re-launch the program, click start, click stop, then restart the program. It will work after that. I'm not sure why the call doesn't work the first time or without restarting.

    - There are possible issues with reporting Events Lost. There may be some scenarios where that value is corrupt, because it seems impossible to lose so many events without elevated CPU use or maxing out handling capacity.
    Last edited by fafalone; May 15th, 2022 at 08:50 PM.

  3. #3

    Thread Starter
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,652

    [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    (also reserved)

  4. #4

  5. #5

    Thread Starter
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,652

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    Thanks. Really appreciate all the help getting it up and running. I'm going to dig into those 64-bit structures I was asking about in the future in pursuit of figuring out the pids that are still -1 and some issues with files opened before the trace starts, but wanted to at least post a rough draft.

  6. #6
    Hyperactive Member
    Join Date
    Mar 2019
    Posts
    416

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    Wonderful contribution. I have wanted to use this API for some time however I found the structures too confusing. Thank you

  7. #7

    Thread Starter
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,652

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    Thanks.

    It was the same for me... numerous times over the years I've wanted to try using it myself, but started looking at the documentation and thought 'holy **** what??'. Even the SDK samples are either irrelevant to kernel logging or even more complicated than what I've done so far. I finally committed to giving it a serious attempt. It was just pure luck that it even came together... a single 1 line comment in a reply on a general ETW question on Stack Overflow was how I found out the structures needed to be aligned at 8 bytes. Never would have thought of that on my own; I've encountered 4-byte alignment issues before, but not 8 bytes.

    If this doesn't answer my drive spinning up issue, which it might not because FileActivityView hasn't, just wait til what's next on the list... a file system filter device driver. Ever since The trick showed making a kernel mode driver in VB6 was possible, that's been on my mind.

  8. #8
    Hyperactive Member
    Join Date
    Jun 2016
    Location
    España
    Posts
    506

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    Very good contribution.
    I have problems many times it doesn't work and other times it closes by itself.
    It may be a thing of my pc lately it works badly.
    I will investigate later.

    If this doesn't answer my drive spinning up issue, which it might not because FileActivityView hasn't, just wait til what's next on the list... a file system filter device driver. Ever since The trick showed making a kernel mode driver in VB6 was possible, that's been on my mind.
    continue with your thoughts I trust you you are very intelligent.

    a greeting

  9. #9

    Thread Starter
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,652

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    Right now there's an issue if events come in too quickly, the two threads collide while accessing the ActivityLog structure, and that causes a crash. I hadn't realized that the thread ProcessTrace starts is what sends events to the callback, so you've got the callback processing reading/writing that structure while on the other thread, the form is reading it for display in the ListView. I'm working on putting a solution using critical sections to lock access to other threads while in use and redesigning things to have minimal crossover; in the mean time I'd recommend using filters to narrow what you're capturing to minimize how many events per second get recorded.

  10. #10

    Thread Starter
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,652

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    Project Updated

    Version 1.1 uses a separate activity log for the display that's synchronized in a critical section-- previously I had not realized the ProcessTrace thread was the one that sent events into the callback, meaning both threads were constantly reading from the ActivityLog structure. Very high event rates, particularly rundowns if you enabled the old logger, resulted in an access violation and app crash when one thread attempted to use the memory in use by the other. Now when the data is copied from the ActivityLog structure, it's done inside a critical section, which locks access so the other thread waits rather than crashes accessing in use memory. A critical section is also used for updating the message log, which is then copied anew every time with a separate timer that refreshes it every 2 seconds.

    Also corrected the bug with the old logger mode setting, enabled the 'Ignore rundown events' option-- the rundown only happens with the old mode, and tried to gain back some speed by using a different way of converting the fixed-length wchar buffers with filenames to variable length strings in the activity log, and using a much faster version of Replace() for resolving Win32 paths.

    This should eliminate most crashes. But remember if it still does for some reason,
    - If the program crashes for some reason, there's currently a bug with shutting down and restarting the previous session. Re-launch the program, click start, click stop, then restart the program. It will work after that. I'm not sure why the call doesn't work the first time or without restarting.

  11. #11
    Hyperactive Member
    Join Date
    Mar 2019
    Posts
    416

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    What would this structure look like. It enables some PID based filtering which I may want to use. See filter descCount and filterDesc

    https://docs.microsoft.com/en-us/win..._properties_v2

    PS - How often do your drives spin up?

    Code:
    typedef struct _EVENT_TRACE_PROPERTIES_V2 {
      WNODE_HEADER             Wnode;
      ULONG                    BufferSize;
      ULONG                    MinimumBuffers;
      ULONG                    MaximumBuffers;
      ULONG                    MaximumFileSize;
      ULONG                    LogFileMode;
      ULONG                    FlushTimer;
      ULONG                    EnableFlags;
      union {
        LONG AgeLimit;
        LONG FlushThreshold;
      } DUMMYUNIONNAME;
      ULONG                    NumberOfBuffers;
      ULONG                    FreeBuffers;
      ULONG                    EventsLost;
      ULONG                    BuffersWritten;
      ULONG                    LogBuffersLost;
      ULONG                    RealTimeBuffersLost;
      HANDLE                   LoggerThreadId;
      ULONG                    LogFileNameOffset;
      ULONG                    LoggerNameOffset;
      union {
        struct {
          ULONG VersionNumber : 8;
        } DUMMYSTRUCTNAME;
        ULONG V2Control;
      } DUMMYUNIONNAME2;
      ULONG                    FilterDescCount;
      PEVENT_FILTER_DESCRIPTOR FilterDesc;
      union {
        struct {
          ULONG Wow : 1;
          ULONG QpcDeltaTracking : 1;
          ULONG LargeMdlPages : 1;
          ULONG ExcludeKernelStack : 1;
        } DUMMYSTRUCTNAME;
        ULONG64 V2Options;
      } DUMMYUNIONNAME3;
    } EVENT_TRACE_PROPERTIES_V2, *PEVENT_TRACE_PROP
    Last edited by vbwins; Apr 4th, 2022 at 04:48 AM.

  12. #12

    Thread Starter
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,652

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    Why do you need that for pid based filtering; it should be coming in with the event record... EventRecord.EventHeader.ProcessID (or Event.Header.ProcessID with the old method). I guess it might be more efficient at the source, but then that's going to make it more difficult if you want to adjust it while the trace is in progress.

    You'd add
    Code:
    V2Control As Long
    FilterDescCount As Long
    FilterDesc As Long 'VarPtr to structure
    Wow As Long
    QpcDeltaTracking As Long
    LargeMdlPages As Long
    ExcludeKernelStack As Long
    That looks like it would bring the structure into 8 byte alignment, meaning you'd eliminate the 4 byte pad before the logger name.

    My drives spin up pretty much randomly. Could be 30min after spinning down, could actually go until I intentionally access it, and anywhere in between. About half the time, it's because Explorer picked a random file in a random folder (that usually hasn't been accessed in weeks or months by any app, including explorer), and decided to read a few bytes from it. Filter drivers look promising because they can actually block access instead of just report on it... but it's daunting, to say the least. The documentation is just as sparse, every example does things a different way... it's like etw on steroids.
    Last edited by fafalone; Apr 4th, 2022 at 08:37 AM.

  13. #13
    Hyperactive Member
    Join Date
    Mar 2019
    Posts
    416

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    Quote Originally Posted by fafalone View Post
    Why do you need that for pid based filtering
    Because I know what pids I am interested in before hand and the documentation advises that this is a way to reduce overhead. At least I think that is what it means. Cheers

  14. #14

    Thread Starter
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,652

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    I'm not sure 'reducing overhead' should come before 'get it working'.

    Either way, somewhere it's going to have to go through a list of pids. I'm not sure just how much overheard can be reduced by having the provider run the list for you. Might not be worth the extra trouble.

  15. #15

    Thread Starter
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,652

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    Project Update: Version 1.2

    While for whatever reason starting a 2nd logger to get a rundown of open handles just wasn't working, I figured out a way to trigger it at the start, in a way that works for both the single (Win7) and shared (Win8+) logging modes. This means you're now able to see disk read/writes to files that were already open before the trace started; a lot of events were missed like this. You'll now see the $Mft and other special NTFS location read/write totals like ProcessHacker. Added an event toggle for this. Also fixed some bugs, added some defs, and enforced numbers only on the refresh interval textbox, which can be used to modify a running session now by pressing enter.

  16. #16
    Hyperactive Member
    Join Date
    Mar 2019
    Posts
    416

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    This API is a real mixed bag and requires some tinkering to make it reliable across Win versions.

    I was trying to use it to get quicker notification of a new process than using a WMI instant event and found the following:

    https://docs.microsoft.com/en-us/win...ess-typegroup1

    In the structure above:

    XP

    processID contains the ID of the new process

    Windows 7

    ProcessId contains the ID of then new process

    > Windows 7

    ProcessID contains -8194
    ParentId contains the real PID of the new process

    I am unable to find any documentation that explains this behaviour.

  17. #17

    Thread Starter
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,652

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    Are you using XP 32-bit or 64-bit? 64bit XP existed but was fairly uncommon. I believe your issue is related to using 32bit XP.

    When you look at the documentation, you see this for the first member,

    UniqueProcessKey

    Data type: uint32

    Access type: Read-only

    Qualifiers: WmiDataId(1), Pointer

    The address of the process object in the kernel.
    Pointer data types are 4 bytes on 32bit kernels, 8 bytes on 64bit kernels, regardless of the fact your application is only 32bits, because the data is being generated by the kernel and isn't filtered through the WOW64 layer on 64bit Windows.

    So if you're using the same data structure on both, the process id is getting shifted 4 bytes out of frame to the adjacent member. (This is why my project here is currently restricted to 64bit Windows)

    This is actually documented here, but you'd be forgiven for missing it, it's not linked from anywhere but the table of contents, so the only way you're going to find it is if you did like me and read through every single entry in the TOC that comes up when you find the page for the MOF class, which in turn doesn't come up from any of the event tracing API documentation. Unsurprisingly, the worst API ever is paired with terrible documentation.

    You need 2 different structures if you're going to support both,

    Code:
    Public Type Process_TypeGroup1_32
        UniqueProcessKey As Long
        ProcessID As Long
        ParentID As Long
        SessionID As Long
        ExitStatus As Long
        DirectoryTableBase As Long
        EndBuffer(0 To 3000) As Byte 'The last 3 members, UserSID, ImageName, and CommandLine are all variable-length and require significant processing to separate in VB. The buffer will hold them for manual processing later.
    End Type
    
    Public Type Process_TypeGroup1_64
        UniqueProcessKey As Currency
        ProcessID As Long
        ParentID As Long
        SessionID As Long
        ExitStatus As Long
        DirectoryTableBase As Currency
        EndBuffer(0 To 3000) As Byte 'The last 3 members, UserSID, ImageName, and CommandLine are all variable-length and require significant processing to separate in VB. The buffer will hold them for manual processing later.
    End Type
    On Vista+ with the EVENT_RECORD structure, it's easy to tell which one you're getting, because there's a flag in the header, but if you're using XP, presumably you're using EVENT_TRACE instead of EVENT_RECORD... in that case, you're going to have to use some other method to determine whether you're running on a 64bit system.


    And actually, you might need even more than that, depending on which version XP uses, because all those Process_V0, Process_V1 entries you see on the left of the MSDN page you linked are previous versions from earlier Windows versions, and as you can see, even the entries that exist in all of them change order. Most of them haven't changed much since 7, but XP is old enough this might be an issue (I think you're lucky here, my original structures were derived from the XP wmicore.mof file, because I hadn't found the MSDN documentation yet so turned to the XP source code, so it might be ok). The EVENT_TRACE_HEADER Version member will tell you which version of the MOF class you're getting. I haven't found any reliable documentation for which OS uses which version, but actually checking that version member is a better method anyway. I've been working on a generic processor using tdh.dll that would handle all this for you... but that's Vista+ and holy hell is it difficult to call from VB, even compared to event tracing itself.

    (A little more about the EndBuffer-- you don't neccessarily need the size; I've done some work eliminating fixed buffers in future versions of my project, but it's non-trivial for 3 different ones, so it's best to use a fixed buffer for now. The minimum size is MAX_PATH * 2 * 2 + a reasonable guess for maximum size of a sid, the nearly 1000 bytes far exceeds it.
    Last edited by fafalone; May 7th, 2022 at 07:05 AM.

  18. #18

    Thread Starter
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,652

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    If you wanted the last few members... I haven't tested this... but here's what it should look like for the 64-bit version.

    First, declare the full structure with native VB types, we're going to fill them without a fixed buffer/

    Code:
    Public Type Process_TypeGroup1_64Ex
        UniqueProcessKey As Currency
        ProcessID As Long
        ParentID As Long
        SessionID As Long
        ExitStatus As Long
        DirectoryTableBase As Currency
        UserSID As String
        ImageFileName As String
        CommandLine As String
    End Type
    Then here's how we fill the structure, given MofData and MofLength of EVENT_TRACE, or UserData/UserDataLength of EVENT_RECORD:
    Code:
    Public Sub Fill_ProcessTypeGroup1_64(ptr As Long, cb As Long, pStruct As Process_TypeGroup1_64Ex)
    Dim bBuf() As Byte
    Dim bSid() As Byte
    Dim cbSid As Long
    Dim lpSidStr As Long
    Dim cbIFN As Long
    Dim nAlignOffset As Long
    
    'Copy the fixed part
    CopyMemory pStruct, ByVal ptr, 32&
    'Copy the rest into a buffer
    ReDim bBuf(cb - 33&)
    CopyMemory bBuf(0&), ByVal ptr + 32&, cb - 32&
    
    'Validate the UserSID
    If IsValidSid(VarPtr(bBuf(0&))) Then
        cbSid = GetLengthSid(ByVal VarPtr(bBuf(0&))) 'Returns the size of UserSID in bytes
        ReDim bSid(cbSid - 1&)
        CopyMemory bSid(0&), bBuf(0&), cbSid 'Copy sid to own buffer
        ConvertSidToStringSid VarPtr(bSid(0&)), lpSidStr
        If lpSidStr Then
            pStruct.UserSID = LPWSTRtoStr(lpSidStr)
        End If
    End If
    
    If (cbSid Mod 8) Then nAlignOffset = 8 - (cbSid Mod 8) 'MSDN says aligned on 8 byte boundary...
    
    SysReAllocStringLen VarPtr(pStruct.ImageFileName), ptr + 32& + cbSid + nAlignOffset, lstrlenW(ByVal ptr + 32& + cbSid)
    
    cbIFN = LenB(pStruct.ImageFileName) + 2& '+nullchar
    
    SysReAllocStringLen VarPtr(pStruct.CommandLine), ptr + 32& + cbSid + nAlignOffset + cbIFN, lstrlenW(ByVal ptr + 32& + cbSid + cbIFN)
    
    
    End Sub
    I'm *hoping* the sid functions work as the documentation tells us to get the size manually; i.e. they look at the start of the structure, rather than count raw bytes. If not, that would have to be done manually.

    For 32bit, you'd change the 32s to 24, since the fixed part is 8 fewer bytes.

    I'm not entirely sure about any of this... this is what the documentation says...

    Sid
    The data represents a binary blob SID. The MOF data type must be object.
    The SID is of a variable length. The value contained in the first 4-bytes (ULONG) indicates whether the blob contains a SID. If the first 4-bytes (ULONG) of the blob is nonzero, the blob contains a SID. The first part of the blob contains the TOKEN_USER (the structure is aligned on an 8-byte boundary) and the second part contains the SID. To address the SID portion of the blob:
    Set a byte pointer to the beginning of the blob
    Multiply the pointer size for the event log by 2 and add the product to the byte pointer (the PointerSize member of TRACE_LOGFILE_HEADER contains the pointer size value)

    You can use the following macro to determine the length of the SID.

    #define SeLengthSid( Sid ) \
    (8 + (4 * ((SID *)Sid)->SubAuthorityCount))
    Is the blob a SID or not? Can't make any damn sense of that. It's a "binary blob SID", but no, it's really a TOKEN_USER structure, but then when you want the SID, you start at the beginning of the blob. Why does the pointer size come into play at all? Is it a SID or is it a PSID?? If it's a PSID then it's not a binary blob SID, but if it's a binary blob SID, it's not a PSID therefore not a pointer.

    And if the first part is a TOKEN_USER structure, that means there's a SID and ULONG attributes. But the ULONG is *after* the SID, and the SID is a PSID. So what, is there a PSID *and* a SID? What's the ULONG at the beginning?

    God damn MS is f'cking with us.

    It would be a miracle if my code worked. I don't know how sound my theory is that it's as the first sentence says, a binary blob sid, in which case the normal sid APIs can be used. But it's a starting point.
    Last edited by fafalone; May 7th, 2022 at 09:59 AM.

  19. #19
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,412

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    That is some pretty awful wording on MSs part. Here's how I read it:

    The data is a binary blob that may contain a variable length SID.

    If the first 4 points of the data are non-zero then the blob does contain a SID, otherwise it does not contain a SID. Since the blob begins with a TOKEN_USER structure, you must find the starting point of the SID data as follows (assuming the data is stored in a VB6 0-based ByteArray called "Data"): Data(EventLogPointerSize*2)

    I'm not sure about the length calculation, but perhaps SIDLength = 8 + (4 * Data(EventLogPointerSize*2+1))?

  20. #20
    PowerPoster
    Join Date
    Aug 2010
    Location
    Canada
    Posts
    2,412

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    Quote Originally Posted by jpbro View Post
    That is some pretty awful wording on MSs part. Here's how I read it:

    The data is a binary blob that may contain a variable length SID.

    If the first 4 points of the data are non-zero then the blob does contain a SID, otherwise it does not contain a SID. Since the blob begins with a TOKEN_USER structure, you must find the starting point of the SID data as follows (assuming the data is stored in a VB6 0-based ByteArray called "Data"): Data(EventLogPointerSize*2)

    I'm not sure about the length calculation, but perhaps SIDLength = 8 + (4 * Data(EventLogPointerSize*2+1))?
    I could be way off on the above, that's just what I would try on a first pass based on how I read the documentation you provided.

  21. #21
    Hyperactive Member
    Join Date
    Mar 2019
    Posts
    416

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    Thanks guys. Will take a look at that. Both the Win7 and XP versions are 32 bit though.

    I don't need the SID but I need to know where it ends so I can allocate space and get ImageFileName & CommandLine
    Last edited by vbwins; May 9th, 2022 at 04:09 AM.

  22. #22
    Hyperactive Member
    Join Date
    Mar 2019
    Posts
    416

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    That suggested approach does not really work. I put together something to dump the data and ran telnet. One of the crazy things is that the image name appears to not be in Unicode but the command line is.

    Dump below. The SID thing I have no idea where to go with.

    Code:
    User data length is 85
    
    Created buffer for user data. Upper bound 60
    
    Call CopyMemory bBuff(0), ByVal ptrMofData + 24, tRecord.UserDataLength - 24
    
    Call IsValidSid(VarPtr(bBuff(0)))
    
    SID is not valid
    
    Dumping buffer from 0 to 60
    
    
    Code:
    Index	Byte	Asc	Chr
    
    0	192	49	À
    1	167	49	§
    2	110	49	n
    3	173	49	*
    4	0	48	 
    5	0	48	 
    6	0	48	 
    7	0	48	 
    8	1	49	
    9	5	53	
    10	0	48	 
    11	0	48	 
    12	0	48	 
    13	0	48	 
    14	0	48	 
    15	5	53	
    16	21	50	
    17	0	48	 
    18	0	48	 
    19	0	48	 
    20	79	55	O
    21	93	57	]
    22	193	49	Á
    23	187	49	»
    24	8	56	
    25	193	49	Á
    26	110	49	n
    27	62	54	>
    28	137	49	‰
    29	215	50	×
    30	89	56	Y
    31	45	52	-
    32	232	50	è
    33	3	51	
    34	0	48	 
    35	0	48	 
    36	116	49	t
    37	101	49	e
    38	108	49	l
    39	110	49	n
    40	101	49	e
    41	116	49	t
    42	46	52	.
    43	101	49	e
    44	120	49	x
    45	101	49	e
    46	0	48	 
    47	116	49	t
    48	0	48	 
    49	101	49	e
    50	0	48	 
    51	108	49	l
    52	0	48	 
    53	110	49	n
    54	0	48	 
    55	101	49	e
    56	0	48	 
    57	116	49	t
    58	0	48	 
    59	0	48	 
    60	0	48

  23. #23

    Thread Starter
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,652

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    Were you using the 32-bit structure on Windows 7 then? With UniqueProcessKey as Long instead of Currency? That would explain why it's shifted if you weren't.

    Why it would work on XP if both are 32bit... you can check the version numbers, XP giving V0 would explain it.

    For the SID, you're just going to have to brute force it. Try every permutation of what that ridiculous, ambiguous documentation could mean until you find the right one. But solve the process id issue first, otherwise whatever solution you find will only work on one version too. In fact you're already likely at the wrong starting position for the variable length data if the process id is in the wrong place.


    (PS- The documentation does tell you one is Unicode and one is not, CommandLine has Qualifiers: WmiDataId(9), StringTermination("NullTerminated"), Format("w"), w=Unicode. Without that, it's ANSI.
    Last edited by fafalone; May 9th, 2022 at 07:37 PM.

  24. #24
    Hyperactive Member
    Join Date
    Mar 2019
    Posts
    416

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    The windows 7 version is 6.1 build 7601 32 bit. Its an old VM. My understanding is that all 32 bit versions will be using long?

  25. #25
    Hyperactive Member
    Join Date
    Mar 2019
    Posts
    416

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    After a lot of messing around I found the following. Sid is located at offset endBuffer(20) in 64 bit and endBuffer(8) in 32 bit. I then use getLengthSid API and add the offset and the length of the SID together

    So in 64 bit if length of sid is 32 you add the offset of 20 to 32 to get the starting location of imageFileName which in this case is at endBuffer(52)

    Same works with 32 bit except the offset is 8

    I have no idea why this works or if it will scale across versions.

    Code:
    Private Type Process_TypeGroup1_32
        UniqueProcessKey As Long
        ProcessID As Long
        ParentID As Long
        SessionID As Long
        ExitStatus As Long
        directoryTableBase As Long
        endBuffer(3000) As Byte
        
    End Type
    
    Private Type Process_TypeGroup1_64
        UniqueProcessKey As Currency
        ProcessID As Long
        ParentID As Long
        SessionID As Long
        ExitStatus As Long
        directoryTableBase As Currency
        endBuffer(3000) As Byte
    End Type

  26. #26

    Thread Starter
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,652

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    For both XP and 7? Have you confirmed which version you're receiving between the two? EVENT_TRACE.EVENT_TRACE_HEADER.wVersion?

    Also, when you say " Sid is located at offset endBuffer(20) in 64 bit and endBuffer(8) in 32 bit."

    Is this *after* accounting for the different structures? I.e. none of the 12 byte difference is explained the 8 bytes difference between UniqueProcessKey and DirectoryTableBase? Because it would at least make some sense if that's not true... then one of the possibilities I mentioned... there's both a TOKEN_USER with PSID, immediately followed by a SID-- that would explain it, because with the PSID, you'd have 3 pointers... UniqueProcessKey+DirectTableBase+PSID.

  27. #27

    Thread Starter
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,652

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    Project Updated to Version 2.0

    This version brings extensive new features, improvements and bug fixes. A major bug was updates were never synchronized, so only displayed if they occurred between syncs; this left most read/writes incorrect, and an update was the only way they ever got set at all, since they weren't initially added. On the new feature front, there's now an optimized mode for tracking only activity to the disk (rather than including cached activity). Running as administrator is no longer required *if* you're a member of the administrators group, the Performance Log Users group, or another group with permission to enable the SeSystemProfilePrivilege. There's now an option to merge similar activities; e.g. there's new columns for open and delete count, for programs repeatedly opening the same file. This will merge IO as well. After seeing a research paper suggest it, I implemented thread context switch tracking for process attribution. It's completely useless. Only accurate when the events are already attributed. Nonetheless, I left the option in for those who want to play around with it (it's in the new additional options popup when you click 'More'). Options to control when rundowns are done, and control process caching. See the full changelog for a few more.

    Code:
    'Version 2.0: -Running as administrator is no longer strictly required. However,
    '              you must still be a member of the administators group, a member
    '              of the Performance Log Users group, or another user/group that has
    '              access to the SeSystemProfilePrivilege, which the code now calls
    '              AdjustTokenPrivileges to set.
    '
    '             -Project has been optimized to track disk read/write alone. Because
    '              of caching, FileIo events don't neccessarily trigger disk activity,
    '              so if you only wanted to watch disk activity, disabling everything
    '              except DiskIO will enable DiskIO Exclusive Mode; FileIo events not
    '              activated by the DiskIO flags will be disabled (and can't be enabled
    '              while the trace is in progress, since this is done in the inital
    '              flags), and DiskIO will properly attribute IO to Create events,
    '              where when disabled, create events use FileIO_ReadWrite, which
    '              doesn't reflect disk io because of caching.
    '              DiskIO includes open/delete events that trigger disk activity.
    '
    '             -Added option to merge certain activities by the same process on the
    '              same file. Create/open/read/write/delete can be merged. Multiple
    '              events may still exist where an initial pid was -1 and then updated.
    '              An additional option restricts this to same opcode only (combining
    '              only read/write opcodes).
    '
    '             -The default configuration is now DiskIO Exclusive Mode with merging
    '              enabled. Remember that enabling any FileIO operation takes it out
    '              of that mode, and that FileIO operations do not represent disk read
    '              and write activity due to caching.
    '
    '             -There's now an option to disable the initial rundown; you'll miss
    '              activity on open files during the trace, but idle disks won't spin
    '              up. If this option is used, the rundown may be performed at the end
    '              of the trace (always, if old logger mode, optionally with new), and
    '              the disk io logs will be scanned for missed events, and added then.
    '
    '             -Process caching now has an option to be disabled (never, always, or
    '              only when running in DiskIO Exclusive Mode), for situations where
    '              process id use may be a problem (processes rapidly being created
    '              and exiting).
    '
    '             -Context switch tracking for process attribution is now implemented,
    '              however I've found it entirely useless. It only works in cases where
    '              the pid was already returned.
    '
    '             -Log sync interval can now be adjusted.
    '
    '             -Changed how filenames are read to no longer use a fixed buffer;
    '              the MOF structure now uses String (variable length) and is set
    '              by a Fill_<type> routine in modEventTrace that first copies the
    '              data into a variable length byte array, then sets the MOF type
    '              String. This allows long file name support without using a
    '              massive fixed buffer.
    '
    '             -There's now an event for unattributed FileIo_ReadWrite events like
    '              there is for DiskIo_ReadWrite, to catch read/writes on files already
    '              open before the trace started.
    '
    '             -So few events were in the type enum I got rid of the enum and made
    '              them byte constants to improve performance. Also added constants
    '              for the FileIo opcodes.
    '
    '             -Added some more flags in the Misc column.
    '
    '             -Cleaned up some lingering potentially unsafe thread data accesses.
    '
    '             -There's reports of poor critical section performance on Win8+ with
    '              the default dynamic spin count adjustment; trying 4000 instead per
    '              MSDN recommendation as optimal; will see how it goes.
    '
    '             -Bug fix: AddActivity didn't add read/write size, so it was only ever
    '                       non-zero if an update came in, which was generally never for
    '                       FileIo since MSDN lied about FileKey being what you use to
    '                       correlate FileIo_ReadWrite events.
    '
    '             -Bug fix: While read/write totals were updated, sync between the
    '                       ActivityLog struct and threadmain copy consisted only of
    '                       adding new items, so only updates that occured between sync
    '                       calls were ever updated on the ListView.
    '                       This also impacted pid updates.
    PS- Work continues on implementing a VB6 version of a generic item data reader using tdh.dll, but calling it from VB6 makes everything in this project so far seem simple, so progress is very slow. It wasn't anywhere complete enough to justify holding off on updating. Might be another month or two given the complexity and how many other projects I'm working on.

  28. #28
    Hyperactive Member
    Join Date
    Mar 2019
    Posts
    416

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    Quote Originally Posted by fafalone View Post
    For both XP and 7? Have you confirmed which version you're receiving between the two? EVENT_TRACE.EVENT_TRACE_HEADER.wVersion?

    Also, when you say " Sid is located at offset endBuffer(20) in 64 bit and endBuffer(8) in 32 bit."

    Is this *after* accounting for the different structures? I.e. none of the 12 byte difference is explained the 8 bytes difference between UniqueProcessKey and DirectoryTableBase? Because it would at least make some sense if that's not true... then one of the possibilities I mentioned... there's both a TOKEN_USER with PSID, immediately followed by a SID-- that would explain it, because with the PSID, you'd have 3 pointers... UniqueProcessKey+DirectTableBase+PSID.
    If you look at the structure definitions I included endBuffer byte array is located after the directoryTable base in both the 32 bit and 64 bit structures. I am not checking the header version. This is on Windows 7 (32 bit) and Windows 10 (64 bit)

    So the side returns valid using isValidSid passing the 20th element in endBuffer on 64 bit and the 8th element on 32. I then call getLengthSid using that same offset which returns the length. Adding the length to the offset then points the the place in endBuffer where imageName starts. I have no idea why there is a 12 byte difference. In fact I dont know why the sid does not start at endBuffer(0) but it does not.

    I don't understand why because you know the documentation is less than stellar but it does work,

  29. #29

    Thread Starter
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,652

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    Quote Originally Posted by vbwins View Post
    If you look at the structure definitions I included endBuffer byte array is located after the directoryTable base in both the 32 bit and 64 bit structures. I am not checking the header version. This is on Windows 7 (32 bit) and Windows 10 (64 bit)

    So the side returns valid using isValidSid passing the 20th element in endBuffer on 64 bit and the 8th element on 32. I then call getLengthSid using that same offset which returns the length. Adding the length to the offset then points the the place in endBuffer where imageName starts. I have no idea why there is a 12 byte difference. In fact I dont know why the sid does not start at endBuffer(0) but it does not.

    I don't understand why because you know the documentation is less than stellar but it does work,
    Right I was just asking if you were indeed using the 32 bit structure on 32bit Windows. If you weren't that would have offered an explanation. But if you are....

    I found some undocumented versions. Apparently even XP/2003 had V4.

    Code:
    [Dynamic,
     Description("Process Create/Exit Event") : amended,
     EventType{1, 2, 3, 4, 39},
     EventTypeName{"Start", "End", "DCStart", "DCEnd", "Defunct"} : amended
    ]
    class Process_V4_TypeGroup1:Process_V4
    {
        [WmiDataId(1),
         Description("UniqueProcessKey") : amended,
         pointer,
         read]
         uint32  UniqueProcessKey;
        [WmiDataId(2),
         Description("ProcessId") : amended,
         format("x"),
         read]
         uint32  ProcessId;
        [WmiDataId(3),
         Description("ParentId") : amended,
         format("x"),
         read]
         uint32  ParentId;
        [WmiDataId(4),
         Description("SessionId") : amended,
         read]
         uint32  SessionId;
        [WmiDataId(5),
         Description("ExitStatus") : amended,
         read]
         sint32  ExitStatus;
        [WmiDataId(6),
         Description("DirectoryTableBase") : amended,
         pointer,
         read]
         uint32  DirectoryTableBase;
        [WmiDataId(7),
         Description("Flags") : amended,
         read]
         uint32  Flags;
        [WmiDataId(8),
         Description("UserSID") : amended,
         extension("Sid"),
         read]
         object  UserSID;
        [WmiDataId(9),
         Description("ImageFileName") : amended,
         StringTermination("NullTerminated"),
         read]
         string  ImageFileName;
        [WmiDataId(10),
         Description("CommandLine") : amended,
         StringTermination("NullTerminated"),
         format("w"),
         read]
         string  CommandLine;
        [WmiDataId(11),
         Description("PackageFullName") : amended,
         StringTermination("NullTerminated"),
         format("w"),
         read]
         string  PackageFullName;
        [WmiDataId(12),
         Description("ApplicationId") : amended,
         StringTermination("NullTerminated"),
         format("w"),
         read]
         string  ApplicationId;
    };
    To get to the bottom of it, need to compare versions and/or same OS (e.g. Win10 32bit, Win10 64bit).

  30. #30

    Thread Starter
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,652

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    Project Updated to Version 2.1

    Quick bug fix to address FileIo_DirEnum; the pattern and InfoClass were not being reported correctly. Also implemented smarter scrolling behavior: If you scroll up away from the bottom, it will stay there instead of send you right back to the bottom next time items are added. Once you scroll back to the bottom, it will resume staying there as new items are added. FileIo_ReadWrite appears to actually have an 8-byte TTID; the class definitions for the others are incorrect. The IoSize info is now correct here. There's also some (disabled) debug routines that will help analyze the data structures, if you're on a different OS and having trouble.

  31. #31
    Hyperactive Member
    Join Date
    Mar 2019
    Posts
    416

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    I am getting v2 back for connections and v4 for new process

  32. #32

    Thread Starter
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,652

    Re: [VB6] Event Tracing for Windows - Monitoring File Activity with ETW

    There's now an x64 version of this project, thanks to twinBASIC, which is impressively far along.

    [twinBASIC] x64-compatible port of Event Tracing for Windows File Activity Monitor

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width