Thumbnail images for video files are accessible via Shell32's Automation (i.e. ActiveX) interface.
These can be obtained using a ShellFolderItem object's ExtendedProperty() method. The tough part is that these are returned as PROPVARIANT values and not regular Variants as we know them in VB6. For many ExtendedProperty() return values VB6 can handily coerce the PROPVARIANT to a Variant, even though you can still end up with unsupported subtypes (like unsigned integers of varying lengths).
Here we need to request the property by passing SCID_THUMBNAILSTREAM to ExtendedProperty. In this case it returns a PROPVARIANT with a VT_STREAM content subtype, making it a bit troublesome in that VB6 cannot convert it to a standard Variant.
Luckily that can be done by calling the PropVariantToVariant function, though it handles the situation by converting the VT_STREAM to a Variant's VT_UNKNOWN. Still, the IStream object reference is there and usable.
This particular stream is designed for consumption by GDI+/WindowsCodecs, and isn't a simple PNG or JPEG in a stream. But GdipCreateBitmapFromStream() can read it just fine, and from there you are on your way.
The Code
Most of the code in the sample program relates to a version of my ThumbnailsView UserControl. Without this the program would be a lot smaller, but we don't have many good controls for multi-image display. You almost always end up working with things like nested PictureBox controls and a scrollbar or two and some logic to glue it all together. There is also a funky SimpleProgressBar UserControl for visual feedback.
You can pretty much ignore both of those along with helper modules Picture2BMP.bas, BmpGen.cls, and ScaledLoader.cls that are used by ThumbnailsView.ctl.
Even Form1's code is mostly gingerbread, dealing with picking a folder of video files to fetch thumbnails for. Its ListAttributes() subroutine is the part that enumerates the folder items and retrieves the thumbnails, with the help of GdipLoader.cls to convert the stream into a StdPicture object.
Code:
Private Sub ListAttributes(ByVal FolderPath As String)
Dim Folder As Shell32.Folder
Dim ShellFolderItem As Shell32.ShellFolderItem
Dim GdipLoader As GdipLoader
Dim ExtProp As Variant
Set Folder = ShellObject.NameSpace(FolderPath & "\")
If Folder Is Nothing Then
Unload Me
Else
Set GdipLoader = New GdipLoader
With SimpleProgressBar1
.Caption = ""
.Max = Folder.Items.Count
End With
For Each ShellFolderItem In Folder.Items
If Not (ShellFolderItem.IsFolder Or ShellFolderItem.IsLink) Then
PropVariantToVariant ShellFolderItem.ExtendedProperty(SCID_THUMBNAILSTREAM), ExtProp
'Note: vbDataObject value is really VT_UNKNOWN (i.e. 13):
If VarType(ExtProp) = vbDataObject Then
ThumbnailsView1.Add GdipLoader.LoadPictureStream(ExtProp), ShellFolderItem.Name
End If
End If
SimpleProgressBar1.Value = SimpleProgressBar1.Value + 1
Next
End If
End Sub
Issues
The first issue is that this isn't speedy. I'm not sure what Shell32 is going through to get thethumbnails even though I had assumed they were retrieved using typical OLE Storages & Streams mechanisms. It almost seems too slow for that though and since it uses video codecs there must be more involved.
The second issue is that depending on the video encoding format and codecs you have installed you might get nice results or a junk image back. I was almost ready to give it up until I realized I had VB6.exe in ffdshow's blacklist on my development PC.
But testing the compiled EXE (and allowing ffdshow use via the Compatibility Manager popup) returned good images from everything I tested but some DIVX formats.
By getting rid of ThumbnailsView and using some other multi-image control and associated code you can eliminate WIA 2.0 and Edanmo's olelib.tlb.
The Demo
Running the program you'll see the single Form user interface. It has a status/progress bar, a ThumbnailsView taking up the rest of the Form, and a single menu item "Choose Folder" you use to start the process.
The running progress shows you the current and total folder items being processed. Subfolders and links are skipped, as are files that do not have a ThumbnailStream return value.
Some videos using standard Windows codecs
Some videos that require additional codecs (MP4 mostly).
Work fine when ffdshow is used.
Remaining Issues
In theory all kinds of files ought to return thumbnails this way. However I get nothing useful for music folders, Word document folders, etc. I can only assume that the returned results are in another format this demo rejects (GDI+ Image object?). I'll have to investigate further.
Last edited by dilettante; Apr 21st, 2014 at 11:43 AM.
Hmm, it also seems to be able to retrieve thumbnail (album art) from many WMA and MP3 audio files as well. Not all, since many times you don't have this embedded.
I'm not getting any JPEG thumbnails though.
So far tested on Windows Vista SP2 and Windows 7 SP1.
True, though in this case the machine I was programming on happens to have ffdshow installed. It isn't used for video transcoding, which I do on a Windows Home Server machine.
The point was that you need to have the necessary codecs for your video files no matter what filters and codecs package you use.
The point was that you need to have the necessary codecs for your video files no matter what filters and codecs package you use.
Of course i was just pointing out the LAV thing in case you didn't follow such things and found it helpful to know. I was originally going to say 'fyaori' (for yours and other readers info) rather than 'fyi' but as it was an acronym I'd just made up I went with the more traditional form...
If you don't know where you're going, any road will take you there...
I'm not sure what Shell32 is going through to get thethumbnails even though I had assumed they were retrieved using typical OLE Storages & Streams mechanisms. It almost seems too slow for that though and since it uses video codecs there must be more involved.
Thumbnails are cached in Windows. That's what the hidden file Thumbs.db is all about. You can read on it at Wikipedia here.
C++ programmers will dismiss you as a cretinous simpleton for your inability to keep track of pointers chained 6 levels deep and Java programmers will pillory you for buying into the evils of Microsoft. Meanwhile C# programmers will get paid just a little bit more than you for writing exactly the same code and VB6 programmers will continue to whitter on about "footprints". - FunkyDexter
There's just no reason to use garbage like InputBox. - jmcilhinney
The threads I start are Niya and Olaf free zones. No arguing about the benefits of VB6 over .NET here please. Happiness must reign. - yereverluvinuncleber
Those are just the "dead" thumbnails that don't get updated, and are smaller 96x96 pixel versions used for Explorer views. They'd already have to have been created or else you can't get anything programmatically.
These thumbnails are extracted from the files on request, as evidenced by the time involved and the need for video codecs. They tend to be 200x133 pixels, 200x112, etc. based on the aspect ratio of the video.
Those are just the "dead" thumbnails that don't get updated, and are smaller 96x96 pixel versions used for Explorer views. They'd already have to have been created or else you can't get anything programmatically.
These thumbnails are extracted from the files on request, as evidenced by the time involved and the need for video codecs. They tend to be 200x133 pixels, 200x112, etc. based on the aspect ratio of the video.
Explorer automatically shows video thumbnails after CODEC installed. Not sure the mechanism. Need to study further.
Well don't miss the forest for the trees here, in other words a thumbnail viewer wasn't the point of this thread. The idea was to retrieve video thumbnails for other purposes.
After all, a user can just use Explorer to look at them and if they don't already exist it will create them on first visit to the folder.
But it would be interesting to see a working VB6 example of the use of IExtractImage to obtain Shell thumbnails from the cache. The more options we have, the better.
Here the idea was to show a fairly easy way to get video thumbnails without actually playing each video partway and grabbing a frame a second or so in yourself. I'm pretty sure now that this is what Explorer/Shell is doing for these SCID_ThumbnailStream requests.
I had looked pretty hard and found no working VB6 code for either the ExtendedProperties or the IExtractImage approaches.
Here is a second sample. It is far more minimal, a batch thumbnail "grabber" that writes the thumbnail images to an output folder instead of displaying any user interface at all.
The idea is you'd run it from the command line passing the video folder as the single argument it expects. Or basically the very same thing by dragging the video folder from an explorer window and dropping it on the program's icon in another Explorer window.
One problem I've found is that on my system video files with the MP4 extension are not seen as video by the Shell. I suspect the extension was never registered properly on this machine even though they play fine if I open them in a player.
Just had a quick play with the second demo before going out; folder contained a mix of formats (AVI, MP4, WMV). The vector param in VectorContains was always empty when called and, hence, errored at that point. Don't have time right now to provide more info so if that in itself is not helpful I'll report back later...
btw; couldn't run the first as I don't have wiaaut.dll and don't know how to get hold of a legit copy for my XP SP3?
If you don't know where you're going, any road will take you there...
And so we begin the sad tale of a dead operating system. Sorry to say it, but XP is done, stick a fork in it!
Issue #1: WIA 2.0
While Microsoft once provided a redist version for XP SP1 and later they pulled it quite a while back as part of XP's end of support. I'm not sure where you can obtain it now, though I'm sure there are sites risking the wrath of illegally hosting the installer package for the WIA 2.0 SDK & Redist.
Issue #2: Windows Search tags (SCID_Kind vector)
From some testing here I can confirm that this information is not available from XP's Shell32. Not even with Search 4.0 installed! This brings us to...
Issue #3: Propsys.dll
This does not ship in any version of Windows XP. The only way to get it there is to install Windows Desktop Search 3.0 or later, but you still can't get the Search "Kind" tags from Shell32 as far as I can tell.
Issue #4: Shell32's Automation Interface
For whatever bizarre reason Microsoft broke binary compatibility between XP and Vista. This means that unless you rewrite everything to use late binding a program compiled on XP will not run on Vista or later, and vice versa.
So if you're targeting Windows Vista (which I hope you are :-), why should you care?
Well, the answer is we care less and less because we can't care. XP is simply too old and too incomplete to deal with many things that have come along in 8 years.
In any case this might only be useful to somebody writing some special purpose video management application, such as one creating an index database for referencing offline archived videos. I would think location and timestamp more useful but people do seem to love their thumbnails.
Here's another version of the batch grabber with a minor change to scale down the images, which seems to be needed for Windows 7.
It doesn't address any XP issues and probably not the Win7/PNG issues but I need to retest on Win7.
Edit:
Found the bug. A silly error in the GDI+ code, though oddly one that Vista is immune to. PNG and GIF both work even on Win7 now.
Edit:
Added a test to confirm the Variant subtype of SCID_Kind values before trying to examine them. Should avoid some types of errors when run on XP though the program still won't work on XP (will never find any "video" files). But checking SCID_Kind could be removed entirely if you wanted to.
Last edited by dilettante; Apr 21st, 2014 at 07:53 AM.
Reason: replaced attachment
I use Geraint Davie's dll's from here: http://www.gdcl.co.uk/vb.htm (capstill.dll) and extract a thumbnail at the time a file is added to the Library (though user can later swap that for any frame of their choosing). I believe we once discussed it in another of your threads...
If you don't know where you're going, any road will take you there...
I use Geraint Davie's dll's from here: http://www.gdcl.co.uk/vb.htm (capstill.dll) and extract a thumbnail at the time a file is added to the Library (though user can later swap that for any frame of their choosing). I believe we once discussed it in another of your threads...
Which it accomplishes by playing the movie to locate a frame, just as Shell32 does for you here. You have to because lots of video compression formats are made up of occasional complete frames and "delta" information between them rather than being a giant flipbook of complete frames.
Yes, I'm aware of that - it's called temporal compression. The earliest codecs used only spatial compression.
I'm very interested in how you do this via shell32 as I'd rather not have that dependency on capstill.dll. Currently I am, of course, building a filter graph which that dll references. Were I to use shell32 instead it would not be possible to grab the frame that the user is currently viewing via that graph (unless, I assume, another graph is built behind the scenes via shell32 and seeks to the same frame).
If you don't know where you're going, any road will take you there...
I'm not really doing it at all, that's sort of the point of this entire thread.
I just ask Shell32 to give me the extended property "thumbnail stream" and it seems to do everything itself. All I have to do is take the GDI+/WindowsCodecs stream and create a usable image from that.
Code:
// Name: System.ThumbnailStream -- PKEY_ThumbnailStream
// Type: Stream -- VT_STREAM
// FormatID: (FMTID_SummaryInformation) F29F85E0-4FF9-1068-AB91-08002B27B3D9, 27
//
// Data that represents the thumbnail in VT_STREAM format that GDI+/WindowsCodecs supports (jpg, png, etc).
DEFINE_PROPERTYKEY(PKEY_ThumbnailStream, 0xF29F85E0, 0x4FF9, 0x1068, 0xAB, 0x91, 0x08, 0x00, 0x2B, 0x27, 0xB3, 0xD9, 27);
Yes, i could see from looking at the code that it was fairly 'black box-ish' (even though I cannot run it as explained above). Shame you can't pick a particular frame using this method. Incidentally, where is it getting its frame from time-wise? The exact middle or a few seconds in? I'm guessing the latter to avoid slow grabbing when files have distant key-frames...
Oh, and are you able to spy on the graph that shell32 is building (I'm assuming that's what it's doing on the basis that it needs DS filters to be installed)? If so, is it adding the SampleGrabber filter?
If you don't know where you're going, any road will take you there...
System.ThumbnailStream contains 100% of the documentation I can find on this. I assume it uses SampleGrabber but I have no way of knowing.
You ought to be able to run the first example on XP SP2 or later as long as you have Desktop Search installed... once you replace the WIA 2.0 operations with lower level GDI+ Flat API calls or use some other GDI+ wrapper library.
The second example is done with only Flat API calls so WIA isn't an issue. So if you just remove the check for an SCID_Kind value of "video" it should run on XP SP2 or later too as long as you have Desktop Search.
Thanks for the additional info. I'll probably have a play with with this at some point though my interest is somewhat diminished by the fact that grabbing a thumbnail for a particular point-in-time doesn't look possible.
If you don't know where you're going, any road will take you there...
Well I'm not sure how any automated "thumbnail grabber" is supposed to know what frame you might want. As you can see you often get all-black, all-blue, etc. frames when you browse a folder in Explorer too. I assume it just goes into the video some set duration and then takes what it has there.
You're expecting a lot.
The only alternative I can think of is either to sit through every video and pick a frame by eyeball or else grab every frame and do some analysis to see whether it deviates a lot from all one color.
Well, as already said, Vee-Hive grabs a default thumbnail (from the middle) whenever a video is added to a Library. However, my users often prefer to pick their own frames to represent that file (and that, I believe, is as it should be in a media management program). In addition, the program allows users to define segments from within a single file so it makes sense for each of those segments to differ, visually, from one another. And even beyond that, a user can grab a frame whilst watching a file (from their armchair via remote control if they wish) and have it represent the file; in other words, it's done unobtrusively and at their convenience.
Believe me, if I were to down-grade this functionality (to the extent that a potentially all-black image represented a file's content) I don't think they'd be very pleased!
EDIT: Just remembered, Vee-Hive allows story-board creation as well, with frames grabbed at intervals, so yet another reason why this functionality is important. It's not that I'm expecting a lot; it's that I can do already what I expect!
Last edited by ColinE66; Apr 21st, 2014 at 03:35 PM.
If you don't know where you're going, any road will take you there...
I guess it comes down to a tradeoff between requirements and development cost. If you want premium functionality you put out premium effort! I never claimed this did anything but get video thumbnails via the Shell. And it does so without any 3rd party libraries.
I think you must have looked at the first post screenshots and thought I was doing something more. Sorry.
Here's another version that strips out more stuff. It bypasses calling PropVariantToVariant, uses late binding to Shell objects, and uses SCID_PerceivedType instead of SCID_Kind. Sadly XP doesn't seem to support SCID_PerceivedType either.
As a workaround you could probably use your own list of video file extensions and compare against that file by file to eliminate non-video files.
No need to apologise - it's a very interesting project and I learned something new.
To be honest, at first glance, I didn't think you were dong anything more than using Windows own thumbnail cache so it was interesting to read on and find that this was not the case. But you most certainly did not over-state what you were doing and as a result I did not mis-interpret it; I was merely exploring whether the potential existed to remove the dependency on capstill.dll. Alas, it would appear not, which is a shame
If you don't know where you're going, any road will take you there...
I vaguely recall investigating thumbs.db ages (years) ago and that (for reasons i don't remember) they were hard to access. Don't quote me, but it may have been something to do with the proprietary, mystical format of that particular file...
If you don't know where you're going, any road will take you there...
C++ programmers will dismiss you as a cretinous simpleton for your inability to keep track of pointers chained 6 levels deep and Java programmers will pillory you for buying into the evils of Microsoft. Meanwhile C# programmers will get paid just a little bit more than you for writing exactly the same code and VB6 programmers will continue to whitter on about "footprints". - FunkyDexter
There's just no reason to use garbage like InputBox. - jmcilhinney
The threads I start are Niya and Olaf free zones. No arguing about the benefits of VB6 over .NET here please. Happiness must reign. - yereverluvinuncleber
As far as I can remember it's an OLE Structured Storage, a sort of filesystem in a file. But those thumb.db files are only useful when indexed and beginning with Vista the organization of them changed anyway so you'd want to use some API to get those images.
So after a comment I made to dilettante in another thread concerning my opinion of shell automation, I realized I was being a bit of a hyprocite in that I was using this method in my code. So for those who love dilettante's project, but wish they didn't have to use shell automation, fear not! Grab v1.71 or higher of oleexp (see sig), and then it goes like this:
Code:
Public Function AddThumbviewVideoISI(pidl As Long) As Long
Dim psi As IShellItem2
Dim vProp As Variant, vrProp As Variant
Dim pKey_ThumbStream As PROPERTYKEY
Dim gidTS As UUID
Dim sguid As String
sguid = "{F29F85E0-4FF9-1068-AB91-08002B27B3D9}"
Call CLSIDFromString(StrPtr(sguid), gidTS)
pKey_ThumbStream.pid = 27
pKey_ThumbStream.fmtid = gidTS
Call SHCreateItemFromIDList(pidl, IID_IShellItem2, psi)
If (psi Is Nothing) = False Then
psi.GetProperty pKey_ThumbStream, vProp
PropVariantToVariant vProp, vrProp
If VarType(vrProp) = vbDataObject Then
'[do whatever you want with vrProp
ThumbnailsView1.Add GdipLoader.LoadPictureStream(vrProp), [some other way of getting name]
End If
End If
End Function
'support
Public Declare Function SHCreateItemFromIDList Lib "shell32" (ByVal pidl As Long, riid As UUID, ppv As Any) As Long
Public Declare Function CLSIDFromString Lib "ole32" (ByVal lpszGuid As Long, pGuid As Any) As Long
Public Function IID_IShellItem2() As UUID
'7e9fb0d3-919f-4307-ab2e-9b1860310c93
Static IID As UUID
If (IID.Data1 = 0) Then Call DEFINE_UUID(IID, &H7E9FB0D3, CInt(&H919F), CInt(&H4307), &HAB, &H2E, &H9B, &H18, &H60, &H31, &HC, &H93)
IID_IShellItem2 = IID
End Function
Public Sub DEFINE_UUID(Name As UUID, L As Long, w1 As Integer, w2 As Integer, B0 As Byte, b1 As Byte, b2 As Byte, B3 As Byte, b4 As Byte, b5 As Byte, b6 As Byte, b7 As Byte)
With Name
.Data1 = L
.Data2 = w1
.Data3 = w2
.Data4(0) = B0
.Data4(1) = b1
.Data4(2) = b2
.Data4(3) = B3
.Data4(4) = b4
.Data4(5) = b5
.Data4(6) = b6
.Data4(7) = b7
End With
End Sub
If you don't already have the pidl, it == ILCreateFromPathW(StrPtr(fullpath))
Also, if you wanted a standard hBitmap to work with, vrProp can be passed to the function below (slightly modified version of code from dilettante's project):
Code:
Public Function hBitmapFromStream(ByVal ImageSteam As olelib.IUnknown) As Long
Dim GdipBitmap As Long
Dim hBitmap As Long 'GDI Bitmap handle.
If gdipInitToken Then
If GdipCreateBitmapFromStream(ImageSteam, GdipBitmap) = 0 Then 'GDIP_OK Then
If GdipCreateHBITMAPFromBitmap(GdipBitmap, hBitmap, 0) = 0 Then 'GDIP_OK Then
hBitmapFromStream = hBitmap
Else
Debug.Print "hBitmapFromStream failed at hbmpfrombmp|"
End If
Else
Debug.Print "hBitmapFromStream failed at bmpFromStream"
End If
Else
Debug.Print "GDIP not initialized|"
End If
End Function
If you're handling thumbnails for more than just video, and want to know which handler to direct things to, the best way to do it is the same way Explorer handles things, Perceived Type:
Code:
Public Declare Function AssocGetPerceivedType Lib "shlwapi.dll" (ByVal pszExt As Long, ptype As PERCEIVED, pflag As PERCEIVEDFLAG, ppszType As Long) As Long
Public Enum PERCEIVED
PERCEIVED_TYPE_CUSTOM = -3
PERCEIVED_TYPE_UNSPECIFIED = -2
PERCEIVED_TYPE_FOLDER = -1
PERCEIVED_TYPE_UNKNOWN = 0
PERCEIVED_TYPE_TEXT = 1
PERCEIVED_TYPE_IMAGE = 2
PERCEIVED_TYPE_AUDIO = 3
PERCEIVED_TYPE_VIDEO = 4
PERCEIVED_TYPE_COMPRESSED = 5
PERCEIVED_TYPE_DOCUMENT = 6
PERCEIVED_TYPE_SYSTEM = 7
PERCEIVED_TYPE_APPLICATION = 8
PERCEIVED_TYPE_GAMEMEDIA = 9
PERCEIVED_TYPE_CONTACTS = 10
End Enum
Public Enum PERCEIVEDFLAG
PERCEIVEDFLAG_UNDEFINED = &H0 'No perceived type was found (PERCEIVED_TYPE_UNSPECIFIED.
PERCEIVEDFLAG_SOFTCODED = &H1 'The perceived type was determined through an association in the registry.
PERCEIVEDFLAG_HARDCODED = &H2 'The perceived type is inherently known to Windows.
PERCEIVEDFLAG_NATIVESUPPORT = &H4 'The perceived type was determined through a codec provided with Windows.
PERCEIVEDFLAG_GDIPLUS = &H10 'The perceived type is supported by the GDI+ library.
PERCEIVEDFLAG_WMSDK = &H20 'The perceived type is supported by the Windows Media SDK.
PERCEIVEDFLAG_ZIPFOLDER = &H40 'The perceived type is supported by Windows compressed folders.
End Enum
You then call it with a pointer to the extension; you can't pass a full file name.. the last 3 parameters are all outputs that you must have variables for. Call AssocGetPerceivedType(StrPtr(sEx), lType, lFlag, lpsz)
lpsz can be recovered with SysReAllocString for a string like "text" "video" etc, but even if ignored should be freed with CoTaskMemFree(lpsz).
Last edited by fafalone; May 31st, 2015 at 09:09 PM.
Reason: Typo
"Shell Automation" is implemented within exactly the same DLL: Shell32.DLL, so I'm not sure why people seem to fear it so much. Its classes wrap the interfaces fairly thinly and the only real penalty is some loss of flexibility, mostly because Microsoft has stopped extending it.
Thus it works "out of the box" for everyone since it is part of Windwos Explorer. There is nothing to deploy and not even a need for a 3rd party typelib at compile time... and better yet no finicky code required to avoid memory leaks and crashes.
-Still have to add a reference to it, not much difference between that and a typelib
-It wraps things in its own custom types. The minute you want to do something that it doesn't include, you have to completely start over with the references. FolderItem is useful exclusively for that single object- whereas doing things with pidls and/or IShellFolder/IShellItem are what lots of other system apis and interfaces expect. SCID_x? Useful only here. PROPERTYKEY? Used in dozens of interfaces and APIs. It's simplification at the cost of being in a walled garden... much better to do things in a way that are more consistent with other things.
-The typelib doesn't use anything that's not native. There's no incompatibility beyond newer features requiring a newer OS.
-It's got its own finicky code; why learn the unique way of using this one particular object instead of extending standard components like system shell interfaces?
-The pain is good for you! I guarantee you'll be a better programmer after using the slightly more complex, but system standard and applicable to all language method.
Last edited by fafalone; May 31st, 2015 at 09:08 PM.
The SCIDs seem to be for backward compatibility. On Vista and later this works just fine:
Code:
Dim ShellFolderItem As Shell32.ShellFolderItem
With New Shell32.Shell
Set ShellFolderItem = .NameSpace(ssfDESKTOP).ParseName(DllFile)
With ShellFolderItem
FileVersion = .ExtendedProperty("System.FileVersion")
Company = .ExtendedProperty("System.Company")
End With
End With
Set ShellFolderItem = Nothing
Those strings are the canonical names for the SCIDs. See: my latest codebank post that dives into the pain of the extended property system, and propkey.h.