Delete DLL file at runtime

I'm working on an application that upon startup is scanning a directorty for dll files. I have a global function, RegisterComponent, that can register/unregister the dll file from the registry. For each dll I do a createobject to an object array called MyDLL(). I also add the filename to a listview. It looks something like this:

If Right$(FileName, 3) = "dll" Then
'Register the dll in the registry
RegisterComponent ProductDir(i) & FileName, DllRegisterServer
buffer() = Split(FileName, ".")
Set MyDLL(MyDLLCnt) = CreateObject(buffer(0) & ".Class1")
'Add the Script name without extension to the Scripts list
frmMain.lstScripts.ListItems.Add , , FileName
'Add DLL index
frmMain.lstScripts.ListItems.item(frmMain.lstScripts.ListItems.Count).SubItems(6) = MyDLLCnt
'Send information about the Log Listview to the DLL
MyDLL(MyDLLCnt).SetLogWindow frmMain.txtLog
MyDLL(MyDLLCnt).SetReadyParam frmMain.chkScriptReady
MyDLLCnt = MyDLLCnt + 1

End If

SetLogWindow and SetReadyParam are two functions that exist in all dll's. The DLL index is also stored in the listview so I can know how to access each dll in runtime.

Ok, this works just fine but what I want to do is to be able to click on one of the items in the listview and click delete. Then the dll file shall be deleted from the hd and I call the scan function again to refresh the list.

First I made a small function that just delete the file but then I got a file/path error stating that the file is in use and can't be deleted. So I figured that the dll first must be unregistered from the registry. So I added a line for this before I try to delete the file. Well, I still got the file/path error even if I checked the registry and the DLL was really unregistrered. Then I thought that it is because the object is in use by my program so I added

Set MyDLL(frmMain.lstScripts.ListItems.item(frmMain.lstScripts.ListItems.Count).SubItems(6)) = Nothing

I guess this is the way to terminate an object class? I still got the same error. Then I looked at the dll code and found that I had no Terminate event. So I added a Terminate event and added:

Set txtLog = Nothing
Set chkFinished = Nothing

txtLog and chkFinished are the to objects that were passed from the main program to the dll in the above code. The result now is that I get a Runtime Error 28, Out of Stack space on the Set MyDLL(frmMain.lstScripts.ListItems.item(frmMain.lstScripts.ListItems.Count).SubItems(6)) = Nothing row.

And here is where I'm stuck!

So, what I want to know is how to delete a dll file that has been registered with CreateObject?

After some searching I finally find a way to unload a DLL at runtime using FreeLibrabry. However, this will cause a crash upon exit of the program because the VB runtimes think that the DLL is still loaded. Is this avoidable?