Non-Reg DLL failing under Vista / Win 7 called in-proc from Outlook AddIn

[rookpsu]

I'm stumped.

I have a old VB6 Oulook Addin that I was recently contracted to "skin" in order to bring the UI up to speed for a client.

To do so I utilized a third party component that handled the requisite Win32 API calls.

Anyhow.. I built in an XP environment. Worked fine. Now, as I test deployment on Vista/Win7 machines I'm greeted by the infamous "Run Time Error 48, File Not Found" the first time I make a call to it.

As noted, it's a non-registering DLL, so the calls are made through late binded function calls:

(i.e) Public Declare Function "File" Lib "File.DLL" - in a VB6 module.

Anyhow.. it IS finding the file. I pulled up ProcMon and it's showing NO issues locating and interacting with the DLL so it seems to be failing internally for some reason.

I set up a dev environment on Win7/Vista and I CAN run the application if operating out of the VB6 process. When compiled however and running as an addin within the Outlook.EXE process, it fails.

I also set up a small hello world type app that utilizes the DLL and that works as well, compiled or in debug (although in its own process, not Outlook.EXE).

I'm REALLY stumped now. Any insight would be greatly appreciated as I'm approaching a deadline.

I also loaded up Dependency Walker although I can't profile since it's a late binded DLL that runs through the Outlook process. It's shows two issues on the test machine where it's failing but I feel like they may be false flags.

Here's a screen shot of the log:

https://dl.dropbox.com/u/5064321/depends.png

[rookpsu]

As an update: I profiled Outlook.EXE while attempting to run my add-in. We do get an error:

00:00:15.109: Loaded "c:\program files\microsoft office\office14\SKINFEATUREVB6.DLL" at address 0x10000000 by thread 1 "Main". Successfully hooked module.
00:00:15.109: Loaded "c:\windows\system32\MSVCP60.DLL" at address 0x694B0000 by thread 1 "Main". Successfully hooked module.
00:00:15.109: DllMain(0x694B0000, DLL_PROCESS_ATTACH, 0x00000000) in "c:\windows\system32\MSVCP60.DLL" called by thread 1 "Main".
00:00:15.109: DllMain(0x694B0000, DLL_PROCESS_ATTACH, 0x00000000) in "c:\windows\system32\MSVCP60.DLL" returned 1 (0x1) by thread 1 "Main".
00:00:15.109: DllMain(0x10000000, DLL_PROCESS_ATTACH, 0x00000000) in "c:\program files\microsoft office\office14\SKINFEATUREVB6.DLL" called by thread 1 "Main".
00:00:15.125: First chance exception 0xC0000005 (Access Violation) occurred in "c:\program files\microsoft office\office14\SKINFEATUREVB6.DLL" at address 0x10001000 by thread 1 "Main".
00:00:15.125: DllMain(0x694B0000, DLL_PROCESS_DETACH, 0x00000000) in "c:\windows\system32\MSVCP60.DLL" called by thread 1 "Main".
00:00:15.125: DllMain(0x694B0000, DLL_PROCESS_DETACH, 0x00000000) in "c:\windows\system32\MSVCP60.DLL" returned 1 (0x1) by thread 1 "Main".
00:00:15.125: Unloaded "c:\program files\microsoft office\office14\SKINFEATUREVB6.DLL" at address 0x10000000 by thread 1 "Main".
00:00:15.125: Unloaded "c:\windows\system32\MSVCP60.DLL" at address 0x694B0000 by thread 1 "Main".
00:00:15.125: LoadLibraryA("SkinFeatureVB6.dll") returned NULL by thread 1 "Main". Error: Invalid access to memory location (998).

[rookpsu]

I can now confirm this is NOT a Vista / Win 7 issue but an Outlook/Office 2010 issue. Office 2003 and 2007 work fine, but 2010 fails.

[Nightwalker83]

Is it possible for you to run the code via the IDE? If so post which line causes the error.

[rookpsu]

Is it possible for you to run the code via the IDE? If so post which line causes the error.

Yes, running it from the IDE, and therefore in the VB6.EXE process, runs without issue.

The code blows up the first time I attempt to call a function from the problematic DLL.

[si_the_geek]

Thread moved from the 'CodeBank VB6' forum (which is for you to post working code examples, not questions) to the 'VB6 and earlier' forum

[rookpsu]

Any ideas?

[rookpsu]

OK. The problem was the introduction of DEP (Data Execution Prevention) in Outlook 2010. My legacy, third party DLL is apparently accessing some memory in a way that DEP disagrees with. Disabling it in the Outlook Trust Center allows it to run. Unfortunately, this solution likely won't fly with my client.

I attempted to disable DEP for JUST my add-in via System Properties > Performance > DEP however that didn't do the trick. I also added, the offending DLL to that list, in addition to the VB6 Virtual machine DLL that it runs under and the Outlook.EXE file. It still doesn't work.

If anyone is familiar with DEP in this context and could point me in the right direction, i'd be greatly appreciative.

[rookpsu]

If you view Outlook 2010 using ProcessExplorer, you'll see that Outlook runs with a DEP (Permanent) designation which I BELIEVE precludes exclusions. Ugh.