invalid hook procedure

[dave2k]

hi guys, i have a class mehod which tries to set a hook from my dll, but is throwing an invalid hook procedure error. What is the most likely cause of this?

my calling code and dll code is below

calling:bool Window::SetHook()
{
//SetCurrentDirectory(".");
// Find the child RICHEDIT control and return its handle
HWND hwndCntrl = ::FindWindowEx(hwnd, NULL, "RICHEDIT", NULL);

if(!::IsWindow(hwndCntrl)) return false;

// Load HookProc.dll
hinstDLL = LoadLibrary((LPCTSTR)"HP.dll");
//if(hinstDLL == NULL) return false;
HOOKPROC hkprcSysMsg = (HOOKPROC)GetProcAddress(hinstDLL, "CallWndProc");


// Set the hook
DWORD ThreadIdCntrl = GetWindowThreadProcessId(hwndCntrl, 0);
hh = ::SetWindowsHookEx(WH_CALLWNDPROC, hkprcSysMsg, hinstDLL, ThreadIdCntrl);

Util::ReportLastError();
return true;
}


hook procedure(dll):// HProc.cpp : Defines the entry point for the DLL application.

#include "stdafx.h"

//extern "C" __declspec(dllexport) LRESULT CALLBACK CallWndProc(int nCode, WPARAM wParam, LPARAM lParam);

CWPSTRUCT *lpMsg;

LRESULT CALLBACK CallWndProc(int nCode, WPARAM wParam, LPARAM lParam)
{
MessageBeep(0);
if(nCode < 0) return CallNextHookEx(0, nCode, wParam, lParam);
char* test = "dave";
lpMsg =(CWPSTRUCT *)lParam;
COPYDATASTRUCT cds = {0};

/* size of message */
cds.cbData = 200;
cds.dwData = 0;
cds.lpData = (char*)lpMsg->lParam;

HWND hGetLines = FindWindow(NULL, "GetLines");
//SendMessage(hGetLines, WM_COPYDATA, (WPARAM)lpMsg->hwnd, (LPARAM)&cds);

switch(lpMsg->message)
{
case EM_REPLACESEL:
MessageBeep(0);
SendMessage(hGetLines, WM_COPYDATA, (WPARAM)lpMsg->hwnd, (LPARAM)&cds);
//SendNotifyMessage(hGetLines, WM_COPYDATA, (WPARAM)lpMsg->hwnd, (LPARAM)&cds);
break;

default:
break;
}

return CallNextHookEx(0, nCode, wParam, lParam);
}

cheers

[dave2k]

ok, when i call the error function afert hinstDLL = LoadLibrary((LPCTSTR)"HP.dll");it says The handle is invalid. why would this be? in my .h file i have static HINSTANCE hinstDLL; and at the top of the c== file, it's initialised as HINSTANCE Window::hinstDLL = NULL;cheers

[kirants]

on what API call do you get this error ?
Also, are you sure LoadLibrary is returning non NULL ?

[dave2k]

when i take the dll out of the directory, the error message is basically 'dll missing', so i am shure it's picking it up.

I think it's somthing to do with LoadLibrary is being inside a class method, and the dll method is not todo with a class?

i get the The handle is invalid error after LoadLibrary,

cheers

[dave2k]

can anybody find anything obviously wrong with my code?

cheers :)

[dave2k]

take that as a no then :(

[kirants]

You mean, something like this gives you the error ?

HMODULE hMod = LoadLibrary(_T("HP.dll"));
if(NULL == HMod)
{
DWORD dwRet = GetLastError();
//is dwRet indicating ERROR_INVALID_HANDLE ?
}

[dave2k]

hi kirants,

i have an error function which basicaly call GetLastError and FormatMessage: HWND hwndCntrl = FindWindowEx(hwnd, NULL, "RICHEDIT", NULL);

if(!::IsWindow(hwndCntrl)) return false;
DWORD ThreadIdCntrl = GetWindowThreadProcessId(hwndCntrl, 0);

HOOKPROC hkprcSysMsg;
static HINSTANCE hinstDLL;
static HHOOK hhookSysMsg;

hinstDLL = LoadLibrary((LPCTSTR) "HP.dll");
Util::ReportLastError();
hkprcSysMsg = (HOOKPROC)GetProcAddress(hinstDLL, "CallWndProc");
hhookSysMsg = SetWindowsHookEx(WH_CALLWNDPROC,hkprcSysMsg,hinstDLL,ThreadIdCntrl);

return true;

as you can see, i call it straight after LoadLibrary, and a message box comes up with "The handle is invalid"

cheers

p.s. i tested for an error before the LoadLibrary line, and there were no errors, meaning loadlibrary failed

[kirants]

cool. That's better.
LoadLibrary is failing, as is evident.
Why would it fail.
Possible reasons:

HP.dll is not present in the exe location, or any other search paths used by windows. You said, the dll is present. So that is ruled out.
The DllMain of HP.DLL returned FALSE. Highly unlikely.
HP.dll may be dependent on other dlls that needs to be implicitly loaded too before HP.dll loads. If any of these fail to load, HP.dll will fail to load too.


I am suspecting point 3. To verify, drop your HP.dll into system32 folder and see if this error happens still.
Also, do you know how to use depends program available with Visual Studio. Try opening HP.dll using that. It shows a list of dependent DLLs. All these dlls must be present in the right paths for windows to be able to load them

[dave2k]

To verify, drop your HP.dll into system32 folder and see if this error happens still. i did this, and the error still happens. what does this mean?

[kirants]

When you run, depends.exe and open HP.dll, what DLLs does it show in it's dependency list ?

BTW, what is hinstDLL on return from LoadLibrary ? Is it NULL ?

[dave2k]

it's not null.

funny, my code works perfectly when i compile my .dll in devc++..

really weird

[kirants]

if it is not NULL , then why are you using GetLastError() ?? ;)

[dave2k]

cus it still causes an error

[kirants]

Hm.. At exactly what point does it cause an error ? i.e. which exact API is failing ?
It is important to locate that. Earlier, we spent time looking at something which wasn't even a problem.