We're using a third-party dll provided by a hardware manufacturer, apparently the developers thought it's a great idea to show messages from dll in stead of just returning an error code.

We're using a third-party dll provided by a hardware manufacturer, apparently the developers thought it's a great idea to show messages from dll in stead of just returning an error code.

The problem:
- dll calls MessageBoxA whenever it feels like, we don't know all cases in which it display a message, the application is a kiosk, therefore it's unacceptable

Partial solution:
- currently, using https://code.google.com/p/delphi-detours-library/ we can intercept all calls to MessageBoxA and log them, however, ideally, we would want to hook just the call from the dll or somehow, figure out that the caller is the dll

Any help is appreciated!

Comments

  1. Depending on the use-case you could either run a thread which automatically answers / closes the message boxes. Or (evil!) you  patch the dll, disabling calls to the MessageDialog functions.

    ReplyDelete
  2. Oliver Funcke thank you for your input:
    thread, auto-click : we've solved it a bit more elegant by patching the MessageBoxA(windows api)

    patch dll : this isn't very useful, if tomorrow we get a hotfix, we need to patch that as well, can't just replace the older version...

    ReplyDelete
  3. What about that API intercept toolkit that was posted a few weeks back?

    ReplyDelete
  4. in theory you could do some kind of stack trace, and then walk through the caller list...
    But hmmm, I think jclDebug.pas also contained a function to get the caller, and based on the address you could check if it is from the dll. So yes it is possible but no direct ready to use code for it

    ReplyDelete
  5. Lars Fosdal unless it's DDetours(posted link above), I don't know which toolkit you're referring to

    ReplyDelete
  6. That was the one I was thinking of.

    ReplyDelete
  7. Dorin Duminica You wouldn't do this manually, of course. A small patching tool to search and destroy could automtize that.

    But, isn't it possible to the provider of the dll that these messages can be switched off optionally?

    ReplyDelete
  8. quick search (no demo yet :):
    var p := jclDebug.Caller();
    var dll_or_exe := jclDebug.ModuleOfAddr(p)

    ReplyDelete
  9. Does the DLL open the dialog with a parent hWnd ?

    ReplyDelete
  10. Lars Fosdal not sure which toolkit you're referring to

    Oliver Funcke there's no option in documentation or anything remotely connected as an exported function

    André Mussche do I have to install the whole JCL pack for that? ):

    ReplyDelete
  11. Lars Fosdal yes, it does pass a hwnd value different than "GetDesktopWindow()", but, we don't know all the cases in which it will show the dialog and can't rely that it will always pass one.

    ReplyDelete
  12. You can path the DLL to use USER31.DLL instead of USER32.DLL then you create this DLL with an empty MessageBoxA function and all the functions required by the DLL
    ---
    library USER31;

    uses
      Windows;

    function MessageBoxA(hWnd: HWND; lpText, lpCaption: PAnsiChar; uType: UINT): Integer; stdcall;
    begin
      // do nothing
    end;

    export
      // export all what required by the DLL directly from Windows.pas
      MessageBoxA;
    begin
    end.

    ReplyDelete
  13. yes and no: I made an extract of  the files needed for jclDebug for my AsmProfiler, so you can also get (somewhat older) files from my dir: https://code.google.com/p/asmprofiler/source/browse/trunk/EXT/jcl/

    ReplyDelete
  14. Just put a breakpoint at the start of your version of MessageBoxA API, wait to hit it when an error occurs and look at the callstack. If there is no callstack that leads to the function you seek show CPU and hit Run Until Return (Shift+F8) and Step Over (F8) or Trace Into (F7). This will land you at the function that shows the error messages, scroll up until you locate the prolog code for this function and you will have the address of the function you need to hook. You will also have to find the types of the parameters if any and the calling conversion. If the function you seek is before the function you are just hit Run Until Return and repeat the rest of the steps.

    ReplyDelete
  15. Instead of trying to patch and tweak the dll, have you considered just ask the supplier to modify this dll, if necessary by reminding him the incongruity of having put a GUI in a dll?

    ReplyDelete
  16. I guess what you would need is API hooking like you can do with madCodeHook as described here: http://help.madshi.net/ApiCodeHooking.htm

    ReplyDelete
  17. As others pointed out:
    1. Make the DLL call a patched version of user32 that redirects MessageBoxA.
    2. Contact the vendor. He might not be aware of the problem. We have quite a good experience with that approach.

    ReplyDelete

Post a Comment