Using NT Security DLL To Investigate DCOM Issues

Environment: VC6 SP5, NT4 SP6a

Overview

The usual approach to investigate DCOM issues is analyzing network packets via network monitor [NetMon]. Unfortunately this method has the following drawbacks:

  • Being quite good in displaying SSP and DCE RPC packets NetMon leaves DCOM itself unparsed showing DCOM data in hexadecimal stubs.
  • It is hard to use NetMon for more than just viewing in case when there is a need to play with DCOM data programmatically.

I am going to show below an alternative way of researching DCOM by replacing NT security DLL. After putting our transparent DLL on top of a system one we are going to try to address multihomed server slow activation issue. Currently there are no solutions of multihomed server activation delay available. All suggestions end at swapping network cards or disabling DCOM on one of the IP addresses which obviously do not work for architectures requiring DCOM on both networks.

Replacing NT Security DLL

To allow applications to have standard access to a variety of security protocols, NT security DLL exposes common security support provider interface [SSPI]. (Actually the real DLL has three additional undocumented functions: NtLmSspControl, SealMessage and UnsealMessage.) To allow custom security control MS RPC has the following two registry entries:

  • HKLM\Software\Microsoft\Rpc\SecurityService\10 and
  • HKLM\Software\Microsoft\Rpc\SecurityService\DefaultProvider.
Both entries are set to security.dll by default. Edit them to point to our AKSecurity.dll and reboot the computer. By doing that we do not extend nor replace the standard NT SSPI. Our DLL is absolutely transparent. Being loaded it loads the original NT security.dll and passes all the calls to it. In addition our DLL provides an extensive logging of security buffers passed to MakeSignature and VerifySignature functions. In particular I have expanded remote activation logging to the extent I could.

Investigating DCOM Multihomed Server Slow Activation

Multihomed computer has more than one IP address. In case of remote server activation on multihomed computer the strings are bind to all (two in our case) addresses:

 Version: 0   N of buffers: 5
 Sec buffer[1]: 0x00B2F108 type: 0x80000001 size: 80
    RPC protocol version: 4
    Packet type: 2 flags1: 8 flags2: 0
    Object    ID: {00000000-0000-0000-0000-000000000000}
    Interface ID: {4D9F4AB8-7D1C-11CF-861E-0020AF6E7C57} // IRemoteActivation
    Activity  ID: {BF1B213D-7BDB-11D5-A990-006008A0E9D5}
    Interface version: 0 sequence number: 1 operation number: 0
    Interface hint: 0xFFFF activity hint: 0x0060 fragment number: 0
    Authentication: 0x000A serial: 0.0
 Sec buffer[2]: 0x00B2F114 type: 0x00000001 size: 704
    ORPC that flags: 1
   RemoteActivation
    OXID: 2920746192717563652
    Dual String Array size: 216
     String bindings
      1 tower: 0x0008   192.168.5.1[2165] // 1st Multihomed IP address
      2 tower: 0x0008   10.1.99.230[2165] // 2nd Multihomed IP address

Microsoft OXID resolver uses simplified sequential algorithm to discover which IP address is accessible from the client. It pings the first address until the given timeout occurs. Then it tries the next one... If the first IP address is not visible from the client the default activation delay is going to be around 34 seconds over UDP and couple of minutes over TCP. The parallel OXID resolver algorithm could send requests to all IP addresses simultaneously and catch the first responded. Even the sequential algorithm could be optimized by sorting IP addresses based on some criteria.

Swapping Bindings on Client

Let's try to use our DLL for a little more than just logging. When it is getting loaded let it find out its local IP address and store it in some global variable:

    WSADATA wsadata;
    WSAStartup(0x0101, &wsadata);
    char hostname[128];
    gethostname(hostname, 128);
    hostent * phost = gethostbyname(hostname);
    g_MyIP = *((in_addr *)phost->h_addr);
    WSACleanup(); 

Now activating multihomed server let our DLL try to swap string bindings based on the following simple criteria:

// The function tries to calculate 
// how close 2 ip addresses are
// max value 15 
short CalcCritIP(in_addr ip1, in_addr ip2)
{
    short crit = 0;
    if( ip1.S_un.S_un_b.s_b1 == ip2.S_un.S_un_b.s_b1 ) crit += 8;
    if( ip1.S_un.S_un_b.s_b2 == ip2.S_un.S_un_b.s_b2 ) crit += 4;
    if( ip1.S_un.S_un_b.s_b3 == ip2.S_un.S_un_b.s_b3 ) crit += 2;
    if( ip1.S_un.S_un_b.s_b4 == ip2.S_un.S_un_b.s_b4 ) crit += 1;
    return crit;
}

As expected this approach removes multihomed server activation delay.

Conclusions

Custom NT security DLL can be used for DCOM, DCE RPC, SSP debugging. It can be extended to provide extensive logging. It also can be used for some remote protocols tweaking.

References


Downloads

Download AKSecurity project - 20 Kb
Download Echo project - 28 Kb


Comments

  • Can't compile

    Posted by Legacy on 02/01/2002 12:00am

    Originally posted by: kurt

    I open the .DSP file in Visual Studio 6, and press [F7] to compile the .DLL. I receive 45 errors and 4 warnings. Is anyone able to compile, and repost the .DLL? Or, suggest what to reconfigure in VS6 so I can compile this .DLL myself?

    Much thanks.
    Regards,
    Kurt

    ==== sample of errors from Visual Studio 6 ====
    aksecurity.h(359) : error C2039: 'EXTENDED_NAME_FORMAT' : is not a member of 'SEC'
    aksecurity.h(359) : error C2065: 'EXTENDED_NAME_FORMAT' : undeclared identifier
    aksecurity.h(359) : error C2146: syntax error : missing ')' before identifier 'NameFormat'
    aksecurity.h(359) : warning C4229: anachronism used : modifiers on data are ignored
    aksecurity.h(362) : error C2059: syntax error : ')'
    aksecurity.h(367) : error C2039: 'EXTENDED_NAME_FORMAT' : is not a member of 'SEC'
    aksecurity.h(367) : error C2146: syntax error : missing ')' before identifier 'NameFormat'

    Reply
  • It does not work in windows 2000

    Posted by Legacy on 09/30/2001 12:00am

    Originally posted by: naridingjie

    security.dll in windows NT is changed to secur32.dll in windows 2000
    

    Reply
Leave a Comment
  • Your email address will not be published. All fields are required.

Top White Papers and Webcasts

  • The explosion in mobile devices and applications has generated a great deal of interest in APIs. Today's businesses are under increased pressure to make it easy to build apps, supply tools to help developers work more quickly, and deploy operational analytics so they can track users, developers, application performance, and more. Apigee Edge provides comprehensive API delivery tools and both operational and business-level analytics in an integrated platform. It is available as on-premise software or through …

  • As mobile devices have pushed their way into the enterprise, they have brought cloud apps along with them. This app explosion means account passwords are multiplying, which exposes corporate data and leads to help desk calls from frustrated users. This paper will discover how IT can improve user productivity, gain visibility and control over SaaS and mobile apps, and stop password sprawl. Download this white paper to learn: How you can leverage your existing AD to manage app access. Key capabilities to …

Most Popular Programming Stories

More for Developers

Latest Developer Headlines

RSS Feeds