Using NT Security DLL To Investigate DCOM Issues

Environment: VC6 SP5, NT4 SP6a

Overview

The usual approach to investigate DCOM issues is analyzing network packets via network monitor [NetMon]. Unfortunately this method has the following drawbacks:

• Being quite good in displaying SSP and DCE RPC packets NetMon leaves DCOM itself unparsed showing DCOM data in hexadecimal stubs.
• It is hard to use NetMon for more than just viewing in case when there is a need to play with DCOM data programmatically.

I am going to show below an alternative way of researching DCOM by replacing NT security DLL. After putting our transparent DLL on top of a system one we are going to try to address multihomed server slow activation issue. Currently there are no solutions of multihomed server activation delay available. All suggestions end at swapping network cards or disabling DCOM on one of the IP addresses which obviously do not work for architectures requiring DCOM on both networks.

Replacing NT Security DLL

To allow applications to have standard access to a variety of security protocols, NT security DLL exposes common security support provider interface [SSPI]. (Actually the real DLL has three additional undocumented functions: NtLmSspControl, SealMessage and UnsealMessage.) To allow custom security control MS RPC has the following two registry entries:

• HKLM\Software\Microsoft\Rpc\SecurityService\10 and
• HKLM\Software\Microsoft\Rpc\SecurityService\DefaultProvider.

Investigating DCOM Multihomed Server Slow Activation

Multihomed computer has more than one IP address. In case of remote server activation on multihomed computer the strings are bind to all (two in our case) addresses:

 Version: 0   N of buffers: 5
 Sec buffer[1]: 0x00B2F108 type: 0x80000001 size: 80
    RPC protocol version: 4
    Packet type: 2 flags1: 8 flags2: 0
    Object    ID: {00000000-0000-0000-0000-000000000000}
    Interface ID: {4D9F4AB8-7D1C-11CF-861E-0020AF6E7C57} // IRemoteActivation
    Activity  ID: {BF1B213D-7BDB-11D5-A990-006008A0E9D5}
    Interface version: 0 sequence number: 1 operation number: 0
    Interface hint: 0xFFFF activity hint: 0x0060 fragment number: 0
    Authentication: 0x000A serial: 0.0
 Sec buffer[2]: 0x00B2F114 type: 0x00000001 size: 704
    ORPC that flags: 1
   RemoteActivation
    OXID: 2920746192717563652
    Dual String Array size: 216
     String bindings
      1 tower: 0x0008   192.168.5.1[2165] // 1st Multihomed IP address
      2 tower: 0x0008   10.1.99.230[2165] // 2nd Multihomed IP address

Microsoft OXID resolver uses simplified sequential algorithm to discover which IP address is accessible from the client. It pings the first address until the given timeout occurs. Then it tries the next one... If the first IP address is not visible from the client the default activation delay is going to be around 34 seconds over UDP and couple of minutes over TCP. The parallel OXID resolver algorithm could send requests to all IP addresses simultaneously and catch the first responded. Even the sequential algorithm could be optimized by sorting IP addresses based on some criteria.

Swapping Bindings on Client

Let's try to use our DLL for a little more than just logging. When it is getting loaded let it find out its local IP address and store it in some global variable:

    WSADATA wsadata;
    WSAStartup(0x0101, &wsadata);
    char hostname[128];
    gethostname(hostname, 128);
    hostent * phost = gethostbyname(hostname);
    g_MyIP = *((in_addr *)phost->h_addr);
    WSACleanup(); 

Now activating multihomed server let our DLL try to swap string bindings based on the following simple criteria:

// The function tries to calculate 
// how close 2 ip addresses are
// max value 15 
short CalcCritIP(in_addr ip1, in_addr ip2)
{
    short crit = 0;
    if( ip1.S_un.S_un_b.s_b1 == ip2.S_un.S_un_b.s_b1 ) crit += 8;
    if( ip1.S_un.S_un_b.s_b2 == ip2.S_un.S_un_b.s_b2 ) crit += 4;
    if( ip1.S_un.S_un_b.s_b3 == ip2.S_un.S_un_b.s_b3 ) crit += 2;
    if( ip1.S_un.S_un_b.s_b4 == ip2.S_un.S_un_b.s_b4 ) crit += 1;
    return crit;
}

As expected this approach removes multihomed server activation delay.

Conclusions

Custom NT security DLL can be used for DCOM, DCE RPC, SSP debugging. It can be extended to provide extensive logging. It also can be used for some remote protocols tweaking.

References

• Programming Windows Security
• Professional DCOM Programming

Downloads