Any tips for debugging multi-threaded services?

[monalin]

Hi guys, I've been trying to work on this web based chat client for awhile now it works pretty well and is almost complete. I'm at the point now where I'm trying to fix the last few bugs that pop up now and then. There's one bug left that I know of and it's being very stubborn so I was hoping to see if anyone who has experience debugging multi-threaded applications. I can't post the code here because there are thousands and thousands of lines but i can explain briefly how it works and whats happening.

I have a Win32 Service which is holding all the messages that are sent through the web based chat. Each user in the chat room always has an AJAX request waiting for a new message to be sent from other users. What I mean is that its not a standard polling AJAX chat where there is a request sent every second to check for a new message. This system makes a request and waits for a message. Times out in 45 seconds if not messages were sent. That being said...

When a new message is sent all the clients that are connected to the service are returned with the contents of the new message. Even if a client was not waiting for the message when it was sent they will receive it next time they look for any new messages because each client is keeps track of what messages its been sent and compares it with the queue currently on the server.

Every once in awhile a message will not get delivered back to the user who sent it, but everyone else will receive it. Cannot put a finger on this, doesn't happen often maybe once every 1,000 messages sent, but it could be the first message they send or the last message. Anyways I'm not looking for someone to solve the problem for me, though if you have suggestions they are defiantly welcome. I'm looking for guidance on how to approach debugging something of this nature. It's hard for me to reproduce this in a development environment and running a debugger on the live server is not an option at this point.

Using VS 2008 Team Foundation Server and .NET 3.5 SP1

[Arjay]

If you haven't already done so I would any message errors to the event log. I would also employ some sort of tracing to log files with timestamps so in order that you may identify a pattern on how or where the error occurs.

Lastly, as a hack you may consider marking a message with a sender id. Since you mentioned you compared the client list with the server list, if you identify that a message is present in the client, but not in the server, you can take additional action because you know that the missing message is from your client.

[monalin]

If you haven't already done so I would any message errors to the event log. I would also employ some sort of tracing to log files with timestamps so in order that you may identify a pattern on how or where the error occurs.

Lastly, as a hack you may consider marking a message with a sender id. Since you mentioned you compared the client list with the server list, if you identify that a message is present in the client, but not in the server, you can take additional action because you know that the missing message is from your client.

Thanks for the advice. I already have error messages being sent to the event log however its not throwing an error. I wish it was throwing an error because then I would know where to begin looking hah.

I already have the sender id attached to the messages because I have to handle them slightly different on the client side. I don't actually compare a list of messages received I just record the last message that was received for a specific client. I would like to store a list of all the messages sent and received but not all clients receive all messages. There are PM's between clients that are only seen by moderators and the two involved parties and then there are specific messages sent to only one person.

Going to take another look at the bug today and approach it from another angle hopefully with better luck than before. Thanks for the help.