Web Crawlers

[Matt Connors]

We have a problem where I work that occurs about the same time every day. We suspect the problem could be caused by a Web Crawler accessing our system. My understanding is that Web Crawlers issues a command that causes only header information to be returned rather than a page's complete content.

What I would like to do is simulate a Web Crawler in an attempt to recreate the problem under controled conditions.

Am I on the right track? Can someone give me a clue how I would go about doing this?

Thanks in advance,
Matt