TheCPUWizard
March 16th, 2004, 10:18 PM
I am in the process of developing a Web Site probe for monitoring the health of various sites I control. The idea is to retrieve all of the links from a page and then traverse the links to determine what they are linked to and if they are valid.
Speed is of the utmost concern here. I am dealing with a few hundred sites, each with a few thousand links.
I am interested in seeing what people can come up with (multi-threaded solutions are a basic necessity). The server this will be running on is a Quad Processor with a direct connection to a fiber backbone. It should be capable of tremendous throughput if properly implemented.
Speed is of the utmost concern here. I am dealing with a few hundred sites, each with a few thousand links.
I am interested in seeing what people can come up with (multi-threaded solutions are a basic necessity). The server this will be running on is a Quad Processor with a direct connection to a fiber backbone. It should be capable of tremendous throughput if properly implemented.