Tuesday, May 27, 2008

Web Crawler (HTTP) 1

I have been wanting to work on a Web crawler, a program that will go to site automatically and gather information, for quite some time now. There area a number of other projects i want to have it implemented in, but first i need some solid code for the HTTP and socket/TCP part of it. In the Socket (.zip) link you'll find some code that implements a VERY basic socket class. the idea is to incorporate a separate HTTP request class and have the two work together. I ran into some snags, but it works... for now. I need to make it improve the Receive function and build the HTTP request class. Any suggestions just comment or email me. And Btw I'm using some (a very small amount) of code from someone else, I'm not sure who but they deserve some credit (as well as i do if you choose to use this code).

No comments: