网络爬虫的设计与实现+文献综述
关键词：搜索引擎；JAVA；广度优先.
The Design and Implementation of
Distributed Web Crawler
Abstract: With the rapid development of Internet, search engines as the main entrance of the Internet plays a more and more important role. Web crawler is a very important part of the search engines, a program which can auto collect information form Internet,which is responsible to collect web pages from Internet. These pages are used to build index and provide support for search engines. Spider can collect data for search engines ,also can be a directional information collector,collects specifically informations from some web sites,such as HR informations,house rent informations.In this paper,use JAVA implements a breadth-first algorithm Spider. The paper，discussing from the application of the search engine，searches the importance and function of Web Crawler in the search engine,and puts forward its demand of function and design．On the base of analyzing Web Crawler’s system strtucture and working elements,this paper also researches the method and strategy of multithreading scheduler，Web page crawling and HTML parsing．And then,a program of web page crawling based on Java is applied and analyzed．Through the crawler can collect a site or multiple site URL.Links outside the network,you can crawl most of China’s major large-scale portal sites,such as:Baidu , Sina , Netease.
Keywords：Search Engine;JAVA;Breadth First Search
目录
目录 iii
1 绪论 1
1.1 课题开发背景 1
1.2 网络爬虫研究现状 3
1.3 网络爬虫的工作原理 4
1.4 论文的组织结构 4
2 系统开发工具和平台及相关技术介绍 5
2.1 程序开发工具 5
2.1.1 JDK 5
2.1.2 MYECLIPSE 5
2.1.3 TOMCAT 6
2.1.4 SERVLET原理及特点 7
2.2 解析HTML 8
2.3 网络爬虫常用搜索策略 8
2.3.1 宽度或深度优先搜索策略 8
2.3.2 宽度优先搜索算法 9
2.3.3 深度优先搜索 9
2.3.4 聚焦搜索策略 9
2.3.5 基于链接结构评价的搜索策略 10
3 系统分析 12
3.1 系统可行性分析 12
3.1.1 系统上的可行性 12
3.1.2 经济上的可行性 12
3.1.3 管理上的可行性 12
3.2 需求分析 12
3.2.1 用户需求 13
3.2.2 性能需求 13