搜索引擎中crawler的設(shè)計(jì)、實(shí)現(xiàn)與擴(kuò)展優(yōu)化

1、電子科技大學(xué)碩士學(xué)位論文搜索引擎中Crawler的設(shè)計(jì)、實(shí)現(xiàn)與擴(kuò)展優(yōu)化姓名：楊帥申請(qǐng)學(xué)位級(jí)別：碩士專業(yè)：計(jì)算機(jī)系統(tǒng)結(jié)構(gòu)指導(dǎo)教師：劉玓20090501ABSTRACTABSTRACTSearch吼gineisasoftwaresystemwhichisappliedonWebItusessomestrategytodiscoverandcollectinformationonWebAftertheinformationisprocessi

2、ngandorganization，itprovidesuserswi伍webinformationserviceTllispaperfirstlymakesafullanalysisofsearch饑畫晴’Scoretechnologies，including：indexersearcherspiderwebsitequality硒sessmcntalgorith皿lexiealanalysis，Chinesewordsegmenta

3、tiontechniques，invertedfiletechnologyBooleanquerytheoryThenonbasicofsearchengine’s∞他technologies，basedonalightweightarchitectureitsthreemainmodulesworedesigned：crawlerindexerandsearcherAndmainlyfocusontheimplementationof

4、thewebpagegathermodule繆舒pagegathermodule：onbasicofimplementingitscorefunctionthispaperputforwardsomeoptimizationmethodsasfollows：1Incrementalmodd：onlyupdatepartofthepagestorefreshthepageset皿：lismothod鋤significantlyreduce

5、thenumberofbulkupdatethusimprovethefreshnessofthepagesets2Distributedstrategy：putsinglecollectionnodetomultiplenodes，nodescancommunicatewitheachotherandaddacontrolnodetocoordinateothernodes3Websiteweightcalculation：usedt

6、o雒SesStheimportanceofwebpagesItusesGoogle’SPageRankalgorithm4Expandthediskstoragemethod：risemechanismssuchasinheritanceandderivationofobject—orientedlanguagetosupportdata：basestorageandfaulttolerantfileformat5Anewwebsite

7、crawlstrategy：giveeverywebpageaweightnumberbasedontimerstrategywhencrawlingpagesforthefirsttime，anddecidewhichpageistovisitedfirstnexttimeaccordingtotheweightnumberthusavoidbandwidthwasteonlowperformanceserversIndexmodul