網(wǎng)絡(luò)爬蟲(chóng)源代碼07320

1、網(wǎng)絡(luò)爬蟲(chóng)源代碼.txt丶︶￣喜歡的歌，靜靜的聽(tīng)，喜歡的人，遠(yuǎn)遠(yuǎn)的看我笑了當(dāng)初你不挺傲的嗎現(xiàn)在您這是又玩哪出呢？publicclassSpiderimplementsRunnableprivateArrayListurlsURL列表privateHashMapindexedURLs已經(jīng)檢索過(guò)的URL列表privateintthreads初始化線(xiàn)程數(shù)publicstaticvoidmain(Stringargv[])throwsExcept

2、ionif(argv[0]==null)System.out.println(“Missingrequiredargument:[SitURL]“)returnSpiderSpider=newSpider(argv[0])Spider.go()publicSpider(StringstrURL)urls=newArrayList()threads=10urls.add(strURL)threadList=newArrayList()in

3、dexedURLs=newHashMap()if(urls.size()==0)thrownewIllegalArgumentException(“Missingrequiredargument:u[starturl]“)if(threads0)Threadchild=(Thread)threadList.remove(0)child.join()notifyAll()通過(guò)URL解析出網(wǎng)頁(yè)內(nèi)容并解析出頁(yè)面上的URL@paramurl頁(yè)面

4、鏈接@throwsjava.lang.ExceptionprivatevoidindexURL(Stringurl)throwsExceptionbooleanflag=true判斷網(wǎng)頁(yè)鏈接的級(jí)別，系統(tǒng)默認(rèn)為三級(jí)intlevel=1if(indexedURLs.get(url)==null)indexedURLs.put(urlnewInteger(level))elselevel=((Integer)indexedURLs.get(u

5、rl)).intValue()只檢測(cè)到頁(yè)面的第二級(jí)if(level2)returnlevelStringstrBody=nulltry解析頁(yè)面內(nèi)容strBody=loadURL(url)catch(Exceptione)returnif(strBody!=null)StringurlGroups[]=nulltry解析出頁(yè)面所以URLurlGroups=parseURLs(summary)catch(Exceptione)logger.