菜单

python搜罗百科,python达成提取百度查寻结果的点子

2020年1月6日 - 澳门太阳娱乐手机登录

python搜集百度完备的章程,python采撷百科

正文实例陈述了python采撷百度周密的情势。分享给大家供我们参谋。具体如下:

#!/usr/bin/python
# -*- coding: utf-8 -*-
#encoding=utf-8 
#Filename:get_baike.py
import urllib2,re
import sys
def getHtml(url,time=10):
 response = urllib2.urlopen(url,timeout=time)
 html = response.read()
 response.close()
 return html
def clearBlank(html):
 if len(html) == 0 : return ''
 html = re.sub('r|n|t','',html)
 while html.find(" ")!=-1 or html.find(' ')!=-1 :
  html = html.replace(' ',' ').replace(' ',' ')
 return html
if __name__ == '__main__':
  html = getHtml('http://baike.baidu.com/view/4617031.htm',10)
  html = html.decode('gb2312','replace').encode('utf-8') #转码
  title_reg = r'<h1 class="title" id="[d]+">(.*?)</h1>'
  content_reg = r'<div class="card-summary-content">(.*?)</p>'
  title = re.compile(title_reg).findall(html)
  content = re.compile(content_reg).findall(html)
  title[0] = re.sub(r'<[^>]*?>', '', title[0])
  content[0] = re.sub(r'<[^>]*?>', '', content[0])
  print title[0]
  print '#######################'
  print content[0]

澳门太阳娱乐手机登录,python搜罗百科,python达成提取百度查寻结果的点子。盼望本文所述对大家的Python程序设计有所扶持。

本文实例呈报了python搜聚百度周密的不二法门。分享给大家供我们参照他事他说加以考查。具体如下:
#!/usr/bin/python#…

python落成提取百度搜索结果的点子,python百度查寻

正文实例陈说了python实现提取百度搜索结果的艺术。分享给大家供大家仿效。具体落到实处格局如下:

# coding=utf8
import urllib2
import string
import urllib
import re
import random
#设置多个user_agents,防止百度限制IP
user_agents = ['Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) Gecko/20130406 Firefox/23.0', 
    'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0', 
    'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533+ 
    (KHTML, like Gecko) Element Browser 5.0', 
    'IBM WebExplorer /v0.94', 'Galaxy/1.0 [en] (Mac OS X 10.5.6; U; en)', 
    'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)', 
    'Opera/9.80 (Windows NT 6.0) Presto/2.12.388 Version/12.14', 
    'Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) 
    Version/6.0 Mobile/10A5355d Safari/8536.25', 
    'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) 
    Chrome/28.0.1468.0 Safari/537.36', 
    'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0; TheWorld)']
def baidu_search(keyword,pn):
  p= {'wd': keyword} 
  res=urllib2.urlopen(("http://www.baidu.com/s?"+urllib.urlencode(p)+"&pn={0}&cl=3&rn=100").format(pn))
  html=res.read()
  return html
def getList(regex,text):
  arr = []
  res = re.findall(regex, text)
  if res:
    for r in res:
      arr.append(r)
  return arr
def getMatch(regex,text):
  res = re.findall(regex, text)
  if res:
    return res[0]
  return ""
def clearTag(text):
  p = re.compile(u'<[^>]+>')
  retval = p.sub("",text)
  return retval
def geturl(keyword):
  for page in range(10):
    pn=page*100+1
    html = baidu_search(keyword,pn)
    content = unicode(html, 'utf-8','ignore')
    arrList = getList(u"<table.*?class="result".*?>.*?</a>", content)
    for item in arrList:
      regex = u"<h3.*?class="t".*?><a.*?href="(.*?)".*?>(.*?)</a>"
      link = getMatch(regex,item)
      url = link[0]
      #获取标题
      #title = clearTag(link[1]).encode('utf8')
      try:
        domain=urllib2.Request(url)
        r=random.randint(0,11)
        domain.add_header('User-agent', user_agents[r])
        domain.add_header('connection','keep-alive')
        response=urllib2.urlopen(domain)
        uri=response.geturl()
        print uri
      except:
        continue
if __name__=='__main__':
  geturl('python')

愿意本文所述对我们的Python程序设计具备助于。

本文实例呈报了python达成提取百度搜索结果的措施。分享给大家供我们参照他事他说加以考查。具体完成方…

相关文章

发表评论

电子邮件地址不会被公开。 必填项已用*标注

网站地图xml地图