redis的io多路復用機制,遇到Io阻塞時會切換任務之【爬蟲版】

 2023-12-25 阅读 28 评论 0

摘要:? ? #! /usr/bin/env python3 # -*- coding:utf-8 -*-from urllib import request import gevent,time from gevent import monkeymonkey.patch_all() # 把當前程序中的所有io操作都做上標記def spider(url):print("GET:%s" % url)resp = request.urlopen(url)dat

?

?

#! /usr/bin/env python3
# -*- coding:utf-8 -*-from urllib import request
import gevent,time
from gevent import monkeymonkey.patch_all()    # 把當前程序中的所有io操作都做上標記def spider(url):print("GET:%s" % url)resp = request.urlopen(url)data = resp.read()print("%s bytes received from %s.." % (len(data), url))urls = ["https://weibo.com/","https://www.baidu.com/","https://www.yinxiang.com/"
]start_time = time.time()
for url in urls:spider(url)
print("同步耗時:",time.time() - start_time)async_time_start = time.time()
gevent.joinall([gevent.spawn(spider,"https://weibo.com/"),gevent.spawn(spider,"https://www.baidu.com/"),gevent.spawn(spider,"https://www.yinxiang.com/"),
])
print("異步耗時:",time.time() - async_time_start)

redis的io多路復用機制,輸出

GET:https://weibo.com/
6117 bytes received from https://weibo.com/..
GET:https://www.baidu.com/
227 bytes received from https://www.baidu.com/..
GET:https://www.yinxiang.com/
16505 bytes received from https://www.yinxiang.com/..
同步耗時: 0.23460006713867188
GET:https://weibo.com/
GET:https://www.baidu.com/
GET:https://www.yinxiang.com/
227 bytes received from https://www.baidu.com/..
16505 bytes received from https://www.yinxiang.com/..
6117 bytes received from https://weibo.com/..
異步耗時: 0.0946660041809082

?

waitforsingleobject阻塞?轉載于:https://www.cnblogs.com/sea-stream/p/10317051.html

版权声明:本站所有资料均为网友推荐收集整理而来,仅供学习和研究交流使用。

原文链接:https://hbdhgg.com/1/194904.html

发表评论:

本站为非赢利网站,部分文章来源或改编自互联网及其他公众平台,主要目的在于分享信息,版权归原作者所有,内容仅供读者参考,如有侵权请联系我们删除!

Copyright © 2022 匯編語言學習筆記 Inc. 保留所有权利。

底部版权信息