|
本帖最后由 k992007 于 2022-11-25 22:26 编辑
翻墙软件是 代理 vmess服务器,开启后浏览器能正常翻墙访问国外网页,但是用pycharm 爬虫 无法请求网页一直卡住 请求失败,我怀疑翻墙软件导致了无法请求网页,关闭翻墙软件后一切正常,请问各位大佬 ,pycharm怎么设置才能正常翻墙访问国外网页?报错代码:
requests.excepti**.ProxyError: HTTPSConnectionPool(host='www.baidu.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/e ... xy-error-http-proxy, SSLError(SSLError(1, '[SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:852)'),)))
#代码:
import random
import random
import parsel
import requestsfor page in range(1, 2):
user_agent_list = [
"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; WOW64) Gecko/20100101 Firefox/61.0",
"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)",
"Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9.2.15) Gecko/20110303 Firefox/3.6.15"]
print('正在抓取{}页'.format(page))
url = 'https://www.baidu.com/'
headers = {'User-Agent': random.choice(user_agent_list)}
resp**e = requests.get(url=url, headers=headers)
print(resp**e)
|
|