Python requests: ignore exceptions and errors while connecting to proxy server

Question

I wrote my first program in Python.

#This program casts votes in online poll using different proxy servers for each request.
#It works, but some proxy servers cause errors crashing the whole thing.
#To avoid that, I would like it to skip those servers and ignore the errors.

import requests
import time

#Votes to be cast
votes = 5 

#Makes proxy list
f=open('proxy2.txt')
lines=f.read().splitlines()
f.close()

#Vote counter
i = 1 

#Proxy list counter
j = 0 

while (i<=votes):

#Tests and moves to next proxy if there was a problem.
    try:
        r = requests.get('http://www.google.com')
    except requests.exceptions.RequestException:
        j = j + 1

#Headers copied from my browser. Some of them cause errors. Could you tell me why?
    headers =   {
                    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
                    #'Accept-Encoding': 'gzip, deflate',
                    #'Accept-Language': 'pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4',
                    #'Cache-Control': 'max-age=0',
                    #'Connection': 'keep-alive',
                    #'Content-Length': '101',
                    'Content-Type': 'application/x-www-form-urlencoded',
                    #'Host': 'www.mylomza.pl',
                    #'Origin': 'http://www.mylomza.pl',
                    #'Referer': 'http://www.mylomza.pl/home/lomza/item/11780-wybierz-miss-%C5%82ks-i-portalu-mylomzapl-video-i-foto.html',
                    #'Upgrade-Insecure-Requests': '1',
                    #'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
                }

    proxies =   {
                    'http': 'http://'+lines[j] #31.207.0.99:3128
                }

    r = requests.get('http://www.mylomza.pl/home/lomza/item/11780-wybierz-miss-%C5%82ks-i-portalu-mylomzapl-video-i-foto.html', headers=headers, proxies=proxies, timeout=10)

#The funny part - form, that I have to post, requires some kind of ID and this is my way of getting it :P Feel free to suggest an alternative way.
    userid = r.text[(22222-32):22222]

    print('Voter', userid, 'registered.')

    data = {
                'voteid': '141',
                'task_button': 'Głosuj',
                'option': 'com_poll',
                'task': 'vote',
                'id': '25',
                userid: '1'
               }

    r = requests.post('http://www.mylomza.pl/home/lomza/item/index.php', headers=headers, cookies=r.cookies, data=data, proxies=proxies, timeout=10)

    print('Vote nr', i, 'cast from', lines[i])

    i = i + 1
    j = j + 1

    time.sleep(1)

What I need is to make it handle exceptions and errors.

#Tests and moves to next proxy if there was a problem.
try:
    r = requests.get('http://www.google.com')
except requests.exceptions.RequestException:
    j = j + 1

Beside that I could use an alternative way of achieving this:

#The funny part - form, that I have to post, requires some kind of ID and this is my way of getting it :P Feel free to suggest an alternative way.
    userid = r.text[(22222-32):22222]

Sometimes my method doesn't work (example below). First vote went through, second didn't and then all crashed.

Voter 53bf55490ebd07d9c190787c5c6ca44c registered.
Vote nr 1 cast from 111.23.6.161:80
Voter  registered.
Vote nr 2 cast from 94.141.102.203:8080
Traceback (most recent call last):
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\connection.py", line 142, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\util\connection.py", line 91, in create_connection
    raise err
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\util\connection.py", line 81, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 578, in urlopen
    chunked=chunked)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 362, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\http\client.py", line 1083, in request
    self._send_request(method, url, body, headers)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\http\client.py", line 1128, in _send_request
    self.endheaders(body)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\http\client.py", line 1079, in endheaders
    self._send_output(message_body)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\http\client.py", line 911, in _send_output
    self.send(msg)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\http\client.py", line 854, in send
    self.connect()
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\connection.py", line 167, in connect
    conn = self._new_conn()
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\connection.py", line 147, in _new_conn
    (self.host, self.timeout))
requests.packages.urllib3.exceptions.ConnectTimeoutError: (<requests.packages.urllib3.connection.HTTPConnection object at 0x03612730>, 'Connection to 94.141.102.203 timed out. (connect timeout=10)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\adapters.py", line 403, in send
    timeout=timeout
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 623, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\util\retry.py", line 281, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='94.141.102.203', port=8080): Max retries exceeded with url: http://www.mylomza.pl/home/lomza/item/11780-wybierz-miss-%C5%82ks-i-portalu-mylomzapl-video-i-foto.html (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x03612730>, 'Connection to 94.141.102.203 timed out. (connect timeout=10)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\PollVoter.py", line 50, in <module>
    r = requests.get('http://www.mylomza.pl/home/lomza/item/11780-wybierz-miss-%C5%82ks-i-portalu-mylomzapl-video-i-foto.html', headers=headers, proxies=proxies, timeout=10)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\api.py", line 71, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\adapters.py", line 459, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='94.141.102.203', port=8080): Max retries exceeded with url: http://www.mylomza.pl/home/lomza/item/11780-wybierz-miss-%C5%82ks-i-portalu-mylomzapl-video-i-foto.html (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x03612730>, 'Connection to 94.141.102.203 timed out. (connect timeout=10)'))

Solution 1

PROBLEM SOLVED

Turns out that I shouldn't open more than 1 connection per proxy server.

But I have to make 2 requests. The solution was to send first one from my ip then switch to proxy for second one.

r = requests.get(url, headers=headers, timeout=timeout)

try:
    r = requests.post(url, headers=headers, cookies=r.cookies, data=data, timeout=timeout, proxies=proxies)
except:
        j = j + 1

Works perfectly so far. :)

Solution 2

It looks like you're flooding the server with too many requests, that's why you're getting the other errors like requests.packages.urllib3.exceptions.MaxRetryError, since likely the server throttles the number of connections you can make in a given amount of time. You can try handling all the exceptions listed in your output, and you can also try making fewer attempts at the url you're requesting from.

[Edit] Or if you want to brute force and handle all errors and exceptions, try the following instead

except:
    j = j + 1

[Edit:] You could try https: as well as http:

[Edit] Found this:

If the remote server is very slow, you can tell Requests to wait forever for a response, by passing None as a timeout value and then retrieving a cup of coffee.

r = requests.get('https://github.com', timeout=None)

Solution 3

I had the similar thing and used

except:
    continue

which ment to continue the loop over again in case of exceptions and continue 'trying'

Comments

[guuczi]

Thanks for quick reply. When I don't use proxy I can send as many requests as I like. Problems start when I do use them. What about 'socket.timeout: timed out'?

[jfunk]

I see, it looks like your code is having trouble connecting to the proxies. I would play around with the proxy connection. [EDIT:] You could try https: as well as http:.

[Padraic Cunningham]

Don't use a blank except and at the very least log or print any error message