Cookie authentication with Python requests

Question

I am trying to mimic a user action on a site programmatically using Python requests API. to accomplish this programmatically the request must have user/pass authentication and also should pass few NVPs as Cookies in Header. To get the NVPs I initially make a dummy request and the server returns me the cookies. I acquire the required values from these cookies and use this to send the actual request. But the request doesn't succeeds and server complains I am not logged in. But if I use the cookie value from my browser the request succeeds.

The the dummy request to programmatically acquire JSESSIONID,glide_user and glide_user_session params in cookie is

response = requests.get('http://example.com/make_dummy_get',auth=('username','pasword'))
cookie_params = response.cookies.items()

below is the actual request

headers =  {
'Host': 'example.com'
,'Connection': 'keep-alive'
,'Content-Length': 113
,'Cache-Control': 'max-age=0'
,'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
,'Origin': 'example.com'
,'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36'
,'Content-Type': 'application/x-www-form-urlencoded'
,'Referer': 'www.example.com/asdas/'
,'Accept-Encoding': 'gzip,deflate,sdch'
,'Accept-Language': 'en-US,en;q=0.8'
,'Cookie': 'JSESSIONID=B6F7371A11825472CAB0366A4DCDD8EFB; glide_user="SC:Z3Vlc3Q=:b890b38b7f000001121dbe81a08c413ca5"; glide_user_session="SC:Z3Vlc3Q=:b890b38b7f000001121dbe81a08c413ca5"'
}

form_data = {
'param1': 'value1'
,'param2': 'value2'
,'param3': 'value3'
}

res = requests.post('http://example.com/make_post_request',auth=('username','pasword'),data=form_data,headers = headers)

It seems to me that the session created by my dummy request for some reason is getting closed and Hence the second request is rejected and html response says I must login to access the requested resource.

I did the same exercise with Java apache's HttpClient and ended with the same issue.What am I missing here to make the request succeed without any login or authentication issues?

Solution

First you should be using a Session object from requests. This will manage cookies (and prepare them for you) so you do not have to create the cookie header for yourself.

s = requests.Session()
s.get('http://example.com/make_dummy_get',auth=('username','pasword'))
print(s.cookies)

Next I have to strongly advise you to stop setting the following headers:

• Host
• Content-Length
• Content-Type
• Cookie

All four of those headers will be generated by requests for you. The Cookie header will be generated using the CookieJar that the Session uses. The Content-Length and Content-Type will be computed while requests prepares the body.

Also, if you're trying to use cookies to authenticate, the server is likely becoming confused because you're also passing auth=('username', 'password') in your second request. That's generating an authorization header so you're both sending a Cookie header and an Authorization header. The server sees this as suspicious most likely and rightly refuses to accept your request as authenicated.

Comments

[rogue-one]

Session resolved cokkie issue but for some reason the auth param was not working as intended. I saw what the login page is doing and replicated that request instead of using auth and made the subsequent post request which succeeded..

[user136036]

Note: You cannot see the Host header in requests session headers. This is because requests does not add the header. The underlying http module will do that when sending the request: stackoverflow.com/q/57770557/2441026