Putting a `Cookie` in a `CookieJar`

Question

I'm using the Python Requests library to make HTTP requests. I obtain a cookie from the server as text. How do I turn that into a CookieJar with the cookie in it?

Solution 1

Old versions of the Requests library (0.14.2 and older) put new cookies in the jar for you when you pass a CookieJar object:

import requests
import cookielib

URL = '...whatever...'
jar = cookielib.CookieJar()
r = requests.get(URL, cookies=jar)
r = requests.get(URL, cookies=jar)

The first request to the URL fills the jar and the second request sends the cookies back to the server.

This doesn't work starting with Requests 1.0.0, released in 2012.

Solution 2

A Requests Session will receive and send cookies.

s = requests.Session()

s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get("http://httpbin.org/cookies")

print(r.text)
# '{"cookies": {"sessioncookie": "123456789"}}'

(The code above is stolen from Session Objects.)

If you want cookies to persist on disk between runs of your code, you can directly use a CookieJar and save/load them:

from http.cookiejar import LWPCookieJar
import requests

cookie_file = '/tmp/cookies'
jar = LWPCookieJar(cookie_file)

# Load existing cookies (file might not yet exist)
try:
    jar.load()
except:
    pass

s = requests.Session()
s.cookies = jar

s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get("http://httpbin.org/cookies")

# Save cookies to disk, even session cookies
jar.save(ignore_discard=True)

Then look in the file /tmp/cookies:

#LWP-Cookies-2.0
Set-Cookie3: sessioncookie=123456789; path="/"; domain="httpbin.org"; path_spec; discard; version=0

Solution 3

I think many of these answers are missing the point. Sometimes that other library isn't using Requests under the hood. Or it doesn't expose the cookiejar it's using. Sometimes all we have is the cookie string. In my case I'm trying to borrow the auth cookie from pyVmomi.

import requests
import http.cookies

raw_cookie_line = 'foo="a secret value"; Path=/; HttpOnly; Secure; '
simple_cookie = http.cookies.SimpleCookie(raw_cookie_line)
cookie_jar = requests.cookies.RequestsCookieJar()
cookie_jar.update(simple_cookie)

Which gives us the following cookie_jar:

In [5]: cookie_jar
Out[5]: <RequestsCookieJar[Cookie(version=0, name='foo', value='a secret value', port=None, port_specified=False, domain='', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=True, expires=None, discard=False, comment='', comment_url=False, rest={'HttpOnly': True}, rfc2109=False)]>

Which we can use as normal:

requests.get(..., cookies=cookie_jar)

Solution 4

To help you out, I wrote an entire module. I tried it with my personal webpage and Google's cookies, so I'd assume it works.

I got help from How can I add a cookie to an existing cookielib CookieJar instance in Python?.

I have a lot of unpythonic code in here, including a semi-kludge, so your mileage may vary. Tweak it as you wish, especially with the assumed items (such as port 80). The "request" as an argument below is of type requests.request and I realized that the "method" argument must be all capitals.

Note: I haven't had time to add comments for clarification, so you'll have to use the source.

import Cookie,cookielib,requests,datetime,time  # I had this out, but I realized later I needed it when I continued testing

def time_to_tuple(time_string):
    wday = {'Mon':0,'Tue':1,'Wed':2,'Thu':3,'Fri':4,'Sat':5,'Sun':6}
    mon = {'Jan':1,'Feb':2,'Mar':3,'Apr':4,'May':5,'Jun':6,'Jul':7,'Aug':8,'Sep':9,'Oct':10,'Nov':11,'Dec':12}
    info = time_string.split(' ')
    info = [i.strip() for i in info if type(i)==str]
    month = None
    for i in info:
        if '-' in i:
            tmp = i.split('-')
            for m in tmp:
                try:
                    tmp2 = int(m)
                    if tmp2<31:
                        mday = tmp2
                    elif tmp2 > 2000:
                        year = tmp2
                except:
                    for key in mon:
                        if m.lower() in key.lower():
                            month = mon[key]
        elif ':' in i:
            tmp = i.split(':')
            if len(tmp)==2:
                hour = int(tmp[0])
                minute = int(tmp[1])
            if len(tmp)==3:
                hour = int(tmp[0])
                minute = int(tmp[1])
                second = int(tmp[2])
        else:
            for item in wday:
                if ((i.lower() in item.lower()) or (item.lower() in i.lower())):
                    day = wday[item]
            if month is None:
                for item in mon:
                    if ((i.lower() in item.lower()) or (item.lower() in i.lower())):
                        month = mon[item]
    return year,month,mday,hour,minute,second

def timefrom(year,month,mday,hour,minute,second):
    time_now = time.gmtime()
    datetime_now = datetime.datetime(time_now.tm_year,time_now.tm_mon,
                                     time_now.tm_mday,time_now.tm_hour,
                                     time_now.tm_min,time_now.tm_sec)
    then = datetime.datetime(year,month,mday,hour,minute,second)
    return (datetime_now-then).total_seconds()

def timeto(year,month,mday,hour,minute,second):
    return -1*timefrom(year,month,mday,hour,minute,second)



##['comment', 'domain', 'secure', 'expires', 'max-age', 'version', 'path', 'httponly']
def parse_request(request):
    headers = request.headers
    cookieinfo = headers['set-cookie'].split(';')
    name = 'Undefined'
    port=80
    port_specified=True
    c = Cookie.SmartCookie(headers['set-cookie'])
    cj = cookielib.CookieJar()
    for m in c.values():
        value = m.coded_value
        domain = m['domain']
        expires = m['expires']
        if type(expires) == str:
            tmp = time_to_tuple(expires)
            expires = timeto(tmp[0],tmp[1],tmp[2],tmp[3],tmp[4],tmp[5])
        max_age=m['max-age']
        version = m['version']
        if version == '':
            version = 0
        path = m['path']
        httponly = m['httponly']
        if httponly == '':
            if 'httponly' in headers['set-cookie'].lower():
                httponly = True
        else:
            httponly = False
        secure = m['secure']
        comment=m['comment']
        port = 80
        port_specified=False
        domain_specified=True
        domain_initial_dot = domain.startswith('.')
        path_specified=True
        discard = True
        comment_url=None
        rest={'HttpOnly':httponly}
        rfc2109=False
        ck = cookielib.Cookie(version,name,value,port,port_specified,domain,
                              domain_specified,domain_initial_dot,path,path_specified,
                              secure,expires,discard,comment,comment_url,rest,rfc2109)
        cj.set_cookie(ck)
    return cj

Solution 5

Well, cookielib.LWPCookieJar has load and save methods on it. Look at the format and see if it matches the native cookie format. You may well be able to load your cookie straight into a cookie jar using StringIO.

Alternatively, if Requests is using urllib2 under the hood, you could add a cookie handler to the default opener.

View more solutions

Comments

[Matt Joiner]

I see your Python mailing list message, and I see here the reason for your desperation.

[panzi]

I don't see it. What's so bad/convoluted about the top answer?

[Ram Rachum]

The top answer is great, I was made desperate by the answers that came before it.

[Martijn Pieters]

With requests.Session() there is no need to worry about cookie jars. The session object manages receiving and sending cookies for you.

[Marcus Junius Brutus]

See this answer for a way to accomplish that without using the Session object.

[Ram Rachum]

Really overly complex for such a simple task. I guess the only thing I can do is wait for requests to implement cookie-handling.

[Snakes and Coffee]

I'm a web programmer, but I'm mostly involved with sending cookies. I'll revise when I get home since company internet disallows programmatic web-diving.

[Snakes and Coffee]

I have name="undefined" due to the fact that I haven't been able to find out where the name is. If someone could point out where, I would be happy to update the code.

[Snakes and Coffee]

If you have too many cookies (which I doubt, given the function of cookies), feel free to use yield instead of return.

[Kenneth Reitz]

A new release was just pushed (v0.6.0) that allows you to attach cookies to a request with a simple dictionary. docs.python-requests.org/en/latest/user/quickstart/#cookies

[Kenneth Reitz]

A new release was just pushed (v0.6.0) that allows you to attach cookies to a request with a simple dictionary. docs.python-requests.org/en/latest/user/quickstart/#cookies

[Kenneth Reitz]

A new release was just pushed (v0.6.0) that allows you to attach cookies to a request with a simple dictionary. docs.python-requests.org/en/latest/user/quickstart/#cookies

[Ken Cochrane]

@Kenneth Reitz Awesome, Thank you.

[tommy_o]

This wasn't working for me, so I asked a similar question to clarify.

[Martijn Pieters]

Note that this is not needed if you are using a requests.Session object; it'll handle the cookie jar for you, entirely.

[Doormatt]

Just wanted to point out that "import cookielib" is only valid under 2.X - in 3.X, it's "import http.cookiejar".

[MestreLion]

Isn't all this code equivalent to the existing cookielib.CookieJar.extract_cookies(response, request) ???

[djsumdog]

I really wish this was voted to the top. The top answer selected does not work at all with requests-2.3.0

[djsumdog]

I couldn't get this code working. Even passing a cookie jar to each request didn't persist my cookies, but the requests.Session listed here in the comments and further down worked perfectly.

[Marcus Junius Brutus]

This did not work for me with the latest release (v2.18.4). This worked instead: stackoverflow.com/a/47913559/274677 (without using Session). But I guess Session is the way to go.

[Jing He]

This doesn't work for me either, with Python 2.7 and Requests 2.21.0. I have to use jar.set_cookie to put every cookie into the Jar or it gets nothing from the request.get method.

[Michael]

Thank you for the ignore_discard=True hint! I couldn't figure out why my jar saved an empty file.

[Boris Verkhovskiy]

dstanek's answer stopped working in 2012, with Requests 1.0.0

[Boris Verkhovskiy]

This answer stopped working with Requests 1.0.0 which was released in 2012. I have confirmed that it works with Python 2.7.13 and Requests 0.14.2