How to save requests (python) cookies to a file?
Solution 1
There is no immediate way to do so, but it's not hard to do.
You can get a CookieJar
object from the session with session.cookies
, and use pickle
to store it to a file.
A full example:
import requests, pickle
session = requests.session()
# Make some calls
with open('somefile', 'wb') as f:
pickle.dump(session.cookies, f)
Loading is then:
session = requests.session() # or an existing session
with open('somefile', 'rb') as f:
session.cookies.update(pickle.load(f))
The requests
library uses the requests.cookies.RequestsCookieJar()
subclass, which explicitly supports pickling and a dict-like API. The RequestsCookieJar.update()
method can be used to update an existing session cookie jar with the cookies loaded from the pickle file.
Solution 2
After a call such as r = requests.get()
, r.cookies
will return a RequestsCookieJar
which you can directly pickle
, i.e.
import pickle
def save_cookies(requests_cookiejar, filename):
with open(filename, 'wb') as f:
pickle.dump(requests_cookiejar, f)
def load_cookies(filename):
with open(filename, 'rb') as f:
return pickle.load(f)
#save cookies
r = requests.get(url)
save_cookies(r.cookies, filename)
#load cookies and do a request
requests.get(url, cookies=load_cookies(filename))
If you want to save your cookies in human-readable format, you have to do some work to extract the RequestsCookieJar
to a LWPCookieJar
.
import cookielib
def save_cookies_lwp(cookiejar, filename):
lwp_cookiejar = cookielib.LWPCookieJar()
for c in cookiejar:
args = dict(vars(c).items())
args['rest'] = args['_rest']
del args['_rest']
c = cookielib.Cookie(**args)
lwp_cookiejar.set_cookie(c)
lwp_cookiejar.save(filename, ignore_discard=True)
def load_cookies_from_lwp(filename):
lwp_cookiejar = cookielib.LWPCookieJar()
lwp_cookiejar.load(filename, ignore_discard=True)
return lwp_cookiejar
#save human-readable
r = requests.get(url)
save_cookies_lwp(r.cookies, filename)
#you can pass a LWPCookieJar directly to requests
requests.get(url, cookies=load_cookies_from_lwp(filename))
Solution 3
I offer a way by json:
to save cookie -
import json
with open('cookie.txt', 'w') as f:
json.dump(requests.utils.dict_from_cookiejar(bot.cookies), f)
and to load cookie -
import json
session = requests.session() # or an existing session
with open('cookie.txt', 'r') as f:
cookies = requests.utils.cookiejar_from_dict(json.load(f))
session.cookies.update(cookies)
Solution 4
Expanding on @miracle2k's answer, requests Session
s are documented to work with any cookielib CookieJar
. The LWPCookieJar
(and MozillaCookieJar
) can save and load their cookies to and from a file. Here is a complete code snippet which will save and load cookies for a requests session. The ignore_discard
parameter is used to work with httpbin for the test, but you may not want to include it your in real code.
import os
from cookielib import LWPCookieJar
import requests
s = requests.Session()
s.cookies = LWPCookieJar('cookiejar')
if not os.path.exists('cookiejar'):
# Create a new cookies file and set our Session's cookies
print('setting cookies')
s.cookies.save()
r = s.get('http://httpbin.org/cookies/set?k1=v1&k2=v2')
else:
# Load saved cookies from the file and use them in a request
print('loading saved cookies')
s.cookies.load(ignore_discard=True)
r = s.get('http://httpbin.org/cookies')
print(r.text)
# Save the session's cookies back to the file
s.cookies.save(ignore_discard=True)
Solution 5
This will do the job:
session.cookies = LWPCookieJar('cookies.txt')
The CookieJar API requires you to call load() and save() manually though. If you do not care about the cookies.txt format, I have a ShelvedCookieJar implementation that will persist on change.
agrynchuk
By education I am a pharmacist. Programming in Python is my hobby (I started to learn perl).
Updated on September 17, 2021Comments
-
agrynchuk over 2 years
How to use the library
requests
(in python) after a request#!/usr/bin/env python # -*- coding: utf-8 -*- import requests bot = requests.session() bot.get('http://google.com')
to keep all the cookies in a file and then restore the cookies from a file.
-
Elmo over 9 years
requests.utils.dict_from_cookiejar
andrequests.utils.cookiejar_from_dict
are not required. They don't save cookies with the same name for different domains and don't save all the required cookies data. I spent a lot of time debugging just because of these. -
ChaimG almost 9 yearsThis answer is missing a few steps. Here is the full code:
cj = cookielib.LWPCookieJar(cookie_file)
cj.load()
session.cookies = cj
-
MattCochrane over 8 yearspickling a cookiejar doesn't seem to save the host that its associated with. when you load again all cookies are just in the host ''.
-
MarSoft over 7 years@MattClimbs, it is
dict_from_cookiejar
which doesn't save host information. Actually, in current versionsession.cookies
can be pickled and unpickled directly, without converting todict
. Also,requests.utils.dict_from_cookiejar
can be replaced withsession.cookies.get_dict()
, andcookiejar_from_dict
can be replaced withsession.cookies.update(my_dict)
. -
consatan about 7 years
all(isinstance(c, cookielib.Cookie) for c in r.cookies)
isTrue
, so why created each cookie item again insave_cookies_lwp
? -
Javier C. H. over 6 yearsusing
session.cookies.set(cookies.keys()[n],cookies.values()[n])
worked for me as well, thanks! -
User over 5 yearsPlease use a session in your answer
-
Rontron over 5 yearsThis worked for me without the os.path.isdir lines in each function. Calling open() creates the file if it doesn't exist, and writes correctly for my use-case.
-
Martijn Pieters over 5 yearsEditors note: I've updated this top answer rather than add a new post. It was close enough but needed updating for API changes made in the intervening 6 years that make this task all that much easier. Do not use the dictionary utilities, they are not needed at all and do not preserve important cookie medatada. I'm fine with posting my own answer if the author wishes to revert my changes; I'd appreciate a heads-up in that case.
-
Lennart Rolland almost 5 yearsrequests.utils.dict_from_cookiejar <-- exactly what I wanted
-
MarSoft over 4 yearsNote that
dict_from_cookiejar
loses host information and hence does not work if there are several cookies with same name for different hosts. -
Emil over 4 yearsThis should have been best answer - no external libs which are not included by default.
-
Jurass almost 4 yearsthis is not correct, dict_from_cookiejar loses many informations about cookies.
-
Weekend over 3 yearsIn my case (a python newbie), after
open() as f
, there should be af.close()
. Otherwise two consecutiveopen()
would rise "TypeError: can't pickle _thread.RLock objects". -
user136036 over 3 yearsThis basically works fine, but you cannot add/update cookies like you can with
requests.update()
. -
GordonAitchJay about 3 years@Weekend, you don't need to explicitly call
f.close()
when using thewith
statement. The context manager implicitly callsf.close()
for you. See docs.python.org/3/tutorial/… The 2 separatewith
statements in the answer are not nested.