Posting raw data with Python

Question

I'm playing around with the Google Checkout API and I want to pull it into a Django app. I need to post data to Google using basic http authentication. I've been testing this with curl like this:

curl -d "$(cat mytest.xml)" -u username:password https://url

And that posts the content of my test XML file to Google. And it works fine!

But I'm having problems porting that simple line to Python. I've managed several different ways (httplib2, urllib2, pycurl) of connecting with a password and posting something but the respose is always 400 BAD REQUEST.

Is there a python equivalent for posting block of text to a HTTP Basic auth server? I'm running out of walls to bang my head against.

---

Apologies for not adding any code. Here are some of my greatest hits. In each, DATA is an XML string. URL, USERNAME and PASSWORD are constant.

req = urllib2.Request(URL)
req.add_header("Authorization", "Basic %s" % base64.encodestring('%s:%s'%(USERNAME, PASSWORD)))
u = urllib2.urlopen(req, DATA)

Gives me a lovely HTTP Error 400: Bad Request

---

passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, URL, USERNAME, PASSWORD)
authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)
pagehandle = urllib2.urlopen(URL, DATA)

Gives HTTP Error 401: Unauthorized

---

pycurl.global_init(pycurl.GLOBAL_DEFAULT)
c = pycurl.Curl()
c.setopt(pycurl.URL, URL)
c.setopt(pycurl.USERPWD, "%s:%s" % (USERNAME,PASSWORD))
c.setopt(pycurl.POST, 1)
c.setopt(pycurl.HTTPHEADER, ["Content-type: text/xml"])
c.setopt(pycurl.POSTFIELDS, DATA)
b = StringIO.StringIO()
c.setopt(pycurl.WRITEFUNCTION, b.write)
c.perform()

Seems to struggle with passing the DATA string as a POSTFIELD. I've tried urllib.urlencode()ing DATA in several different ways but

---

h = httplib2.Http()
h.add_credentials(USERNAME, PASSWORD)
print = h.request(URL, "POST", body=base64.encodestring(DATA))

The credentials don't seem to do anything - I get an unauthorised message back from Google.

There are more but they're all based on these.

Solution 1

I've been having similar turmoils with the stdlib packages, until somoneone pointed to the awesome requests that supports basic Http Authentication and other authentication means straight out-of-the-box! And it has a beautiful and simple API it hurts!

requests.post(url, data=DATA, headers=HEADERS_DICT, auth=(username, password))

It supports a host of other necessary features (e.g HTTPS, Digest Authentication, etc) Please check it out if u must...

Solution 2

Voidspace has an excellent article on using basic auth with urllib2. I've copied the appropriate code snippet below, changed to use POST.

import urllib2

theurl = 'http://www.someserver.com/toplevelurl/somepage.htm'
username = 'johnny'
password = 'XXXXXX'

passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, theurl, username, password)

authhandler = urllib2.HTTPBasicAuthHandler(passman)

opener = urllib2.build_opener(authhandler)

urllib2.install_opener(opener)

pagehandle = urllib2.urlopen(theurl, open("mytext.xml").read())

Without seeing your code it's hard to say why you would be getting a 400 response.

Solution 3

While editing my post to include some source, I thought I'd have another crack at httplib2 (mainly because it's comparatively small and pretty compared to the others) and noticed that there's a gaping bug in that its add_credentials(..) method doesn't actually do anything. You can work around this by specifying the header (as I did with urllib2) like this:

resp, content = httplib2.Http().request(URL, "POST", body=DATA, headers={
    "Authorization": "Basic %s" % base64.encodestring('%s:%s' %(USERNAME, PASSWORD)
})

And this works.

Comments

[Oli]

I agree! Very hard to know what I've tried if I don't tell you. My apologies. I've the main groups of methods now (I've probably got about 5 permutations for each "method"). With this method it wasn't authorising. It didn't look like the Password Manager was doing anything. But I've fixed httplib2 now (as posted) so hopefully won't have to use another method for this.