How to find header data and name? (Python-requests)

Question

I want to use requests to web scrape on a login site. I already done the code using selenium but it is very inconvenient and slower to do it that way as I want to make it public(every user has to download chrome driver).

The problem is, there are multiple requests from the site and I don't have any experience processing that data and extracting the header data and name. Any help is great, thanks.

Solution

[Premise]

Using requests module you can send requests in these way:

import requests

url = "http://www.example.com"  # request url

headers = {  # headers dict to send in request
  "header_name": "headers_value",
 }

params = {  # params to be encoded in the url
  "param_name": "param_value",
}

data = {  # data to send in the request body
  "data_name": "data_value",
}

# Send GET request.
requests.get(url, params=params, headers=headers)

# Send POST request.
requests.post(url, params=params, headers=headers, data=data)

Once you perform a request, you can get much information from the response object:

>>> import requests

# We perform a request and get the response object.
>>> response = requests.get(url, params=params, headers=headers)
>>> response = requests.post(url, params=params, headers=headers, data=data)

>>> response.status_code  # server response status code
>>> 200  # eg.

>>> response.request.method
>>> 'GET'  # or eventually 'POST'

>>> response.request.headers # headers you sent with the request
>>> {'Accept-Encoding': 'gzip, deflate, br'}  # eg.

>>> response.request.url  # sent request url
>>> 'http://www.example.com'

>>> response.response.body
>>> 'name=value&name2=value2' # eg.

In conclusion, you can retrieve all the information that you can find in Dev Tools in the browser, from the response object. You need nothing else. Dev Tools view Dev Tool view 2

Once you send a GET or POST requests you can retrieve information from Dev Tools:

In General:

Request URL: the url you sent the request to. Corresponds to response.request.url Request Method: corresponds to response.request.method Status Code: corresponds to response.status_code

In Response Headers:

You find response headers which correspond to response.headers

eg. Connection: Keep-Alive, Content-Length: 0, Content-Type: text/html; charset=UTF-8...

In Requests Headers:

You find request headers which correspond to response.request.headers

In Form Data:

You can find the data you passed with data keyword in requests.post. Corresponds to response.request.body

Comments

[Federico Rubbi]

Excuse me, do you mean response headers?

[Tudor Popescu]

In the networks tab(browser), all data is recorded there when you go to a new tab, including the headers.

[Tudor Popescu]

The problem is that I am prohibited to enter the website without any headers using requests in python as i have to log in. And I don't know how to process the data and names are of them in the information provided by the chrome dev tool

[Tudor Popescu]

Hi, I don't really understand where to get the headers from. I kind of understand it but i cant find any of the data. The website login i am trying to access is parents.netherhall.org.

[Tudor Popescu]

I logged in successfully, thanks for your patience as well, initially I thought there was only one name and data string. But now I don't know how to go to the specific URL. Github link to my test code I want to go to the request URL

[Federico Rubbi]

To send a request to a url, simply use session.get(Request_URL) or session.post(Request_URL).

[Tudor Popescu]

When I did that, it blocks me, do I have to do something similar as before?

[Federico Rubbi]

I'll give a look right now. But you should hide your credentials!

[Tudor Popescu]

I forgot about that :/ I censored it from the cookie but not from the variable as I was testing it. oops

[Federico Rubbi]

I'm trying to find a solution. Since I shouldn't edit my answer because I'd get off-topic, I suggest you to post another question. In this way, perhaps somebody and I could help you.