python loop to pull API data for iterating URLs

Question

I'm trying to set up a for loop to pull in elected representatives' data for about 600,000 postal codes. The base URL stays the same, and the only part that changes is the postal code.

I'd ideally like to create a list of all the postal codes, and then use requests.get to pull in data for all the postal codes in my list. I came up with this code below, but it's only pulling in data for the last postal code in my list. I'm not really sure why this is happening and am a python beginner - so any help would be appreciated!

---

#loop test 
postcodes = ['P0L1B0','P5A3P1', 'P5A3P2', 'P5A3P3']

for i in range(len(postcodes)):
   rr = requests.get('https://represent.opennorth.ca/postcodes/{}'.format(postcodes[i]))

data1=json.loads(rr.text)

data1

Solution 1

Your code doesnt works because it overwrites data1.

Try this:

#loop test 
responses = list() # stores responses for postal codes
postcodes = ['P0L1B0','P5A3P1', 'P5A3P2', 'P5A3P3']

for postcode in postcodes:
   rr = requests.get('https://represent.opennorth.ca/postcodes/{}'.format(postcode))
   data=json.loads(rr.text)
   responses.append(data)

Your responses now are saved in responses list.

Tips:
You can iterate over list without using index.

Solution 2

you are just viewing the last response.

#loop test 
postcodes = ['P0L1B0','P5A3P1', 'P5A3P2', 'P5A3P3']

api_data = dict()

for i in postcodes:
   rr = requests.get('https://represent.opennorth.ca/postcodes/{}'.format(i))

   data = json.loads(rr.text)
   api_data.update({i: data})   
   # or print(data)

print(api_data)

here I've added all the response to a dict, with key as postal code and value as the response.

Solution 3

You are overwriting data1 variable every iteration, that is why you end up with only the last one, you need to store it differently.

Example:

postcodes =['P0L1B0','P5A3P1', 'P5A3P2', 'P5A3P3']
results = []

for postcode in postcodes:
    res = requests.get('https://represent.opennorth.ca/postcodes/{}'.format(postcode))

    if res.status_code == 200:
        results.append(res.json())
    else: 
        print("Request to {} failed".format(postcode))

Comments

[cengineer]

Because in each iteration you are overriding the value of data1. You need to save the value of data1 in somewhere (for example in lists).

[M.Ai]

THANK YOU!!! Of course, duh I didn't need an index. Can't believe I forgot to append results.

[M.Ai]

Thanks a ton! Just needs to be api_data.update(), I think?

[P.hunter]

api_data.update() means that you are putting a key-value pair to api_data dictionary, that's a basic operation of dictionaries, run my code for better understanding

[Josef Korbel]

This answer does not contains status code check, which can bring you nice headache if it will fail after a while and none of your responses are being stored. If you will try to json.loads something that is not json it will fail on JSONDecodeError. Also if this dataset is HUGE, dont use lists, but store it in file locally, per postcode and then merge it.

[Josef Korbel]

He was refering to a typo, this code wont run since you are using api_data and then api_dict