Passing a variable in url?

Question

So I'm new in python and I desperately need help.

I have a file which has a bunch of ids (integer values) written in 'em. Its a text file.

Now I need to pass each id inside the file into a url.

For example "https://example.com/[id]"

It will be done in this way

A = json.load(urllib.urlopen("https://example.com/(the first id present in the text file)"))
print A

What this will essentially do is that it will read certain information about the id present in the above url and display it. I want this to work in a loop format where in it will read all the ids inside the text file and pass it to the url mentioned in 'A' and display the values continuously..is there a way to do this?

I'd be very grateful if someone could help me out!

Solution 1

Old style string concatenation can be used

>>> id = "3333333"
>>> url = "https://example.com/%s" % id
>>> print url
https://example.com/3333333
>>> 

The new style string formatting:

>>> url = "https://example.com/{0}".format(id)
>>> print url
https://example.com/3333333
>>> 

The reading for file as mentioned by avasal with a small change:

f = open('file.txt', 'r')
for line in f.readlines():
    id = line.strip('\n')
    url = "https://example.com/{0}".format(id)
    urlobj = urllib.urlopen(url)
    try:
        json_data = json.loads(urlobj)
        print json_data
    except:
        print urlobj.readlines()

Solution 2

lazy style:

url = "https://example.com/" + first_id

A = json.load(urllib.urlopen(url))
print A

old style:

url = "https://example.com/%s" % first_id

A = json.load(urllib.urlopen(url))
print A

new style 2.6+:

url = "https://example.com/{0}".format( first_id )

A = json.load(urllib.urlopen(url))
print A

new style 2.7+:

url = "https://example.com/{}".format( first_id )

A = json.load(urllib.urlopen(url))
print A

Solution 3

The first thing you need to do is know how to read each line from a file. First, you have to open the file; you can do this with a with statement:

with open('my-file-name.txt') as intfile:

This opens a file and stores a reference to that file in intfile, and it will automatically close the file at the end of your with block. You then need to read each line from the file; you can do that with a regular old for loop:

  for line in intfile:

This will loop through each line in the file, reading them one at a time. In your loop, you can access each line as line. All that's left is to make the request to your website using the code you gave. The one bit your missing is what's called "string interpolation", which allows you to format a string with other strings, numbers, or anything else. In your case, you'd like to put a string (the line from your file) inside another string (the URL). To do that, you use the %s flag along with the string interpolation operator, %:

url = 'http://example.com/?id=%s' % line
A = json.load(urllib.urlopen(url))
print A

Putting it all together, you get:

with open('my-file-name.txt') as intfile:
  for line in intfile:
    url = 'http://example.com/?id=%s' % line
    A = json.load(urllib.urlopen(url))
    print A

Solution 4

Python 3+

New String formatting is supported in Python 3 which is a more readable and better way to format a string. Here's the good article to read about the same: Python 3's f-Strings

In this case, it can be formatted as

url = f"https://example.com/{id}"

Detailed example

When you want to pass multiple params to the URL it can be done as below.

name = "test_api_4"
owner = "[email protected]"

url = f"http://localhost:5001/files/create" \
f"?name={name}" \
f"&owner={owner}" \

We are using multiple f-string here and they can be appended by ''. This will keep them in the same line without inserting any new line character between them.

For values which have space

For such values you should import from urllib.parse import quote in your python file and then quote the string like: quote("firstname lastname")

This will replace space character with %20.

View more solutions

Comments

[user1452759]

Thank you very much for helping me out. I'm facing a new problem now. Suppose I try to read a text file with say, just a handful of ids, it has no problem in executing. But my text file from where I'm reading the ids has a LOT of them. And the code fails to execute. What is the reason for this? Is there a way to get around this?

[user1452759]

Thank you very much for helping me out. I'm facing a new problem now. Suppose I try to read a text file with say, just a handful of ids, it has no problem in executing. But my text file from where I'm reading the ids has a LOT of them. And the code fails to execute. What is the reason for this? Is there a way to get around this?

[pyfunc]

@user1452759: What is the error that you are getting? It would be nice to start from there. The solution should work for even if you have many ids. Chances are that not all the url fetch is returning data that can be loaded into json

[pyfunc]

@user1452759: Try the solution as I have mentioned in edited example. It will check if data can be loaded as json or else will print the output of what was fetched.

[Haldean Brown]

Could you post the error message? It's hard to say without knowing what it says the problem is.

[Kevin Lemaire]

And any clue for value with / in it? Like dataset ids in Freenas?