Using argparse with function that takes **kwargs argument

Question

I'm using argparse to take input and pass it to a function that takes as arguments two variables and **kwargs.

Here's my function:

import requests
import sys
import argparse


def location_by_coordinate(LAT, LNG, **kwargs):
    if not kwargs:
        coordinate_url = "https://api.instagram.com/v1/locations/search?lat=%s&lng=%s&access_token=%s" % (LAT, LNG, current_token)
        r = requests.get(coordinate_url).text
    else:
        coordinate_url = "https://api.instagram.com/v1/locations/search?lat=%s&lng=%s&access_token=%s" % (LAT, LNG, current_token)
        for key, value in kwargs.iteritems():
            if 'DISTANCE' in kwargs:
                distance = kwargs.get('DISTANCE')
                if distance > 5000:
                    print distance
                    print "max distance is 5000m, value is reassigned to default of 1000m"
                    distance = 1000
                    coordinate_url = "https://api.instagram.com/v1/locations/search?lat=%s&lng=%s&access_token=%s" % (LAT, LNG, current_token)
                    r = requests.get(coordinate_url).text
                else:
                    pass
                    coordinate_url = "https://api.instagram.com/v1/locations/search?lat=%s&lng=%s&access_token=%s" % (LAT, LNG, current_token)
                    r = requests.get(coordinate_url).text
            if 'FACEBOOK_PLACES_ID' in kwargs:
                fb_places_id = kwargs.get('FACEBOOK_PLACES_ID')
                payload = {'FACEBOOK_PLACES_ID': '%s' % (fb_places_id), 'DISTANCE': '%s' % (DISTANCE)}
                r = requests.get(coordinate_url, params=payload).text
            if 'FOURSQUARE_ID' in kwargs:
                foursquare_id = kwargs.get('FOURSQUARE_ID')
                payload = {'FOURSQUARE_ID': '%s' % (foursquare_id), 'DISTANCE': '%s' % (DISTANCE)}
                r = requests.get(coordinate_url, params=payload).text
            if 'FOURSQUARE_V2_ID' in kwargs:
                foursquare_v2_id = kwargs.get('FOURSQUARE_V2_ID')
                payload = {'FOURSQUARE_V2_ID': '%s' % (foursquare_v2_id), 'DISTANCE': '%s' % (DISTANCE)}
                r = requests.get(coordinate_url, params=payload).text
    #print r
    return r

Given this function and its use of **kwargs, how should I setup the subparsers?

Here's how I've setup the command line parser thus far:

 def main():
        parser = argparse.ArgumentParser(description="API Endpoints tester")
        subparsers = parser.add_subparsers(dest="command", help="Available commands")

        location_by_parser = subparsers.add_parser("location_by_coordinate", help="location function")
        location_by_parser.add_argument("LAT", help="latitude")
        location_by_parser.add_argument("LNG", help="longitude")

        arguments = parser.parse_args(sys.argv[1:])
        arguments = vars(arguments)
        command = arguments.pop("command")
        if command == "location_by_coordinate":
            LAT, LNG = location_by_coordinate(**arguments)
        else:
            print "No command provided..."

    if __name__ == "__main__":
        main()

Obviously, the above main() function works fine with the location_by_coordinate() function when I call it at the command line like this:

$ python argstest.py location_by_coordinate 40.5949799 -73.9495148

But with the code the way it is currently, if I try:

$ python argstest.py location_by_coordinate 40.5949799 -73.9495148 DISTANCE=3000

Obviously, I get:

argstest.py: error: unrecognized arguments: DISTANCE=3000

But I'm not sure how to setup a subparser for **kwargs. If I try to setup a subparser like this:

location_by_parser.add_argument("**kwargs", help="**kwargs")

and then try that command again:

$ python argstest.py location_by_coordinate 40.5949799 -73.9495148 DISTANCE=3000

That doesn't work because the arguments object (which is a dictionary), becomes this:

{'LAT': '40.5949799', 'LNG': '-73.9495148', 'command': 'location_by_coordinate', '**kwargs': 'DISTANCE=3000'}

And this Traceback is returned:

Traceback (most recent call last):
  File "argstest.py", line 118, in <module>
    main()
  File "argstest.py", line 108, in main
    foo = location_by_coordinate(**arguments)
  File "argstest.py", line 40, in location_by_coordinate
    return r
UnboundLocalError: local variable 'r' referenced before assignment

How can I enable argparse to handle/to parse what is entered at the command line that is intended to be passed to the function via **kwargs?

Solution 1

Do you understand what is going on with the

{'LAT': '40.5949799', 'LNG': '-73.9495148', 'command': 'location_by_coordinate', '**kwargs': 'DISTANCE=3000'}

arguments dictionary? You defined a 'positional' argument with the name ('dest') of '**kwargs'. You could just as well named it 'foobar'. The parser assigned the string 'DISTANCE=3000' to that attribute in the args namespace, which turned into a dictionary key:value pair in arguments.

You could, of course, look for arguments['**kwargs'], and parse the value for yourself:

v = arguments['**kwargs']  # or pop if you prefer
if v is not None:
    k, v = v.split('=')
    arguments[k] = int(v)

It could be generalized to handle multiple pairs (defined with `nargs='*').

---

argparse does not handle arguments the same way as Python functions, so there's nothing exactly analogous the **kwargs.

The normal way to accept something like distance is with 'optionals' or flagged arguments.

parser.add_argument('-d','--distance', type=int, help=...)

which will accept

python argstest.py location_by_coordinate 40.5949799 -73.9495148 --distance=3000
python argstest.py location_by_coordinate 40.5949799 -73.9495148 --distance 3000
python argstest.py location_by_coordinate 40.5949799 -73.9495148 --d3000
python argstest.py location_by_coordinate 40.5949799 -73.9495148

It could also be setup to use --DISTANCE or other names. In the last case args namespace will have a default value for distance. The default default is None.

That's the straight forward way of adding kwarg like arguments to argparse.

Accepting arbitrary dictionary like pairs, distance:3000, distance=3000, has been asked before on SO. The answers have always been some variation of the parsing that I sketched above. It could be done in a custom Action class, or post parsing as I suggest.

oops, this answer is nearly a clone of one I wrote a few days ago: https://stackoverflow.com/a/33639147/901925

A similar 2011 question: Using argparse to parse arguments of form "arg= val"

Python argparse dict arg

=================================

(edit)

Example with a function that takes *args:

In [2]: import argparse
In [3]: def foo(*args, **kwargs):
   ...:     print('args',args)
   ...:     print('kwargs',kwargs)
   ...:     
In [4]: parser=argparse.ArgumentParser()
In [5]: parser.add_argument('arg1')
In [6]: parser.add_argument('arg2',nargs='+')

In [7]: args=parser.parse_args('one two three'.split())
In [8]: args
Out[8]: Namespace(arg1='one', arg2=['two', 'three'])

So I have 2 positional arguments, one with a single string value, the other with a list (due to the + nargs).

Call foo with these args attributes:

In [10]: foo(args.arg1)
args ('one',)
kwargs {}

In [11]: foo(args.arg1, args.arg2)
args ('one', ['two', 'three'])
kwargs {}

In [12]: foo(args.arg1, arg2=args.arg2)
args ('one',)
kwargs {'arg2': ['two', 'three']}

I defined 'positionals', but it would have worked just as well with 'optionals'. The distinction between positionals and optionals disappears in the namespace.

If I convert the namespace to a dictionary, I can pass values to foo in various ways, either through the *args or through **kwargs. It's all in how I call foo, not in how they appear in args or arguments. None of this is unique to argparse.

In [13]: arguments = vars(args)
In [14]: arguments
Out[14]: {'arg2': ['two', 'three'], 'arg1': 'one'}

In [15]: foo(arguments['arg2'], arguments['arg1'])
args (['two', 'three'], 'one')
kwargs {}

In [16]: foo(arguments['arg2'], arguments)
args (['two', 'three'], {'arg2': ['two', 'three'], 'arg1': 'one'})
kwargs {}

In [17]: foo(arguments['arg2'], **arguments)
args (['two', 'three'],)
kwargs {'arg2': ['two', 'three'], 'arg1': 'one'}

In [24]: foo(*arguments, **arguments)
args ('arg2', 'arg1')             # *args is the keys of arguments
kwargs {'arg2': ['two', 'three'], 'arg1': 'one'}

In [25]: foo(*arguments.values(), **arguments)
args (['two', 'three'], 'one')    # *args is the values of arguments
kwargs {'arg2': ['two', 'three'], 'arg1': 'one'}

Solution 2

How can I enable argparse to handle/to parse what is entered at the command line that is intended to be passed to the function via **kwargs?

This command:

$ python argstest.py location_by_coordinate 40.5949799 -73.9495148 DISTANCE=3000

does NOT execute the function call:

location_by_coordinate(40.5949799, -73.9495148, DISTANCE=3000)

That is easy to prove:

def location_by_coordinate(x, y, **kwargs):
    print "I was called!"

Go ahead and parse the args, and you'll see that the function isn't called. As a result, all your work setting up a subparser with the name location_by_coordinate was in vain.

The argparse module just examines sys.argv, which is a simple list of strings. Each string is one of the 'words' entered on the command line after the python command.

By default, the argument strings are taken from sys.argv,...
https://docs.python.org/3/library/argparse.html#the-parse-args-method

Yeah, sys.argv is a scary name, but a list of strings is just a list of strings. If you look at the argparse docs, all the examples do this:

parser.parse_args('--foo FOO'.split())

A list of strings you create with split() is no different than some list of strings that sys.argv refers to.

You need to call your location_by_coordinate() function yourself. In order to do that, you need to get the args from the command line, assemble the args that should be kwargs in a dictionary, and call your function like this:

location_by_coordinate(lat, lon, **my_dict)

If you have these values:

lat = 10
lon = 20
my_dict = {'a': 1, 'b': 2}

then the function call above will be equivalent to:

location_by_coordinate(10, 20, a=1, b=2)

Here is an example:

import argparse

def dostuff(x, y, **kwargs):
    print x, y, kwargs

parser = argparse.ArgumentParser()
parser.add_argument("LAT")
parser.add_argument("LON")
parser.add_argument("--distance")
args = parser.parse_args()
my_dict = {}
my_dict["distance"] = args.distance

dostuff(args.LAT, args.LON, **my_dict)

$ python my_prog.py 10 20 --distance 1
10 20 {'distance': '1'}

You can also get a dict from the parser:

...
...
args = parser.parse_args()
args_dict = vars(args)
print args_dict

--output:--
{'LAT': '10', 'distance': '1', 'LON': '20'}

lat = args_dict.pop('LAT')
lon = args_dict.pop('LON')
print args_dict

--output:--
{'distance': '1'}

location_by_coordinates(lat, lon, **args_dict)

If you want to make the user type:

DISTANCE=3000

on the command line, first of all I would not make them type all caps, so lets make the goal:

distance=3000

Add another mandatory argument to the parser:

location_by_parser.add_argument("distance", help="distance")

Then after you parse the following:

$ python argstest.py location_by_coordinate 40.5949799 -73.9495148 distance=3000

you can do this:

arguments = parser.parse_args()
args_dict = vars(arguments)

The args_dict will contain the key/value pair 'distance': 'distance=3000'. You can change that dict entry to 'distance': '3000' by doing the following:

pieces = args_dict['distance'].split('=')

if len(pieces) == 2 and pieces[0] == 'distance':
    args_dict['distance'] = pieces[1]

Or, you can set things up so that the parser will automatically execute that code by creating a custom action that executes when the distance arg is parsed:

class DistanceAction(argparse.Action):
    def __call__(self, parser, namespace, values, option_string=None):
        #values => The value for the distance command line arg
        pieces = values.split('=')

        if len(pieces) == 2 and pieces[0] in ['distance', 'wave_action']:  #only allow 'distance=' and 'wave_action='
            setattr(namespace, self.dest, pieces[1]) #The dest key specified in the parser gets assigned the value
        else:
            raise argparse.ArgumentTypeError('Usage: distance=3000.  Only distance=, wave_action= allowed.')

You can use the action like this:

location_by_parser.add_argument(
    "distance", 
    help="longitude", 
    action=DistanceAction
)

And if you want to get fancy, you can collect all the name=val args specified on the command line into one dictionary named, say, keyword_args, which will allow you to call your method like this:

args = parser.parse_args()
args_dict = vars(args)
keyword_args = args_dict["keyword_args"]

location_by_coordinates(lat, lon, **keyword_args)

Here's the parser configuration:

location_by_parser.add_argument(
    "keyword_args", 
    help="extra args", 
    nargs='*', 
    action=DistanceAction
)

---

import argparse
import sys

def location_by_coordinates(x, y, **kwargs):
    print x 
    print y
    print kwargs

class DistanceAction(argparse.Action):
    def __call__(self, parser, namespace, values, option_string=None):
        allowed_keywords = ['distance', 'wave_action']
        keyword_dict = {}

        for arg in values:  #values => The args found for keyword_args
            pieces = arg.split('=')

            if len(pieces) == 2 and pieces[0] in allowed_keywords:
                keyword_dict[pieces[0]] = pieces[1]
            else: #raise an error                                                         
                #Create error message:
                msg_inserts = ['{}='] * len(allowed_keywords)
                msg_template = 'Example usage: distance=3000. Only {} allowed.'.format(', '.join(msg_inserts))
                msg = msg_template.format(*allowed_keywords)

                raise argparse.ArgumentTypeError(msg)

        setattr(namespace, self.dest, keyword_dict) #The dest key specified in the
                                                    #parser gets assigned the keyword_dict--in
                                                    #this case it defaults to 'keyword_args'

parser = argparse.ArgumentParser(description="API Endpoints tester")
subparsers = parser.add_subparsers(dest="command", help="Available commands")

location_by_parser = subparsers.add_parser("location_by_coordinate", help="location function")
location_by_parser.add_argument("LAT", help="latitude")
location_by_parser.add_argument("LNG", help="longitude")
location_by_parser.add_argument("keyword_args", help="extra args", nargs='*', action=DistanceAction)

arguments = parser.parse_args()
args_dict = vars(arguments)

print args_dict

lat = args_dict['LAT']
lon = args_dict['LNG']
keyword_args = args_dict['keyword_args']

location_by_coordinates(lat, lon, **keyword_args)

Example:

$ python prog.py location_by_coordinate 40.5949799 -73.9495148 distance=3000 wave_action=1.4

{'LAT': '40.5949799', 'LNG': '-73.9495148', 'command': 'location_by_coordinate', 'keyword_args': {'distance': '3000', 'wave_action': '1.4'}}

40.5949799
-73.9495148
{'distance': '3000', 'wave_action': '1.4'}

$ python prog.py location_by_coordinate 40.5949799 -73.9495148 x=10
...
...
  File "2.py", line 25, in __call__
    raise argparse.ArgumentTypeError(msg)
argparse.ArgumentTypeError: Example usage: distance=3000. Only distance=, wave_action= allowed.

Comments

[AdjunctProfessorFalcon]

Thanks for this answer, very informative. It is possible to create an optional argument with argparse that will work with a function that takes *args? In other words, pass a list with argparse to a function?

[hpaulj]

I'm not entirely sure what you are asking, but I've added some examples of passing args values to a function that uses *args.

[AdjunctProfessorFalcon]

Ok, thank you, that's what I asking. Much appreciated!

[Bostone]

Just a hint: you can set parser.add_argument('arg2',nargs='*') which makes optional arguments truly optional