How to parse multiple nested sub-commands using python argparse?
Solution 1
@mgilson has a nice answer to this question. But problem with splitting sys.argv myself is that i lose all the nice help message Argparse generates for the user. So i ended up doing this:
import argparse
## This function takes the 'extra' attribute from global namespace and re-parses it to create separate namespaces for all other chained commands.
def parse_extra (parser, namespace):
namespaces = []
extra = namespace.extra
while extra:
n = parser.parse_args(extra)
extra = n.extra
namespaces.append(n)
return namespaces
argparser=argparse.ArgumentParser()
subparsers = argparser.add_subparsers(help='sub-command help', dest='subparser_name')
parser_a = subparsers.add_parser('command_a', help = "command_a help")
## Setup options for parser_a
## Add nargs="*" for zero or more other commands
argparser.add_argument('extra', nargs = "*", help = 'Other commands')
## Do similar stuff for other sub-parsers
Now after first parse all chained commands are stored in extra
. I reparse it while it is not empty to get all the chained commands and create separate namespaces for them. And i get nicer usage string that argparse generates.
Solution 2
I came up with the same qustion, and it seems i have got a better answer.
The solution is we shall not simply nest subparser with another subparser, but we can add subparser following with a parser following another subparser.
Code tell you how:
parent_parser = argparse.ArgumentParser(add_help=False)
parent_parser.add_argument('--user', '-u',
default=getpass.getuser(),
help='username')
parent_parser.add_argument('--debug', default=False, required=False,
action='store_true', dest="debug", help='debug flag')
main_parser = argparse.ArgumentParser()
service_subparsers = main_parser.add_subparsers(title="service",
dest="service_command")
service_parser = service_subparsers.add_parser("first", help="first",
parents=[parent_parser])
action_subparser = service_parser.add_subparsers(title="action",
dest="action_command")
action_parser = action_subparser.add_parser("second", help="second",
parents=[parent_parser])
args = main_parser.parse_args()
Solution 3
parse_known_args
returns a Namespace and a list of unknown strings. This is similar to the extra
in the checked answer.
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--foo')
sub = parser.add_subparsers()
for i in range(1,4):
sp = sub.add_parser('cmd%i'%i)
sp.add_argument('--foo%i'%i) # optionals have to be distinct
rest = '--foo 0 cmd2 --foo2 2 cmd3 --foo3 3 cmd1 --foo1 1'.split() # or sys.argv
args = argparse.Namespace()
while rest:
args,rest = parser.parse_known_args(rest,namespace=args)
print args, rest
produces:
Namespace(foo='0', foo2='2') ['cmd3', '--foo3', '3', 'cmd1', '--foo1', '1']
Namespace(foo='0', foo2='2', foo3='3') ['cmd1', '--foo1', '1']
Namespace(foo='0', foo1='1', foo2='2', foo3='3') []
An alternative loop would give each subparser its own namespace. This allows overlap in positionals names.
argslist = []
while rest:
args,rest = parser.parse_known_args(rest)
argslist.append(args)
Solution 4
The solution provide by @Vikas fails for subcommand-specific optional arguments, but the approach is valid. Here is an improved version:
import argparse
# create the top-level parser
parser = argparse.ArgumentParser(prog='PROG')
parser.add_argument('--foo', action='store_true', help='foo help')
subparsers = parser.add_subparsers(help='sub-command help', dest='subparser_name')
# create the parser for the "command_a" command
parser_a = subparsers.add_parser('command_a', help='command_a help')
parser_a.add_argument('bar', type=int, help='bar help')
# create the parser for the "command_b" command
parser_b = subparsers.add_parser('command_b', help='command_b help')
parser_b.add_argument('--baz', choices='XYZ', help='baz help')
# parse some argument lists
argv = ['--foo', 'command_a', '12', 'command_b', '--baz', 'Z']
while argv:
print(argv)
options, argv = parser.parse_known_args(argv)
print(options)
if not options.subparser_name:
break
This uses parse_known_args
instead of parse_args
. parse_args
aborts as soon as a argument unknown to the current subparser is encountered, parse_known_args
returns them as a second value in the returned tuple. In this approach, the remaining arguments are fed again to the parser. So for each command, a new Namespace is created.
Note that in this basic example, all global options are added to the first options Namespace only, not to the subsequent Namespaces.
This approach works fine for most situations, but has three important limitations:
- It is not possible to use the same optional argument for different subcommands, like
myprog.py command_a --foo=bar command_b --foo=bar
. - It is not possible to use any variable length positional arguments with subcommands (
nargs='?'
ornargs='+'
ornargs='*'
). - Any known argument is parsed, without 'breaking' at the new command. E.g. in
PROG --foo command_b command_a --baz Z 12
with the above code,--baz Z
will be consumed bycommand_b
, not bycommand_a
.
These limitations are a direct limitation of argparse. Here is a simple example that shows the limitations of argparse -even when using a single subcommand-:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('spam', nargs='?')
subparsers = parser.add_subparsers(help='sub-command help', dest='subparser_name')
# create the parser for the "command_a" command
parser_a = subparsers.add_parser('command_a', help='command_a help')
parser_a.add_argument('bar', type=int, help='bar help')
# create the parser for the "command_b" command
parser_b = subparsers.add_parser('command_b', help='command_b help')
options = parser.parse_args('command_a 42'.split())
print(options)
This will raise the error: argument subparser_name: invalid choice: '42' (choose from 'command_a', 'command_b')
.
The cause is that the internal method argparse.ArgParser._parse_known_args()
it is too greedy and assumes that command_a
is the value of the optional spam
argument. In particular, when 'splitting' up optional and positional arguments, _parse_known_args()
does not look at the names of the arugments (like command_a
or command_b
), but merely where they occur in the argument list. It also assumes that any subcommand will consume all remaining arguments.
This limitation of argparse
also prevents a proper implementation of multi-command subparsers. This unfortunately means that a proper implementation requires a full rewrite of the argparse.ArgParser._parse_known_args()
method, which is 200+ lines of code.
Given these limitation, it may be an options to simply revert to a single multiple-choice argument instead of subcommands:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--bar', type=int, help='bar help')
parser.add_argument('commands', nargs='*', metavar='COMMAND',
choices=['command_a', 'command_b'])
options = parser.parse_args('--bar 2 command_a command_b'.split())
print(options)
#options = parser.parse_args(['--help'])
It is even possible to list the different commands in the usage information, see my answer https://stackoverflow.com/a/49999185/428542
Solution 5
Improving on the answer by @mgilson, I wrote a small parsing method which splits argv into parts and puts values of arguments of commands into hierarchy of namespaces:
import sys
import argparse
def parse_args(parser, commands):
# Divide argv by commands
split_argv = [[]]
for c in sys.argv[1:]:
if c in commands.choices:
split_argv.append([c])
else:
split_argv[-1].append(c)
# Initialize namespace
args = argparse.Namespace()
for c in commands.choices:
setattr(args, c, None)
# Parse each command
parser.parse_args(split_argv[0], namespace=args) # Without command
for argv in split_argv[1:]: # Commands
n = argparse.Namespace()
setattr(args, argv[0], n)
parser.parse_args(argv, namespace=n)
return args
parser = argparse.ArgumentParser()
commands = parser.add_subparsers(title='sub-commands')
cmd1_parser = commands.add_parser('cmd1')
cmd1_parser.add_argument('--foo')
cmd2_parser = commands.add_parser('cmd2')
cmd2_parser.add_argument('--foo')
cmd2_parser = commands.add_parser('cmd3')
cmd2_parser.add_argument('--foo')
args = parse_args(parser, commands)
print(args)
It behaves properly, providing nice argparse help:
For ./test.py --help
:
usage: test.py [-h] {cmd1,cmd2,cmd3} ...
optional arguments:
-h, --help show this help message and exit
sub-commands:
{cmd1,cmd2,cmd3}
For ./test.py cmd1 --help
:
usage: test.py cmd1 [-h] [--foo FOO]
optional arguments:
-h, --help show this help message and exit
--foo FOO
And creates a hierarchy of namespaces containing the argument values:
./test.py cmd1 --foo 3 cmd3 --foo 4
Namespace(cmd1=Namespace(foo='3'), cmd2=None, cmd3=Namespace(foo='4'))
Comments
-
Vikas almost 2 years
I am implementing a command line program which has interface like this:
cmd [GLOBAL_OPTIONS] {command [COMMAND_OPTS]} [{command [COMMAND_OPTS]} ...]
I have gone through the argparse documentation. I can implement
GLOBAL_OPTIONS
as optional argument usingadd_argument
inargparse
. And the{command [COMMAND_OPTS]}
using Sub-commands.From the documentation it seems I can have only one sub-command. But as you can see I have to implement one or more sub-commands. What is the best way to parse such command line arguments useing
argparse
? -
Chris about 12 yearsThis doesn't really answer the question. Also, optparse is deprecated (from the python docs "The optparse module is deprecated and will not be developed further; development will continue with the argparse module").
-
Vikas about 12 yearsSorry for downvote, but this is not addressing the question i asked.
-
Vikas about 12 yearsThanks mgilson. This is a nice solution to my question, but i ended up doing it little differently. I added another answer.
-
Vikas almost 11 years@Flavius, after I get
namespace
from the parser by callingnamespace = argparser.parse_args()
, I callparse_extra
withparser
andnamespace
.extra_namespaces = parse_extra( argparser, namespace )
-
hpaulj over 10 yearsYes,
argparse
does allow nested subparsers. But I've only seen them used in one other place - in a test case for a Python issue, bugs.python.org/issue14365 -
jmlopez over 10 yearsI think I understand the logic, but what is
parser
in the code that you have. I only see it being used to add theextra
argument. Then you mentioned it again in the above comment. Is it supposed to beargparser
? -
Vikas over 10 years@jmlopez yeah it should be
argparser
. Will edit it. -
augurar almost 10 yearsThis assumes that the commands have a nested structure. But the question is asking for "parallel" commands
-
kzyapkov almost 8 years
-
HEADLESS_0NE almost 7 yearsReviewing your code above, I ran into one problem. On line 18, you refer to
split_argv[0]
which is actually empty insplit_argv
, because you append[c]
tosplit_argv
(intially set to[[]]
). If you change line 7 tosplit_argv = []
, everything works as expected. -
HEADLESS_0NE almost 7 yearsI did some more fixes (again) to the code you shared (fixing some issues I was running into) and ended up with this: gist.github.com/anonymous/f4be805fc3ff9e132eb1e1aa0b4f7d4b
-
wizebin over 6 yearsThis answer is pretty decent, you can determine which
subparser
was used by adding dest to theadd_subparsers
method stackoverflow.com/questions/8250010/… -
Adrian W over 6 yearsarghandler provides a nice way for declaring subcommands. However I don't see how this helps to solver OP's question: parsing multiple subcommands. The first subcommand parsed will eat up all remaining arguments, so further commands will never be parsed. Please give a hint on how to solve this with arghandler. Thanks.
-
Adrian W over 6 yearsWorks nicely. There is a flaw however: if there is a misspelled option somewhere (e.g.
rest = '--foo 0 cmd2 --foo2 2 --bar cmd3 --foo3 3 cmd1 --foo1 1'.split()
), then argparse will end inerror: too few arguments
instead of pointing out the invalid option. This is because the bad option will be left inrest
until we are out of command arguments. -
Adrian W over 6 yearsThe comment
# or sys.argv
should be# or sys.argv[1:]
. -
MacFreek about 6 yearsNote that this solution fails for subcommand-specific optional arguments. See my solution below (stackoverflow.com/a/49977713/428542) for an alternative solution.
-
MacFreek about 6 yearsHere is an example of how this fails. Add the following 3 lines:
parser_b = subparsers.add_parser('command_b', help='command_b help')
;parser_b.add_argument('--baz', choices='XYZ', help='baz help')
;options = argparser.parse_args(['--foo', 'command_a', 'command_b', '--baz', 'Z'])
; This fails with an errorPROG: error: unrecognized arguments: --baz Z
. The reason is that during the parsing ofcommand_a
, the optional arguments ofcommand_b
are already parsed (and are unknown for the subparser ofcommand_a
). -
Gary over 2 yearsI am trying to create multiple subparsers like
service
in this case since I demarcation of commands. It errs out saying I cannot have duplicate commands? I want some thing like thiscommand --user us pass --debug false subcommandone subcommandoption
orcommand --user us pass --debug false subcommandtwo subcommandoption
orcommandtwo --user us pass subcommandthree subcommandoption
. I intend to add complete help docs accordingly also. The code will be run aspython test.py command --user us pass --debug false subcommandtwo subcommandoption
-
Gary over 2 yearsI was able to work with
import argparse parser = argparse.ArgumentParser() parser.add_argument('--bar', type=int, help='bar help') parser.add_argument('commands', nargs='*', metavar='COMMAND', choices=['command_a', 'command_b'])