Passing command line arguments to argv in jupyter/ipython notebook

Question

I'm wondering if it's possible to populate sys.argv (or some other structure) with command line arguments in a jupyter/ipython notebook, similar to how it's done through a python script.

For instance, if I were to run a python script as follows:

python test.py False

Then sys.argv would contain the argument False. But if I run a jupyter notebook in a similar manner:

jupyter notebook test.ipynb False

Then the command line argument gets lost. Is there any way to access this argument from within the notebook itself?

Solution 1

After a lot of looking around I found very cumbersome, custom libraries, but solved it with a few lines of code which I thought was pretty slick. I used nbconvert to end up with an html report as output that contains all graphics and markdown from the notebook, but accepts command line parameters just as always through a minimal python wrapper:

The python file test_args.py (which takes command line params as normal):

import sys,os
IPYNB_FILENAME = 'test_argv.ipynb'
CONFIG_FILENAME = '.config_ipynb'

def main(argv):
    with open(CONFIG_FILENAME,'w') as f:
        f.write(' '.join(argv))
    os.system('jupyter nbconvert --execute {:s} --to html'.format(IPYNB_FILENAME))
    return None

if __name__ == '__main__':
    main(sys.argv)

The notebook contains:

import sys,os,argparse
from IPython.display import HTML
CONFIG_FILE = '.config_ipynb'
if os.path.isfile(CONFIG_FILE):
    with open(CONFIG_FILE) as f:
        sys.argv = f.read().split()
else:
    sys.argv = ['test_args.py', 'input_file', '--int_param', '12']

parser = argparse.ArgumentParser()
parser.add_argument("input_file",help="Input image, directory, or npy.")
parser.add_argument("--int_param", type=int, default=4, help="an optional integer parameter.")
args = parser.parse_args()
p = args.int_param
print(args.input_file,p)

and I can run the python notebook with arguments parsed as usual:

python test_args.py my_input_file --int_param 12

I tend to paste the block with argparse calls into the python wrapper so that command line errors are caught by the python script and -h works properly.

Solution 2

There are two projects I've found that do what you ask for

• Papermill, will add a cell to your notebook with arguments that you pass to it on the command line. So this is quite straightforward, you define your defaults in the first cell (the should have parameters tag)
• nbparameterise it is a similar concept but you don't tag your cell with defaults, it has to be first.

Here is a good resource discussing the issue: https://github.com/jupyter/help/issues/218

Solution 3

I think this Gist may help you : https://gist.github.com/gbishop/acf40b86a9bca2d571fa

This is an attempt at a simple argument parser for mostly key=value pairs that can be used both on the command line and in IPython notebooks. It support query parameters in notebook URLs and a Run command for notebooks.

Solution 4

Using args = parser.parse_args(args=[]) would work.

or for testing, you can declare it as class format.

class Args:
  data = './data/penn'
  model = 'LSTM'
  emsize = 200
  nhid = 200

args=Args()

Solution 5

If the goal is to run a notebook with configurable arguments passed from commandline, I think the easiest way is to use environment variables, like this:

NB_ARGS=some_args jupyter nbconvert --execute --to html --template full some_notebook.ipynb

Then in the notebook, you can import os and use os.environ['NB_ARGS']. The variable value can be some text that contains key-value pairs or json for example.

View more solutions

Comments

[Cody Gray]

A link to a solution is welcome, but your answer must be self-contained and useful even with the link: please quote or summarize the relevant information from the page you're linking to, in case that page becomes unavailable.

[justadampaul]

Hey, cool! Thanks for stopping by. I had long since moved on to other projects and I'm sure I found some easy way around my problem back then, or else employed one of the cumbersome libraries you mention. But this is a neat little fix and I'll mark it as the answer for any future enquirers.

[justadampaul]

Papermill looks excellent. Thanks for the updated information.

[Joseph]

Note that the syntax to set the environment variable will be different on Windows. In a batch file, you can have set NB_ARGS=some_args followed by the jupyter command on the next line.

[amka66]

In test_args.py, you might want to remove IPYNB_FILENAME and receive the notebook's filename as the first command line argument. This will make test_args.py a generic script to apply to any notebook requiring arguments. Otherwise, you need an additional script per notebook.

[UrbanConor]

Thank you for sharing Papermill, after lots of searching online I wouldn't have found this library otherwise!

[Rachel Shalom]

papermill is great and actually solved my problem needing to run jupyter notebook with arguments. I recommend this blog for quick start: towardsdatascience.com/introduction-to-papermill-2c61f66bea3‌​0

[M.E.]

This is by far the simplest and cleanest way to pass arguments to a Jupyter Notebook