Python: Respond to Command Line Prompts
Solution 1
In the comments you mentioned that xx viewproject < answers.txt > output.txt
works but you can't use it because answers depend on the output from the subprocess.
In general pexpect
-like modules such as winpexpect
(for Windows) could be used. Something like:
import re
import sys
from functools import partial
from winpexpect import EOF, winspawn as spawn
p = spawn('xx viewproject')
p.logfile = sys.stdout
patterns = ['the project:', re.escape('? [ynYN](n)'), EOF]
for found in iter(partial(p.expect, patterns), 2): # until EOF
if found == 0:
p.sendline(project_name)
elif found == 1:
filename = get_filename_from_prompt(p.before) # a regex could be used
answer = yes_or_no_from_subproject.get(filename, 'no') # a dict
p.sendline(answer)
If the prompts are terminated with a newline (and the subprocess doesn't buffer them); you could read line by line using subprocess
module directly:
from subprocess import Popen, PIPE
with Popen(["xx", "viewproject"], stdin=PIPE, stdout=PIPE,
universal_newlines=True) as p:
for line in p.stdout:
if line.startswith("Please enter the name of the project"):
answer = project_name
elif line.startswith("Would you like to recurse into the subproject"):
filename = get_filename_from_prompt(line) # a regex could be used
answer = yes_or_no_from_subproject.get(filename, 'n') # a dict
else:
continue # skip it
print(answer, file=p.stdin) # provide answer
p.stdin.flush()
To test that you can read something from the xx
using subprocess
:
from subprocess import Popen, PIPE, STDOUT
with Popen(["xx", "viewproject"], bufsize=0,
stdin=PIPE, stdout=PIPE, stderr=STDOUT) as p:
print(repr(p.stdout.read(1)))
Solution 2
Yes, first of all you may create subprocess as an object by:
p = subprocess.Popen('xx viewproject', shell=True, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, universal_newlines=True)
Then you'll have methods like communicate()
available, for instance:
newline = os.linesep # [1]
commands = ['y', 'n', 'y', 'n', 'y']
p.communicate( newline.join( commands))
1 - os.linesep
Which will send all the answers at once (and hopefully it'll be enough) relying on the same order of question every time.
You may also try parsing p.stdout
and then writing to p.stdin
, but this may cause deadlock when one buffer will get full while waiting for another, so be careful with this. Luckily there are some complex examples on google.
Simple version would be:
p = Popen(...)
line = p.stdout.readline() # At this point, if child process will wait for stdin
# you have a deadlock on your hands
parse_line( line)
p.stdin.write( newline.join( commands).encode( 'utf-8'))
I would also consider rewriting:
p = subprocess.Popen('si viewproject --project=d:/Projects/test.pj', shell=True,
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
To:
p = subprocess.Popen( ['si', 'viewproject', '--project=d:/Projects/test.pj'],
shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
Unless you explicitly need Shell invocation.
Stoating
Updated on June 05, 2022Comments
-
Stoating almost 2 years
I am trying to use Python to interact with another program via the command line. The main problem I am having is a specific call that has multiple follow-up prompts. Initially the command line call asks for the name of a project and then proceeds to ask if I would like to view any of the subfolders of the project. I need to answer y/n to each of these in order and the answer to each is unfortunately not all y or n. Additionally, I cannot know the answer to the question without reading the individual prompts so I am incapable of sending a block of 'y's or 'n's all at once.
This is the command line call:
si viewproject
After entering the command, the command line prompts:
Enter the project name:
And an example response would be:
Enter the project name: c:/test.pj
After entering the project, it prompts the following:
Do you want to recurse into the subproject test_subprj.pj? [ynYN](n)
At which point I need to respond with either a y or n depending on if I need that subproject. Again, the response to this question is dependent on the subproject. I need to be able to read the subproject in this prompt in order to respond to it with a 'y' or 'n'
Currently I need to manually enter in the project and each of the y's and n's respectively. My goal is to automate this process using Python.
Is there a way to respond to these command line prompts automatically?
Current Progress
Subprocess Strategy
project_path = "c:/test.pj" with Popen(["si", "viewproject", "--project=" + project_path], stdin=PIPE, stdout=PIPE, universal_newlines=True) as p: for line in p.stdout: if line.startswith("Do you want"): answer = 'n' else: continue # skip it print(answer, file=p.stdin) # provide answer p.stdin.flush()
This method is hanging after the with Popen statement. It never errors, but it never enters or exits the for statement and never completes. Currently I am defaulting all answers to "n", but that will be replaced with logic later.
Winpexpect Strategy
import re import sys from functools import partial import winpexpect project_path = "c:/test.pj" p = winpexpect.winspawn('si viewproject --project=' + project_path) p.logfile = sys.stdout patterns = [re.compile('ynYN'), winpexpect.EOF] for found in iter(partial(p.expect, patterns), 1): # until EOF if found == 0: answer = 'n' p.sendline(answer)
Returns the following error message:
Traceback (most recent call last): File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\winpexpect.py", line 541, in read_nonblocking handle, status, data = self.child_output.get(timeout=timeout) File "C:\Python33\lib\queue.py", line 175, in get raise Empty queue.Empty During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1378, in expect_loop c = self.read_nonblocking (self.maxread, timeout) File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\winpexpect.py", line 543, in read_nonblocking raise TIMEOUT('Timeout exceeded in read_nonblocking().') pexpect.TIMEOUT: Timeout exceeded in read_nonblocking(). During handling of the above exception, another exception occurred: Traceback (most recent call last): File "K:\eclipse_3.6.0\plugins\org.python.pydev_2.6.0.2012062818\pysrc\pydev_runfiles.py", line 432, in __get_module_from_str mod = __import__(modname) File "C:\workspace\Test_prj\Test_prj.py", line 19, in <module> for found in iter(partial(p.expect, patterns), 1): # until EOF File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1311, in expect return self.expect_list(compiled_pattern_list, timeout, searchwindowsize) File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1325, in expect_list return self.expect_loop(searcher_re(pattern_list), timeout, searchwindowsize) File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1409, in expect_loop raise TIMEOUT (str(e) + '\n' + str(self)) pexpect.TIMEOUT: Timeout exceeded in read_nonblocking(). <winpexpect.winspawn object at 0x0144AE50> version: 2.3 ($Revision: 399 $) command: si args: ['si', 'viewproject', '--project=c:/test.pj'] searcher: searcher_re: 0: re.compile("ynYN") 1: EOF buffer (last 100 chars): before (last 100 chars): after: <class 'pexpect.TIMEOUT'> match: None match_index: None exitstatus: None flag_eof: False pid: 6448 child_fd: 4 closed: False timeout: 30 delimiter: <class 'pexpect.EOF'> logfile: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='Cp1252'> logfile_read: None logfile_send: None maxread: 2000 ignorecase: False searchwindowsize: None delaybeforesend: 0.05 delayafterclose: 0.1 delayafterterminate: 0.1 ERROR: Module: Test_prj could not be imported (file: C:\workspace\Test_prj\Test_prj.py).
Installing Winpexpect
Lazy Persons Way
First-World Problems
Python is a new language for me, and I had never installed a package before for Python. Additionally, Python 3.x is a little different than the other versions of Python making installing modules a little bit more of an adventure.
So, to help others get some sweet sweet module action (and to help those who are more knowledgeable see if I did anything wrong) here's a soon to be success story (hopefully) documenting how I got and installed my first module.
Setup
Python allows third-party groups to develop and distribute modules that extend the abilities of the programming language. Naturally, there is a standard way to help third-party developers make modules as easily available to the end-user as possible.
For Python 3.x, that standard for distributing modules is called Distutils.
Here is how a developer uses Distutils: Distributing Python Modules
And here is how the end-user uses Distutils: Installing Python Modules
Normally, navigating to the folder of your downloaded module in the command line and running "setup.py install" will be enough.
BUT
Sometimes life isn't so easy and you may still have problems with your installation. You may, in fact, need something else. For example, you may get the following error:
"ImportError “No Module named Setuptools”"
Luckily, there is a solution for that: Python 3: ImportError "No Module named Setuptools"
As it turns out, not everything uses distutils. Some packages use setuptools. Unfortunately, there is no setuptools for Python 3.x. Rather, Python 3.x uses distribute which is a branch of setuptools.
So for those who use Python 3.x, here is Distribute: Distribute
For those using Python 2.x, here is Setuptools: Setuptools
In the Installation Instructions for Distribute, it says the following: "Download
distribute_setup.py <http://python-distribute.org/distribute_setup.py>
_ and execute it, using the Python interpreter of your choice."It also says: "Notice this file is also provided in the source release."
So I downloaded Distribute and saved it to the computer. Once it was saved to the computer, I ran distribute_setup.py from the source release and got the following error:
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.36.tar.gz Traceback (most recent call last): File "C:\Python33\lib\urllib\request.py", line 1252, in do_open h.request(req.get_method(), req.selector, req.data, headers) File "C:\Python33\lib\http\client.py", line 1049, in request self._send_request(method, url, body, headers) File "C:\Python33\lib\http\client.py", line 1087, in _send_request self.endheaders(body) File "C:\Python33\lib\http\client.py", line 1045, in endheaders self._send_output(message_body) File "C:\Python33\lib\http\client.py", line 890, in _send_output self.send(msg) File "C:\Python33\lib\http\client.py", line 828, in send self.connect() File "C:\Python33\lib\http\client.py", line 806, in connect self.timeout, self.source_address) File "C:\Python33\lib\socket.py", line 406, in create_connection for res in getaddrinfo(host, port, 0, SOCK_STREAM): socket.gaierror: [Errno 11001] getaddrinfo failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\workspace\PythonTest\distribute_setup.py", line 553, in <module> sys.exit(main()) File "C:\workspace\PythonTest\distribute_setup.py", line 549, in main tarball = download_setuptools(download_base=options.download_base) File "C:\workspace\PythonTest\distribute_setup.py", line 204, in download_setuptools src = urlopen(url) File "C:\Python33\lib\urllib\request.py", line 160, in urlopen return opener.open(url, data, timeout) File "C:\Python33\lib\urllib\request.py", line 473, in open response = self._open(req, data) File "C:\Python33\lib\urllib\request.py", line 491, in _open '_open', req) File "C:\Python33\lib\urllib\request.py", line 451, in _call_chain result = func(*args) File "C:\Python33\lib\urllib\request.py", line 1272, in http_open return self.do_open(http.client.HTTPConnection, req) File "C:\Python33\lib\urllib\request.py", line 1255, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [Errno 11001] getaddrinfo failed>
Well that is no good! I honestly still do not know where that error is coming from or why it happened.
Regardless, then I found the following site that ran a .exe to install distribute as well as pip.
So I got those installed and then used the following site to setup my computer to more easily use easy_install: Setting Up Easy Install Made Easy
Once I got this working I then installed nose:Nose
The reason I got nose was because the Winpexpect website says: "WinPexpect includes unit tests. To run the tests, you need nose. Use the following command to run the tests:
$ python setup.py test"
Well that sounds nice :). Now I just wished I knew where to run that test. I know that if you install manually you use the setup.py install command so there will most definitely be a setup.py in the zipped directory online. To see if this was correct, I downloaded and saved the winpexpect file, extracted the information, navigated to it via command-line, and ran setup.py test.
Here was the following result:
running test running build_py running egg_info creating c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info writing c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\PKG-INFO writing dependency_links to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\dependency_links.txt writing top-level names to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\top_level.txt writing requirements to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\requires.txt writing manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt' reading manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt' writing manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt' running build_ext Traceback (most recent call last): File "C:\Documents and Settings\SLZ1FH\Desktop\winpexpect\geertj-winpexpect-76df3cfcb143\setup.py", line 35, in <module> use_2to3 = True File "C:\Python33\lib\distutils\core.py", line 148, in setup dist.run_commands() File "C:\Python33\lib\distutils\dist.py", line 917, in run_commands self.run_command(cmd) File "C:\Python33\lib\distutils\dist.py", line 936, in run_command cmd_obj.run() File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 138, in run self.with_project_on_sys_path(self.run_tests) File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 118, in with_project_on_sys_path func() File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 164, in run_tests testLoader = cks File "C:\Python33\lib\unittest\main.py", line 124, in __init__ self.parseArgs(argv) File "C:\Python33\lib\unittest\main.py", line 168, in parseArgs self.createTests() File "C:\Python33\lib\unittest\main.py", line 175, in createTests self.module) File "C:\Python33\lib\unittest\loader.py", line 137, in loadTestsFromNames suites = [self.loadTestsFromName(name, module) for name in names] File "C:\Python33\lib\unittest\loader.py", line 137, in <listcomp> suites = [self.loadTestsFromName(name, module) for name in names] File "C:\Python33\lib\unittest\loader.py", line 96, in loadTestsFromName module = __import__('.'.join(parts_copy)) File "C:\Python33\lib\site-packages\nose-1.3.0-py3.3.egg\nose\__init__.py", line 1, in <module> from nose.core import collector, main, run, run_exit, runmodule File "C:\Python33\lib\site-packages\nose-1.3.0-py3.3.egg\nose\core.py", line 143 print "%s version %s" % (os.path.basename(sys.argv[0]), __version__) ^ SyntaxError: invalid syntax
Ok, so the Python 3.3 version of Nose contains invalid syntax for Python 3.3?
print "%s version %s" % (os.path.basename(sys.argv[0]), version)...
should definitely have parenthesis around it... This makes me question if nose will actually work here as it clearly looks to be made for earlier versions of Python.
-
Stoating about 11 yearsTypeError: 'str' does not support the buffer interface Enter the project name: *** A value for "--project" is required.
-
Stoating about 11 yearsWoah, sorry about not leaving a description of the error O_o. Oops. For the array of commands, unfortunately I will not know all of the answers at once because the y/n response is based off of the name of the subfolder presented and I do not know when that subfolder will appear. As for the error it is the following line:
p.communicate(newline.join(commands))
For the test project I know that supplying the commands of['n', 'n', 'n']
will sufficiently answer all subsequent prompts after providing the project name. -
Stoating about 11 yearsThe current code looks as such:
p = subprocess.Popen('si viewproject --project=d:/Projects/test.pj', shell = True, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
newline = os.linesep
commands = ['n', 'n', 'n']
p.communicate(newline.join(commands))
Providing the name of the project in the Popen works, and for the test project the three 'n's are sufficient. Regardless, the error onp.communicate(newline.join(commands))
persists. -
Vyktor about 11 years@Stoating could you please paste a traceback here which triggers your error?
-
Stoating about 11 years
Traceback (most recent call last): File "C:\workspace\Subprj_Autogen\Subprj_Autogen.py", line 21, in <module> p.communicate(newline.join(commands)) File "C:\Python33\lib\subprocess.py", line 906, in communicate stdout, stderr = self._communicate(input, endtime, timeout) File "C:\Python33\lib\subprocess.py", line 1180, in _communicate self.stdin.write(input) TypeError: 'str' does not support the buffer interface
-
jfs about 11 years@Stoating:
p.communicate()
expects bytes by default on Python 3.3. You could specifyPopen(..., universal_newlines=True)
to work with Unicode strings instead. -
Vyktor about 11 yearsI'm curious, why suggesting Windows only non-standard module? What are benefits?
-
jfs about 11 years@Vyktor: OP uses Windows.
pexpect
works on *nix if you need it. You can't use.communicate()
because the answers depend on the subprocess output. If the prompts are not terminated by a newline then you need to duplicate/reimplementpexpect
-like code to read/parse them and there could be issues e.g., the subprocess might use block-buffering if it is run non-interactively. -
Vyktor about 11 yearsI know you can't use communicate but it wasn't obvious to be from original question (just from comment) so I've added new example on how to use reading/writing to pipes directly + link how to do this block-buffering-safe... I just prefer applications that are portable to any platform without hacks like
if os.platform == 'nt'
and on clean python installations without any external packages (if possible) so I'm just curious what are the benefits. -
jfs about 11 years@Vyktor: the example you provided does not handle the block-buffering issue: Python process might not see the prompt due to buffering and without an answer the child process blocks.
-
Stoating about 11 yearsI have now tried both methods, but am running into errors with each. Using the subprocess strategy, the code hangs right after the Popen statement and never enters the for loop. For the Winpexpect strategy, there is a very long error message that appears to be a TIMEOUT.
-
jfs about 11 years@Stoating: TIMEOUT means that none of the patterns that you provided matched. You could add TIMEOUT to the patterns explicitly and inspect
p.before
. It seems like your script doesn't see any output from the child process. -
Stoating about 11 years@J.F. Sebastian: I had also added winpexpect.TIMEOUT to the list of patterns available. Then it doesn't cause an error, but simply repeatedly times out. Removing the timeout pattern and being able to see an error message seemed more beneficial for debugging. After reducing the regular expression to simply look for a couple letters I would agree that the script isn't seeing any output from the child process. Am I using winpexpect incorrectly or is there a property of the command prompt that could be causing this? I have never downloaded a module before this.
-
jfs about 11 years@Stoating: I've added a test that something can be read from the subprocess.