qstat and long job names
Solution 1
This on is a bit messy, but it works as a simple solution to have in the command history. All standard tools. Output is pretty much the same as what you get from a normal qstat call, but you won't get the headers:
One-liner:
qstat -xml | tr '\n' ' ' | sed 's#<job_list[^>]*>#\n#g' \
| sed 's#<[^>]*>##g' | grep " " | column -t
Description of commands:
List jobs as XML:
qstat -xml
Remove all newlines:
tr '\n' ' '
Add newline before each job entry in the list:
sed 's#<job_list[^>]*>#\n#g'
Remove all XML stuff:
sed 's#<[^>]*>##g'
Hack to add newline at the end:
grep " "
Columnize:
column -t
Example output
351996 0.50502 ProjectA_XXXXXXXXX_XXXX_XXXXXX user123 r 2015-06-25T15:38:41 [email protected] 1
351997 0.50502 ProjectA_XXX_XXXX_XXX user123 r 2015-06-25T15:39:26 [email protected] 1
351998 0.50502 ProjectA_XXXXXXXXXXXXX_XXXX_XXXX user123 r 2015-06-25T15:40:26 [email protected] 1
351999 0.50502 ProjectA_XXXXXXXXXXXXXXXXX_XXXX_XXXX user123 r 2015-06-25T15:42:11 [email protected] 1
352001 0.50502 ProjectA_XXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXX user123 r 2015-06-25T15:42:11 [email protected] 1
352008 0.50501 runXXXX69 usr1 r 2015-06-25T15:49:04 [email protected] 1
352009 0.50501 runXXXX70 usr1 r 2015-06-25T15:49:04 [email protected] 1
352010 0.50501 runXXXX71 usr1 r 2015-06-25T15:49:04 [email protected] 1
352011 0.50501 runXXXX72 usr1 r 2015-06-25T15:49:04 [email protected] 1
352012 0.50501 runXXXX73 usr1 r 2015-06-25T15:49:04 [email protected] 1
352013 0.50501 runXXXX74 usr1 r 2015-06-25T15:49:04 [email protected] 1
Solution 2
Maybe an easier solution: set SGE_LONG_JOB_NAMES to -1, and qstat will figure out the size of the name column:
export SGE_LONG_JOB_NAMES=-1
qstat -u username
Works for me.
Cheers!
Solution 3
This script works pretty well. It looks like it is from cambridge. http://www.hep.ph.ic.ac.uk/~dbauer/grid/myqstat.py
For Python 3:
#!/usr/bin/python
import xml.dom.minidom
import os
import sys
import string
f=os.popen('qstat -u \* -xml -r')
dom=xml.dom.minidom.parse(f)
jobs=dom.getElementsByTagName('job_info')
run=jobs[0]
runjobs=run.getElementsByTagName('job_list')
def fakeqstat(joblist):
for r in joblist:
try:
jobname=r.getElementsByTagName('JB_name')[0].childNodes[0].data
jobown=r.getElementsByTagName('JB_owner')[0].childNodes[0].data
jobstate=r.getElementsByTagName('state')[0].childNodes[0].data
jobnum=r.getElementsByTagName('JB_job_number')[0].childNodes[0].data
jobtime='not set'
if(jobstate=='r'):
jobtime=r.getElementsByTagName('JAT_start_time')[0].childNodes[0].data
elif(jobstate=='dt'):
jobtime=r.getElementsByTagName('JAT_start_time')[0].childNodes[0].data
else:
jobtime=r.getElementsByTagName('JB_submission_time')[0].childNodes[0].data
print(jobnum, '\t', jobown.ljust(16), '\t', jobname.ljust(16),'\t', jobstate,'\t',jobtime)
except Exception as e:
print(e)
fakeqstat(runjobs)
For Python 2:
#!/usr/bin/python
import xml.dom.minidom
import os
import sys
import string
#import re
f=os.popen('qstat -u \* -xml -r')
dom=xml.dom.minidom.parse(f)
jobs=dom.getElementsByTagName('job_info')
run=jobs[0]
runjobs=run.getElementsByTagName('job_list')
def fakeqstat(joblist):
for r in joblist:
jobname=r.getElementsByTagName('JB_name')[0].childNodes[0].data
jobown=r.getElementsByTagName('JB_owner')[0].childNodes[0].data
jobstate=r.getElementsByTagName('state')[0].childNodes[0].data
jobnum=r.getElementsByTagName('JB_job_number')[0].childNodes[0].data
jobtime='not set'
if(jobstate=='r'):
jobtime=r.getElementsByTagName('JAT_start_time')[0].childNodes[0].data
elif(jobstate=='dt'):
jobtime=r.getElementsByTagName('JAT_start_time')[0].childNodes[0].data
else:
jobtime=r.getElementsByTagName('JB_submission_time')[0].childNodes[0].data
print jobnum, '\t', jobown.ljust(16), '\t', jobname.ljust(16),'\t', jobstate,'\t',jobtime
fakeqstat(runjobs)
Solution 4
I am currently writing my own qstat
wrapper in order to get a clean, useful and customizable output.
Here is the github repository. The project has grown too much for the code to be pasted in this message.
It comes with an installer and should work without any problem with both Python 2.7 and 3 (the installation script makes the modifications if needed). qjobs -h
provides some help on the available options. I will write a more complete documentation in the following days on the github wiki.
I will update this message as often as possible to stick to the current state of the project. Please feel free to comment here (or on github) to ask for features/report problems.
In the near future, I will try to add a fully interactive mode to browse the job list more easily. Of course, the classic text output will be still available (it could be useful to e-mail the output, or for a quick check of the pending/running jobs).
Example output
Command qjobs
gives:
5599109 short_name r 2015-06-25 10:27:39 queue1
5599110 jobName r 2015-06-25 10:35:39 queue2
5599111 a_long_job_name qw 2015-06-25 10:40:39
5599112 foo qw 2015-06-25 10:40:39
5599113 bar qw 2015-06-25 10:40:39
5599114 baz qw 2015-06-25 10:40:39
5599115 beer qw 2015-06-25 10:40:39
tot: 7
r: 2 qw: 5
Command qjobs -o
gives:
tot: 7
r: 2 qw: 5
Command qjobs -o inek -t
gives (e
is elapsed time since start/sub time, the format is customizable using the Format Spec. Mini-Language of Python; k
is complete queue name, with domain):
5598985 SpongeBob 522:02 (21.75 days) [email protected]
5598987 ping_java 521:47 (21.74 days) [email protected]
5598988 run3.14 521:46 (21.74 days) [email protected]
5598990 strange_job_42 521:42 (21.74 days) [email protected]
5598991 coffee-maker 521:39 (21.74 days) [email protected]
5598992 dumbtask 521:29 (21.73 days) [email protected]
qjobs -i
gives a complete list of the available 'items'. Each of this item is available as:
- a column output (with
-o ITEMS
); - as a criteria to count the job and produces total output, with
-t
(e.g.-t s
to count by state as in the two first examples); - as a criteria to sort the job with
-s
, default is-s ips
meaning that the job list is sorted by ID, then by priority and finally by state before being printed.
The result of qjobs -i
is:
i: job id
p: job priority
n: job name
o: job owner
s: job state
t: job start/submission time
e: elapsed time since start/submission
q: queue name without domain
d: queue domain
k: queue name with domain
r: requested queue(s)
l: number of slots used
Solution 5
Thanks to JLT for nice simple code. I've expanded it a bit to fit my needs and make it look nice.
Sample Output:
Job ID Job Name Owner Status
------ ------------------------------------ ------ ------
201716 AtacSilN100400K mtsige R
201771 IsoOnGrap400K mtsige R
202067 AtacOnSilica400K mtsige R
202100 AtacGrapN100400K mtsige R
202135 AtacOnSilc400K mtsige R
202145 AtacOnGrap400K mtsige R
202152 AtacOnGraphN3360K mtsige R
202161 AtacticSilicaN10 mtsige R
202163 AtacGrapN10 mtsige R
202169 AtacSilcN10 mtsige R
202192 wallpmma07 am110 R
202193 wallpmma03 am110 R
202194 att03wpm_95solps am110 R
202202 AtacticSilicaN3 mtsige R
203260 8test18_trop_2p ico R
203359 parseAll_Bob/Sub951By50/Cyl20A_atom1 oge1 R
203360 parseAll_Bob/Sub951By50/Cyl30A_atom1 oge1 R
203361 parseAll_Bob/Sub951By50/Cyl30A_atom2 oge1 R
Code:
#!/opt/bin/python3
import os
import xml.etree.ElementTree as ET
#Fields
fields=['Job_Id','Job_Name','Job_Owner','job_state']
names=['Job ID','Job Name','Owner','Status']
#Get job info
f = os.popen('qstat -x')
tree = ET.parse(f)
root = tree.getroot()
n_fields=len(fields)
jobs=[[job.find(field).text for field in fields] for job in root]
max_lengths=[len(name) for name in names]
sep=' '
#Identify max characer length per field
for j in jobs:
for i in range(n_fields):
#Chop off anything after and including '@' or '.' from all fields
if j[i].find('@')>0:
j[i]=j[i][:j[i].find('@')]
if j[i].find('.')>0:
j[i]=j[i][:j[i].find('.')]
if(len(j[i])>max_lengths[i]):
max_lengths[i]=len(j[i])
#Field names
for i in range(n_fields):
print('{s:^{length}}'.format(s=names[i],length=max_lengths[i]),end=sep)
print()
#Dashes
for i in range(n_fields):
print('-'*max_lengths[i],end=sep)
print()
#Jobs
for j in jobs:
for i in range(n_fields):
if j[i].find('@')>0:
j[i]=j[i][:j[i].find('@')]
print('{s:<{length}}'.format(s=j[i],length=max_lengths[i]),end=sep)
print()
Related videos on Youtube
adrin
Updated on July 09, 2022Comments
-
adrin almost 2 years
How can I get
qstat
to give me full job names?I know
qstat -r
gives detailed information about the task, but it's too much and the resource requirements are included.The
qstat -r
output is like:131806 0.25001 tumor_foca ajalali qw 09/29/2014 15:49:41 1 2-100:1 Full jobname: tumor_focality-TCGA-THCA-ratboost_linear_svc Hard Resources: distribution=wheezy (0.000000) h_rt=72000 (0.000000) mem_free=15G (0.000000) h_vmem=15G (0.000000) h_stack=256M (0.000000) Soft Resources: 131807 0.25001 vital_stat ajalali qw 09/29/2014 15:49:41 1 2-100:1 Full jobname: vital_status-TCGA-LGG-ratboost_linear_svc Hard Resources: distribution=wheezy (0.000000) h_rt=72000 (0.000000) mem_free=15G (0.000000) h_vmem=15G (0.000000) h_stack=256M (0.000000) Soft Resources:
Right now my only option is to
grep
the output as I need:$ qstat -r | grep "Full jobname" -B1 -- 131806 0.25001 tumor_foca ajalali qw 09/29/2014 15:49:41 1 2-100:1 Full jobname: tumor_focality-TCGA-THCA-ratboost_linear_svc -- 131807 0.25001 vital_stat ajalali qw 09/29/2014 15:49:41 1 2-100:1 Full jobname: vital_status-TCGA-LGG-ratboost_linear_svc
Can I do it better to have a nicer output?
-
Ilya Orson about 7 yearsTo get the full job names of all the actual jobs of a given user:
qstat -f | grep -C 1 username@
You can get more information with-C 2
,-C 3
, etc.
-
-
adrin over 9 yearsThanks. I changed your copy/pasted code to support python3. It works now.
-
tflutre about 9 years@PhysicalChemist
os.popen
will be replaced in the future, but I can't make your solution work withsubprocess.Popen
: do you know why? -
adrin about 9 yearsIt works, but it's poorly formatted, and has a bunch of extra whitespaces and new lines. It'll be nice if you could fix them and give an example output.
-
mabahj about 9 yearsThat was strange. I've added an example output, I have no extra whitespaces and newlines here. But if your terminal window isn't wide enough, then it will wrap to a new line since the wish (as I understood it) was to not truncate the job names. It could be that the the solution is a bit fragile and does not work that well on other nix distros/sge versions? (I have SGE 8.1.4 and CentOS 5.11 and 6.6, bash 3.2, zsh 5.0.5)
-
Dima Lituiev almost 8 yearsGreat answer! Creating an alias takes some effort to escape quotation marks, so I post it here:
alias detqstat='qstat -xml | tr '"'"'\n'"'"' '"'"' '"'"' | sed '"'"'s#<job_list[^>]*>#\n#g'"'"' | sed '"'"'s#<[^>]*>##g'"'"' | grep " " | column -t'
-
antass over 5 yearsNice. I was able to create a clean alias without the necessity to escape characters by replacing double quotes with single quotes in the
grep
command:alias qstata="qstat -xml | tr '\n' ' ' | sed 's#<job_list[^>]*>#\n#g' | sed 's#<[^>]*>##g' | grep ' ' | column -t"
-
Jonno_FTW over 5 yearsIn what version was the introduced? The system we use has
SGE 6.2u5
, the man page forqstat
only listsSGE_LONG_QNAMES
-
Andrey Kuehlkamp over 5 years@Jonno_FTW: I used this on Univa Grid Engine (UGE) 8.5.0.
-
Cai about 5 yearsFor me, this didn't include the column headings like
qstat
alone does.