logger cannot find file

Question

I am getting this error when I put this input command:

$ python3.4 cron_e2e.py -f test_web_events -E ctg-clickstream testbrad

error:

$ python3.4 cron_e2e.py -f test_web_events -E ctg-clickstream testbrad
Traceback (most recent call last):
  File "cron_e2e.py", line 421, in <module>
    sys.exit(main(sys.argv))
  File "cron_e2e.py", line 368, in main
    log.addHandler(logging.FileHandler(logfile))
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/logging/__init__.py", line 1006, in __init__
    StreamHandler.__init__(self, self._open())
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/logging/__init__.py", line 1030, in _open
    return open(self.baseFilename, self.mode, encoding=self.encoding)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/bli1/Development/QE/chun-qe-trinity-functional/qe/tests/qe/logs/testbrad_8_21_2015_cron.log'

The error occurs at this line:

log.addHandler(logging.FileHandler(logfile))

Not quite sure where my code isnt creating the log file for me

code:

def main(argv):
    "Test"
    exit_code = 0
    global me; me = os.path.basename(argv[0]) # name of this program
    global mydir; mydir = os.path.dirname(os.path.abspath(__file__))
    parser = argparse.ArgumentParser(description=main.__doc__)
    parser.add_argument("-f", "--functional_test", metavar="FUNCTEST",
                        dest="functest", help="Type of functional test")
    parser.add_argument("-p","--phase", action="append",
                        metavar="POST|HDFS|HIVE|ALL", dest="phase",
                        help="phase of test to run")
    parser.add_argument("-t", "--testfile", metavar="TESTFILE",
                        dest="testfile", default=os.path.join(mydir, "resources/cron.json"),
                        help="file with test commands")
    parser.add_argument("-E","--Event", metavar="EVENTNAME",
                        dest="event_names", action="append",
                        help="[repeatable] topic to which to post")
    parser.add_argument("runtag",
                        metavar="RUNTAG",
                        help="run tag for this run")
    args = parser.parse_args(args=argv[1:])  # will exit on parse error
    log = logging.getLogger(me)
    log.setLevel(logging.INFO)
    dt = datetime.datetime.utcnow()
    logfile = "qe/logs/" + (args.runtag + "_{}_{}_{}".format(dt.month, dt.day, dt.year) + "_cron.log")

    pdb.set_trace()
    if isinstance(logfile, str):
        if os.path.exists(logfile):
            os.remove(logfile)
        # logging.FileHandler() returns FileHandler class, the file is opened and used as the stream for logging
        log.addHandler(logging.FileHandler(logfile))
    console = logging.StreamHandler(sys.stderr); console.setLevel(logging.WARNING); log.addHandler(console)

Solution 1

The issue is that the directory in which you have defined the log file to go to - "qe/logs/" - does not exist.

By default the mode for FileHandler is 'a' , which means that if the log file exists it would open it and start appending to it at the end, otherwise it would create the file.

But the FileHandler would only create the log file if it does not exist , it would not create the directory if that does not exist, if the directory does not exist, it would throw an error like you got.

You would need to create the directory yourself , either manually or programatically. If you want to create the directory programatically, you can use - os.makedirs() - which would -

create all intermediate-level directories needed to contain the leaf directory.

Example -

import os, os.path
if not os.path.exists("qe/logs/"):
    os.makedirs("qe/logs/")

Solution 2

anand-s-kumar answer is great, but there is now a shorter alternative solution, using pathlib:

import pathlib

pathlib.Path('qe/logs/').mkdir(parents=True, exist_ok=True)

Comments

[301_Moved_Permanently]

And you are sure that the path in the last line of the traceback exists? Don't you want to have logfile = '../logs/' + ..... instead?