AWS Elastic Beanstalk logging with python (django)

Question

How do you manage your application logs in AWS elastic beanstalk? Which file you write you application logs to?

I'm using the following Logging configuration in my development environment but this doesn't work when I deploy in AWS.

DEBUG_LOG_DIR = BASE_DIR + "/django_debug.log"
LOGGING = {
    'version': 1,
    'disable_existing_loggers': True,
    # How to format the output
    'formatters': {
        'standard': {
            'format' : "[%(asctime)s] %(levelname)s [%(name)s:%(lineno)s] %(message)s",
            'datefmt' : "%d/%b/%Y %H:%M:%S"
        },
    },
    # Log handlers (where to go)
    'handlers': {
        'null': {
            'level':'DEBUG',
            'class':'django.utils.log.NullHandler',
        },
        'log_file': {
            'level':'DEBUG',
            'class':'logging.handlers.RotatingFileHandler',
            'filename': DEBUG_LOG_DIR,
            'maxBytes': 50000,
            'backupCount': 2,
            'formatter': 'standard',
        },
        'console':{
            'level':'INFO',
            'class':'logging.StreamHandler',
            'formatter': 'standard'
        },
        'mail_admins': {
            'level': 'ERROR',
            'class': 'django.utils.log.AdminEmailHandler',
        },
    },
    # Loggers (where does the log come from)
    'loggers': {
        'repackager': {
            'handlers': ['console', 'log_file'],
            'level': 'DEBUG',
            'propagate': True,
        },
        'django': {
            'handlers':['console'],
            'propagate': True,
            'level':'WARN',
        },
        'django.db.backends': {
            'handlers': ['console', 'log_file'],
            'level': 'WARN',
            'propagate': False,
        },
        '': {
            'handlers': ['console', 'log_file'],
            'level': 'DEBUG',
        },
    }
}

Solution 1

Ok, I figured out a way to do it.

First I connected via ssh to ec2 machine, then I create a folder in /var/log called app_logs with root user:

mkdir /var/log/app_logs

After that I did the follow:

cd /var/log/
chmod g+s app_logs/
setfacl -d -m g::rw app_logs/
chown wsgi:wsgi app_logs/

That ensures that all the files created in this folder will have wsgi as owner and will be writable for the group that the file belongs. I had to do that because I noticed that the log file created by django app had root as owner and owner group but the application runs through wsgi user.

Finally I changed DEBUG_LOG_DIR to /var/log/app_logs/django_debug.log

Solution 2

I had a similar issue but on Elastic Beanstalk, so I created a config file (e.g. applogs.config) in .ebextensions folder of the app. This creates the app-logs folder if it is not there already and sets the file permissions and owner so that the app can write its logs there.

commands:
  00_create_dir:
    command: mkdir -p /var/log/app-logs
  01_change_permissions:
    command: chmod g+s /var/log/app-logs
  02_change_owner:
    command: chown wsgi:wsgi /var/log/app-logs

Finally, in your Django settings:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'file': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': '/var/log/app-logs/django.log',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'DEBUG',
            'propagate': True,
        },
    },
}

Aditionally, if you want your log to be accessible from beanstalk logs using the web, add this to your file in .ebextensions

files:
  "/opt/elasticbeanstalk/tasks/taillogs.d/django.conf":
    mode: "000755"
    owner: root
    group: root
    content: |
      /var/log/app-logs/django.log

Solution 3

There's a simple way that doesn't require any beanstalk configuration.

In your django settings under LOGGING set up a handler directed to the file '/opt/python/log/{log_file_name}'. The logs can then be accessed via the beanstalk environment menu under "Logs".

LOGGING = {
    ...,
    'handlers': {
        'logfile': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/opt/python/log/{log_file_name}',
        },
    },
    'loggers': {
        'debugger': {
            'level': 'DEBUG',
            'handlers': ['logfile'],
        'propagate': False,
    },
}

This location is stated in the documentation here:

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.logging.html#health-logs-instancelocation

Solution 4

-- edit --

This answer was originally written for Amazon Linux AMI, which has now reached end-of-life.

To keeps things clear and separate, I wrote a new answer for Amazon Linux 2.

-- original answer --

Summary

The simplest solution, in my opinion, is to log to the /opt/python/log folder, as suggested by bewestphal and by @thierry-j (under steve-dunlop's answer).

That is also what the official AWS EB Python Sample Application does: see python-v1.zip source

The log file will then be included automatically when you request logs from EB.

This solution works out-of-the-box, without any modification of .ebextensions, as long as you do not call django-admin.py (or other django code) in your .ebextensions.

However, most apps do need to call django-admin.py in .ebextensions, e.g. in order to migrate. That will cause the log file to be created prematurely, with root owner and root group. This leads to permission errors, because the app runs as wsgi:wsgi.

This can be fixed by adding a new command, at the end of your container_commands, to remove the "premature" log file, e.g.:

container_commands:
  ...
  9999_remove_root_log_file:
    command: rm /opt/python/log/django.log
    ignoreErrors: true

Details below.

Background

On a standard pre-configured Amazon Linux/Python platform, which uses Apache with mod_wsgi (see AWS platform docs), the WSGIDaemonProcess for the Django app runs as user wsgi and group wsgi (see /etc/httpd/conf.d/wsgi.conf on your EC2 instance).

In addition, the default folder permissions for the /opt/python/log folder (on my standard EC2 instance) are: drwxrwxr-x 3 root wsgi 4096 Mar 5 14:08 .

That is, the wsgi group has all permissions (rwx), so the Django app (group wsgi) can create log files there.

This works, out-of-the-box, as demonstrated by the official AWS EB Python Sample Application (python-v1.zip).

However, if you do anything in your .ebextensions that causes the logging file-handler to be initialized (like calling django-admin.py), it will break.

Permission issues

Here's how using django-admin.py in .ebextensions breaks your log file permissions:

Elastic Beanstalk container_commands, in .ebextensions, are executed as the root user (see aws docs).

If you call django-admin.py in any of the container_commands, e.g. with collectstatic or migrate, that will cause your logging file handler(s) to be initialized. If the specified log file does not exist yet, at that time, it will be created, with root owner and root group.

That means the Django app, running as part of the wsgi group, will not have permission to write to the log file (which belongs to the root group).

This leads to permission errors, e.g.: PermissionError: [Errno 13] Permission denied: '/opt/python/log/django.log'

How to reproduce

The following snippet illustrates the permissions issue and shows how to fix it.

To reproduce the issue, add these container_commands to a clean project (e.g. following the AWS EB Django tutorial), configure Django settings.py to log to /opt/python/log/django.log, deploy to AWS EB, then check the eb-activity.log to see the output of the container commands.

...

container_commands:
  0100_show_current_user:
    # show that we are running as root user
    command: whoami
  0200_try_to_remove_log_file:
    # we need a clean slate for this example (make sure no log file owned by wsgi is present)
    command: rm /opt/python/log/django.log
    ignoreErrors: true
  0300_break_log_file_permissions:
    # this causes a new log file to be created, owned by root:root (instead of wsgi:wsgi)
    command: django-admin.py
  0400_show_log_file_permissions:
    # prove that a log file was created by root, and show folder permissions
    command: ls -la /opt/python/log
  0500_fix_by_removing_log_file_after_all_django_admin_calls:
    # remove the log file created by django-admin.py, to ensure that a new log file will  
    # be created when the server starts, owned by wsgi:wsgi
    command: rm /opt/python/log/django.log
    ignoreErrors: true

DRY solution

So, there is no need to mess with file/folder permissions explicitly.

If you don't call django code in .ebextensions, logging to /opt/python/log works, out-of-the-box.

If you do call django code in .ebextensions, e.g. django-admin.py collectstatic, simply remove the log file at the end of your container_commands section.

NOTE: If you want to log files to persist between deployments, only remove them if they are owned by root.

Here's a DRY example:

In .ebextensions config:

option_settings:
  # create EB environment property for the log file path
  aws:elasticbeanstalk:application:environment:
    LOG_FILE_PATH: /opt/python/log/django.log
...

container_commands:
  ...
  # django code called here, e.g. "django-admin.py collectstatic"
  ...
  9999_remove_any_existing_django_log_files:
    command: rm $LOG_FILE_PATH      
    ignoreErrors: true

and in settings.py:

...
# get log path from environment variable, with fallback for local development
log_file_path = os.getenv('LOG_FILE_PATH', 'local.log')
# use this as 'filename' for the file handler, as described in the other answers
...

Solution 5

This answer is for Amazon Linux 2 only. For those who have not migrated yet, please see my old answer for Amazon Linux AMI.

Background

The official AWS Python sample-application for Amazon linux 2 uses the /tmp folder for logging.

However, custom log files added to /tmp are not included automatically when requesting logs from Elastic Beanstalk. To include custom log files, we need to create logging tasks in subfolders of /opt/elasticbeanstalk/tasks on the EC2 instance. See instructions in the documentation.

The sample app (source) accomplishes this using .ebextensions. However, the AWS Linux 2 migration docs suggest we should use .platform hooks instead:

We recommend using platform hooks to run custom code on your environment instances. You can still use commands and container commands in .ebextensions configuration files, but they aren't as easy to work with. For example, writing command scripts inside a YAML file can be cumbersome and difficult to test.

This has the additional advantage that output from platform hooks is collected in a separate log file, viz. /var/log/eb-hooks.log, which makes debugging a bit easier.

DRY logging setup for a basic Django app on Amazon Linux 2

Log level and log path are defined in one place, as Elastic Beanstalk environment properties, e.g. in .ebextensions/options.config:

option_settings:
  aws:elasticbeanstalk:application:environment:
    LOG_LEVEL: INFO
    DJANGO_LOG_FILE_PATH: /tmp/django-app.log
    ...

The DJANGO_LOG_FILE_PATH environment property can now be used in a platform hook to create logging tasks:

.platform/hooks/postdeploy/020_create_logging_tasks.sh

#!/bin/bash
TASKS_DIR=/opt/elasticbeanstalk/tasks
# include all app log files in bundle logs (replaces ".log" by "*")
echo "${DJANGO_LOG_FILE_PATH//.log/*}" > "$TASKS_DIR/bundlelogs.d/01-app-log.conf"
# include current app log file in tail logs
echo $DJANGO_LOG_FILE_PATH > "$TASKS_DIR/taillogs.d/01-app-log.conf"

Note that the platform hooks require execution permission, e.g. chmod +x 020_create_logging_tasks.sh. On windows you can use git as described here.

To prevent permission issues, we use another platform hook to ensure that the log file is always owned by webapp. Note this hook runs before the logging-tasks hook:

.platform/hooks/postdeploy/010_create_log_file.sh

#!/bin/bash

if test -f "$DJANGO_LOG_FILE_PATH";
then
  echo "$DJANGO_LOG_FILE_PATH exists"
else
  # create log file
  touch $DJANGO_LOG_FILE_PATH
fi

# set log file owner (we are currently "root", but the app runs as "webapp")
chown webapp:webapp $DJANGO_LOG_FILE_PATH

We also use the LOG_LEVEL and DJANGO_LOG_FILE_PATH environment properties in our Django settings:

settings.py

...
# basic logging with file rotation ()
log_level = os.getenv('LOG_LEVEL', 'INFO')
handlers = dict(file={'class': 'logging.handlers.TimedRotatingFileHandler',
                      'filename': os.getenv('DJANGO_LOG_FILE_PATH'),
                      'when': 'midnight',
                      'interval': 1,
                      'backupCount': 1,
                      'encoding': 'utf-8'})
loggers = dict(django=dict(level=log_level, handlers=['file']),
               myapp=dict(level=log_level, handlers=['file']))
LOGGING = dict(version=1,
               disable_existing_loggers=False,
               handlers=handlers,
               loggers=loggers)
...

Some notes:

• We normally specify custom formatters as well, but I left those out for clarity.

• The application itself can now be found on the EC2 instance in /var/app/current. Also see extending EB Linux platforms for more details.

• The application now runs as webapp with group webapp.

• eb ssh is your friend. See docs.

• We also use platform hooks to run Django's migrate and collectstatic commands, as described here.

EDIT:

As pointed out by @hax0 in the comments, file permission issues may arise if you try to run manage.py commands on the EC2 instance, using SSH, after deployment.

For example, when using eb ssh, you are logged in as ec2-user, but the log file is owned by the webapp user, and, by default, only the owner has write permission (644). Thus, when running python manage.py as ec2-user, you will get an error saying it cannot configure the log file handler because permission is denied.

A quick & dirty workaround is to change file permissions temporarily, e.g. using

sudo chmod 646 /tmp/django-app.log

Another workaround would be to run manage.py as the webapp user, for example like so:

sudo su - webapp <<'EOF'
source $(find /var/app/venv/*/bin/activate)
export $(/opt/elasticbeanstalk/bin/get-config --output YAML environment | 
         sed -r 's/: /=/' | xargs)
python3 /var/app/current/manage.py showmigrations
EOF

View more solutions

Comments

[linqu]

in my setup id had to use another user: command: chown wsgi:wsgi /var/log/app-logs

[Thierry J.]

No, Django server errors do not show up in this file.

[igorsantos07]

they might end up there, but that's only on some cases.

[James Parker]

This should be the correctly accepted answer. If you rebuild the instance with the accepted answer you'll have to re-create the file every time.

[James Parker]

Keep in mind that if you ever need to rebuild your instance this will need to be performed again manually. This will also not be applied to autoscaled instances.

[Thierry J.]

It is actually better to use /opt/python/log/ for the logs. The directory already exists, no need to create it, and it's packaged when getting log bundle or saving them to cloudwatch.

[Ricardo Jacas]

And how, exactly, do you manage to give it permissions for it?

[James Parker]

I agree that this is the ideal solution. But When I try to implement this I'm getting an error stating that permission is denied to the generated log file (django.log). If I SSH into the box and chmod 777 the django.log file it works fine. However that is not an acceptable solution.

[jmwicks]

As suggested in another answer, I added the following to avoid one final permission error: 03_change_default_owner: command: setfacl -d -m g::rw /var/log/app-logs

[ekauffmann]

@jmwicks could you improve your comment by explaining the use of this command and the flags you chose, please?

[Eric Darchis]

@jmwicks is right. If you don't use that command, the log file will be created with user root and group wsgi (thanks to chmod g+s) but umask 022 will force the file not to be group-writable. One option would be to force os.umask in the django config but the setfacl does this externally. -m g::rw means that the files created in that folder will be group writable, regardless of umask.

[bewestphal]

If you're running into a permission issue like that, one way is to make an EC2 launch script with write permissions

[Oded]

I beat my head for hours on why this didn't work for me. I was getting: ValueError: Unable to configure handler 'file': [Errno 13] Permission denied: '/var/log/app-logs/django.log' It turns out it was because the log file already existed, and I needed to make my chmod and chown recursive. Hope this helps someone.

[Christopher Lewis]

Which makes the first sentence of the answer, "There's a simple way that doesn't require any beanstalk configuration", incorrect.

[djvg]

@ThierryJ.: Your suggestion of using /opt/python/log sounds good. That's also what the AWS EB Python Sample Application does: see python-v1.zip on aws docs. Note that this works for the sample app without any permissions customization.

[djvg]

See this discussion on AWS forums.

[djvg]

@JamesParker: possible cause for your permission error below.

[djvg]

PLEASE NOTE: This works (for me) on Amazon Linux AMI, but I haven't tested it (yet) on Amazon Linux 2

[Oded]

It does not work on Amazon Linux 2 out of the box. Possibly because there's no /opt/python/log (or /opt/python for that matter). I'm still trying to see where the recommended place for logs are. I will probably fall back on /var/log/, creating an app-logs subfolder.

[Oded]

Note for Amazon Linux 2 with gunicorn, the user and group changed from wsgi:wsgi to webapp:webapp

[djvg]

@Oded: According to the AWS docs about migrating to Amazon Linux 2, the application now resides in /var/app/current. See the Python section.

[Oded]

Got it. I actually like not having my logs clobbered when the app is reinstalled, so I can compare what happened in a previous version when debugging a problem.

[djvg]

@Oded: I'm not sure if there is an alternative for /opt/python/log that is equally convenient. I could not find any recommendations in the AWS docs, but the official Python sample application for EB on Amazon Linux 2 now logs to the /tmp folder, and uses custom logging tasks to include these logs. See updated answer for details.

[Nick Brady]

Does AL2 have a similar location?

[Hax0]

Deployment works but when we ssh into the instance and try to python manage.py shell, it gives permission error on /tmp/django-app.log file

[djvg]

@hax0: Yes, I can confirm this. It happens because you are ec2-user, whereas the log file belongs to webapp (user and group). By default, only the log file owner (i.e. webapp) has write permission (644). If this is incidental, a quick & dirty workaround is to change file permissions temporarily, e.g. sudo chmod 646 /tmp/django-app.log. If it is a structural thing, you could do something similar in a .platform hook. Do keep in mind any security implications. A better alternative might be to try to run the command as webapp.

[djvg]

@hax0: manage.py shell works for me after changing file permissions. I did manage to run other manage.py commands as the webapp user, but could not get manage.py shell to work that way. See edit in answer.

[Josh]

You can just write the commands in your .ebextensions as a command to execute before application starts to make sure it is always on

[Josh]

I notice the default logging is set to require_debug_true. What if you just removed that filter? docs.djangoproject.com/en/dev/ref/logging/…

[Brad Solomon]

@djvg it looks like your comment there on /opt/python/log may be outdated as the latest Python sample ZIP uses LOG_FILE = '/tmp/sample-app.log' in application.py.

[Brad Solomon]

Using this exact solution on Amazon Linux 2, deployment fails (Gunicorn + Django + Nginx) with a Permission denied: '/tmp/django-app.log'. Perhaps that is because of the manage.py commands being run in container_commands.

[Brad Solomon]

django-app.log appears to be mode 0600, owned by root:root. I believe that's because some other command must be run by the root user, which generates that file as owned by root. (Again, assuming that's the stuff in container_commands but maybe something else.)

[djvg]

@BradSolomon On Amazon Linux 2 we do not use any container_commands at all: we use platform hooks to run manage.py etc. If you do call manage.py in the old container_commands, I expect that could indeed lead to permission issues similar to the ones described in my other answer (for Amazon Linux "1").

[Brad Solomon]

Thanks, yes; once I combined the knowledge that you shared from those two answers, that made sense, and was fixed by a final rm call in container_commands to remove the root-owned log file so that it could be regenerated by the non-root user.

[djvg]

@BradSolomon: I updated the answer with an example of another platform hook that ensures webapp always owns the log file. That should prevent permission issues.