Disable boto logging without modifying the boto files

Question

I am using the Boto library to talk to AWS. I want to disable logging. (Or redirect to /dev/null or other file). I can't find an obvious way to do this. I tried this, but that doesn't seem to help:

import boto
boto.set_file_logger('boto', 'logs/boto.log')

This says it is possible, http://developer.amazonwebservices.com/connect/thread.jspa?messageID=52727&#52727 but as far as I know the documentation doesn't tell how.

Solution 1

You could try

import logging
logging.getLogger('boto').setLevel(logging.CRITICAL)

which will suppress all (other than CRITICAL) errors.

Boto uses logging configuration files (e.g. /etc/boto.cfg, ~/.boto) so see if you can configure it to your needs that way.

The set_file_logger call simply adds a user-defined file to the logging setup, so you can't use that to turn logging off.

Solution 2

I move the boto3 answer from the comments (namely charneykaye and gene_wood) to a proper answer:

import logging

logger = logging.getLogger()
logger.addHandler(logging.StreamHandler()) # Writes to console
logger.setLevel(logging.DEBUG)
logging.getLogger('boto3').setLevel(logging.CRITICAL)
logging.getLogger('botocore').setLevel(logging.CRITICAL)
logging.getLogger('s3transfer').setLevel(logging.CRITICAL)
logging.getLogger('urllib3').setLevel(logging.CRITICAL)

import boto3

s3 = boto3.resource('s3')

for bucket in s3.buckets.all():
    print(bucket.name)

To get all the loggers follow the response from leobarcellos:

import logging
loggers_dict = logging.Logger.manager.loggerDict

Solution 3

Better yet, disable propagate for boto:

import boto
boto.set_file_logger('boto', 'logs/boto.log')
logging.getLogger('boto').propagate = False

Solution 4

This is the only solution, which works for me as of today (2020/01/31):

for name in ['boto', 'urllib3', 's3transfer', 'boto3', 'botocore', 'nose']:
    logging.getLogger(name).setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)

The solution with

boto3.set_stream_logger('', logging.CRITICAL)

was killing my whole non-boto logs. It manipulates the root logger of the standard logging from python.

Try it out for yourself:

import logging
import boto3
import sys
logger = logging.getLogger(__name__)
boto3.set_stream_logger('', logging.CRITICAL)
logging.basicConfig(level=logging.DEBUG, stream=sys.stdout,
                    format='%(asctime)s - %(levelname)s - %(message)s')


if __name__ == '__main__':
    s3_client = boto3.client('s3')
    response = s3_client.list_buckets()
    logger.info(f'bucket list: {response}')

Regardless of where the init of the logger happens, it won't bring up the output. Remove the line of the boto3.set_stream_logger('', logging.CRITICAL) and the non-boto3 logs will re-appear again! Consequently the only working solution is NOT to use the approach with boto3.set_stream_logger() and apply it as I suggested.

Solution 5

This answer is for those who're using logging.config.dictConfig.

It is recommended to disable DEBUG and INFO messages from all external packages, not limited to botocore and boto3:

LOGGING_CONFIG = { # Add your preexisting logging config here.
    "loggers": { # Add your preexisting loggers here.
        "": {"level": "WARNING", "handlers": ["console"], "propagate": False},  # Root logger.
     }

Alternatively, to disable debug messages from botocore and boto3 but not from all external packages:

LOGGING_CONFIG = { # Add your preexisting config here too.
    "loggers": { # Add your preexisting loggers here too.
        "botocore": {"level": "WARNING", "handlers": ["console"], "propagate": False},
        "boto3": {"level": "WARNING", "handlers": ["console"], "propagate": False},
     }

Assuming your logging configuration dict is named LOGGING, run this next:

logging.config.dictConfig(LOGGING)

The above must be run before boto3 is imported, irrespective of whether it is imported directly or indirectly! It won't entirely work if it's run after boto3 is already imported. You can choose to replace "WARNING" above with "INFO" or "ERROR" or "CRITICAL".

View more solutions

Comments

[michela]

Thanks for this. For some reason I can't get the boto config files to switch off logging. code.google.com/p/boto/issues/detail?id=476

[dashesy]

In django setting this is like this: LOGGING['loggers'].update({ 'boto': { 'level': 'CRITICAL', } })

[Charney Kaye]

Using boto3, I had to logging.getLogger('botocore').setLevel(logging.INFO)

[Gerard]

I used; logging.getLogger('boto3').setLevel(logging.CRITICAL)

[gene_wood]

As @charneykaye and @Gerard mention above, because boto3 is actually two distinct modules, boto3 and botocore, you'll need to suppress both with two calls like this : logging.getLogger('botocore').setLevel(logging.CRITICAL) and logging.getLogger('boto3').setLevel(logging.CRITICAL))

[kadrach]

I'm not sure whether this is a new-ish feature, but this should be the accepted answer nowadays.

[Dmitry Verhoturov]

Here are all boto3 loggers: boto3, botocore, nose.

[Vinay Sajip]

@DmitryVerhoturov - not sure why you classify nose as a boto logger - nose is an independent project which just happens to be used for testing of boto, right?

[Dmitry Verhoturov]

@VinaySajip, mindless copypaste, sorry. Just checked logging.Logger.manager.loggerDict.keys() with only boto3 imported, bcdocs, boto3, botocore, ndg are root level loggers which are used.

[hamx0r]

Most logging i'm seeing from "boto" is really from s3transfer as noted in @Iony's list. (boto 1.9.28 and s3transfer 0.1.13). Note that s3transfer doesn't load until boto3 is used (ie doesn't load when boto3 is loaded) so one should import s3transfer before disabling s3transfer loggers

[mchlfchr]

if you are using that, then all the other relevant logs (!= boto*) are disappearing as well

[Yannick Schuchmann]

why do they disappear?

[mchlfchr]

see my extended answer: stackoverflow.com/a/60008283/1654284

[Yannick Schuchmann]

I still don't understand why python's root logger is touched by that. But your example shows. Good job, man :) upvote.

[jake]

I had to use logging.getLogger("botocore").propagate = False.

[Joe W]

Working on an app and botocore killed the logging I was looking to remove.

[Big McLargeHuge]

Why is this "better"?

[Doug Harris]

If I recall correctly (and I wrote this 5+ years ago), by setting propagate to False it prevents it from being handled by other loggers. The second line in my example says "for the 'boto' logger, direct messages to logs/boto.log" and the third line says "...and then stop, don't pass it on to any other log handler"

[ybl]

Probably the cleanest option, considering the logging config will just work for all packages relying on the logging module.

[eduardosufan]

I don't understand why this works. If the level is DEBUG and all the boto3 logs are CRITICAL, why these are not appearing in the logs anymore?

[lony]

DEBUG is the default and the boto3 log level is changed in the way that only errors critical and higher are presented, as the default boto3 messages are lower - I assume info or debug - they are no longer shown. Did that help?