SQLAlchemy: print the actual query

Question

I'd really like to be able to print out valid SQL for my application, including values, rather than bind parameters, but it's not obvious how to do this in SQLAlchemy (by design, I'm fairly sure).

Has anyone solved this problem in a general way?

Solution 1

This works in python 2 and 3 and is a bit cleaner than before, but requires SA>=1.0.

from sqlalchemy.engine.default import DefaultDialect
from sqlalchemy.sql.sqltypes import String, DateTime, NullType

# python2/3 compatible.
PY3 = str is not bytes
text = str if PY3 else unicode
int_type = int if PY3 else (int, long)
str_type = str if PY3 else (str, unicode)


class StringLiteral(String):
    """Teach SA how to literalize various things."""
    def literal_processor(self, dialect):
        super_processor = super(StringLiteral, self).literal_processor(dialect)

        def process(value):
            if isinstance(value, int_type):
                return text(value)
            if not isinstance(value, str_type):
                value = text(value)
            result = super_processor(value)
            if isinstance(result, bytes):
                result = result.decode(dialect.encoding)
            return result
        return process


class LiteralDialect(DefaultDialect):
    colspecs = {
        # prevent various encoding explosions
        String: StringLiteral,
        # teach SA about how to literalize a datetime
        DateTime: StringLiteral,
        # don't format py2 long integers to NULL
        NullType: StringLiteral,
    }


def literalquery(statement):
    """NOTE: This is entirely insecure. DO NOT execute the resulting strings."""
    import sqlalchemy.orm
    if isinstance(statement, sqlalchemy.orm.Query):
        statement = statement.statement
    return statement.compile(
        dialect=LiteralDialect(),
        compile_kwargs={'literal_binds': True},
    ).string

Demo:

# coding: UTF-8
from datetime import datetime
from decimal import Decimal

from literalquery import literalquery


def test():
    from sqlalchemy.sql import table, column, select

    mytable = table('mytable', column('mycol'))
    values = (
        5,
        u'snowman: ☃',
        b'UTF-8 snowman: \xe2\x98\x83',
        datetime.now(),
        Decimal('3.14159'),
        10 ** 20,  # a long integer
    )

    statement = select([mytable]).where(mytable.c.mycol.in_(values)).limit(1)
    print(literalquery(statement))


if __name__ == '__main__':
    test()

Gives this output: (tested in python 2.7 and 3.4)

SELECT mytable.mycol
FROM mytable
WHERE mytable.mycol IN (5, 'snowman: ☃', 'UTF-8 snowman: ☃',
      '2015-06-24 18:09:29.042517', 3.14159, 100000000000000000000)
 LIMIT 1

Solution 2

In the vast majority of cases, the "stringification" of a SQLAlchemy statement or query is as simple as:

print(str(statement))

This applies both to an ORM Query as well as any select() or other statement.

Note: the following detailed answer is being maintained on the sqlalchemy documentation.

To get the statement as compiled to a specific dialect or engine, if the statement itself is not already bound to one you can pass this in to compile():

print(statement.compile(someengine))

or without an engine:

from sqlalchemy.dialects import postgresql
print(statement.compile(dialect=postgresql.dialect()))

When given an ORM Query object, in order to get at the compile() method we only need access the .statement accessor first:

statement = query.statement
print(statement.compile(someengine))

with regards to the original stipulation that bound parameters are to be "inlined" into the final string, the challenge here is that SQLAlchemy normally is not tasked with this, as this is handled appropriately by the Python DBAPI, not to mention bypassing bound parameters is probably the most widely exploited security holes in modern web applications. SQLAlchemy has limited ability to do this stringification in certain circumstances such as that of emitting DDL. In order to access this functionality one can use the 'literal_binds' flag, passed to compile_kwargs:

from sqlalchemy.sql import table, column, select

t = table('t', column('x'))

s = select([t]).where(t.c.x == 5)

print(s.compile(compile_kwargs={"literal_binds": True}))

the above approach has the caveats that it is only supported for basic types, such as ints and strings, and furthermore if a bindparam without a pre-set value is used directly, it won't be able to stringify that either.

To support inline literal rendering for types not supported, implement a TypeDecorator for the target type which includes a TypeDecorator.process_literal_param method:

from sqlalchemy import TypeDecorator, Integer


class MyFancyType(TypeDecorator):
    impl = Integer

    def process_literal_param(self, value, dialect):
        return "my_fancy_formatting(%s)" % value

from sqlalchemy import Table, Column, MetaData

tab = Table('mytable', MetaData(), Column('x', MyFancyType()))

print(
    tab.select().where(tab.c.x > 5).compile(
        compile_kwargs={"literal_binds": True})
)

producing output like:

SELECT mytable.x
FROM mytable
WHERE mytable.x > my_fancy_formatting(5)

Solution 3

Given that what you want makes sense only when debugging, you could start SQLAlchemy with echo=True, to log all SQL queries. For example:

engine = create_engine(
    "mysql://scott:tiger@hostname/dbname",
    encoding="latin1",
    echo=True,
)

This can also be modified for just a single request:

echo=False – if True, the Engine will log all statements as well as a repr() of their parameter lists to the engines logger, which defaults to sys.stdout. The echo attribute of Engine can be modified at any time to turn logging on and off. If set to the string "debug", result rows will be printed to the standard output as well. This flag ultimately controls a Python logger; see Configuring Logging for information on how to configure logging directly.

Source: SQLAlchemy Engine Configuration

If used with Flask, you can simply set

app.config["SQLALCHEMY_ECHO"] = True

to get the same behaviour.

Solution 4

We can use compile method for this purpose. From the docs:

from sqlalchemy.sql import text
from sqlalchemy.dialects import postgresql

stmt = text("SELECT * FROM users WHERE users.name BETWEEN :x AND :y")
stmt = stmt.bindparams(x="m", y="z")

print(stmt.compile(dialect=postgresql.dialect(),compile_kwargs={"literal_binds": True}))

Result:

SELECT * FROM users WHERE users.name BETWEEN 'm' AND 'z'

Warning from docs:

Never use this technique with string content received from untrusted input, such as from web forms or other user-input applications. SQLAlchemy’s facilities to coerce Python values into direct SQL string values are not secure against untrusted input and do not validate the type of data being passed. Always use bound parameters when programmatically invoking non-DDL SQL statements against a relational database.

Solution 5

So building on @zzzeek's comments on @bukzor's code I came up with this to easily get a "pretty-printable" query:

def prettyprintable(statement, dialect=None, reindent=True):
    """Generate an SQL expression string with bound parameters rendered inline
    for the given SQLAlchemy statement. The function can also receive a
    `sqlalchemy.orm.Query` object instead of statement.
    can 

    WARNING: Should only be used for debugging. Inlining parameters is not
             safe when handling user created data.
    """
    import sqlparse
    import sqlalchemy.orm
    if isinstance(statement, sqlalchemy.orm.Query):
        if dialect is None:
            dialect = statement.session.get_bind().dialect
        statement = statement.statement
    compiled = statement.compile(dialect=dialect,
                                 compile_kwargs={'literal_binds': True})
    return sqlparse.format(str(compiled), reindent=reindent)

I personally have a hard time reading code which is not indented so I've used sqlparse to reindent the SQL. It can be installed with pip install sqlparse.

View more solutions

Comments

[bukzor]

I don't see why the SA folk believe it's reasonable for such a simple operation to be so hard.

[Corey O.]

This is awesome ... Will have to add this to some debug libs so that we can easily access it. Thanks for doing the footwork on this one. I'm amazed that it had to be so complicated.

[bukzor]

I'm pretty sure that this is intentionally hard, because newbies are tempted to cursor.execute() that string. The principle of consenting adults is commonly used in python though.

[OrganicPanda]

Thank you! render_literal_value worked well for me. My only change was: return "%s" % value instead of return repr(value) in the float, int, long section because Python was outputting longs as 22L instead of just 22

[gsakkis]

This recipe (as well as the original) raises UnicodeDecodeError if any bindparam string value is not representable in ascii. I posted a gist that fixes this.

[Zitrax]

"STR_TO_DATE('%s','%%Y-%%m-%%d %%H:%%M:%%S')" % value.strftime("%Y-%m-%d %H:%M:%S") in mysql

[zzzeek]

@bukzor - I don't recall being asked if the above is "reasonable" so you can't really state that I "believe" it is - FWIW, it's not! :) please see my answer.

[bukzor]

We're stuck on 0.7 for now, so I still need to use my own answer.

[bukzor]

@gsakkis: As far as I can see, your fix uses str(), which uses the ascii codec. How does your change fix the issue?

[jmagnusson]

@zzzeek Why isn't pretty-printing queries included in sqlalchemy by default? Like query.prettyprint(). It eases the debugging pain with big queries immensely.

[zzzeek]

@jmagnusson because beauty is in the eye of the beholder :) There are ample hooks (e.g. cursor_execute event, Python logging filters, @compiles, etc.) for any number of third party packages to implement pretty-printing systems.

[zzzeek]

@buzkor re: limit that's been fixed in 1.0 bitbucket.org/zzzeek/sqlalchemy/issue/3034/…

[slashdottir]

Please show an example of how to use it? Thank you

[Piotr Dobrogost]

How do you know what the Python DBAPI driver for Oracle uses for bind params of datetime.datetime type? Why do you think it's TO_DATE()?

[wolfmanx]

Very nice indeed. I took the liberty and incorporated this into stackoverflow.com/a/42066590/2127439, which covers SQLAlchemy v0.7.9 - v1.1.15, including INSERT and UPDATE statements (PY2/PY3).

[Alfonso Embid-Desmet]

from file import render_query; print(render_query(query))

[Martin Thoma]

For me it was looking for from sqlalchemy.dialects import postgresql; print(query.statement.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True})))

[fougerejo]

That's the only example of this whole page that worked for me ! Thanks !

[Sekhar C]

very nice. but is it converting as below. 1) query(Table).filter(Table.Column1.is_(False) to WHERE Column1 IS 0. 2) query(Table).filter(Table.Column1.is_(True) to WHERE Column1 IS 1. 3) query(Table).filter(Table.Column1 == func.any([1,2,3])) to WHERE Column1 = any('[1,2,3]') above conversions are incorrect in syntax.

[janezj]

How to solve default value in insert statement? ``` python from sqlalchemy import MetaData, insert from sqlalchemy.schema import Column, Table meta = MetaData() TGR = Table('tbl', meta, Column('a'), Column('b'), Column('c', default='xxx') ) s = TGR.insert().values(a=1, b=2) print(literalquery(s)) ``` INSERT INTO tbl (a, b, c) VALUES (1, 2, :c)

[c8999c 3f964f64]

This is by far my favourite solution, but recently I've been getting the following error: __init__() got an unexpected keyword argument '_enums' Any idea why it can't compile the query?

[Talha Junaid]

from where long is imported?

[Samkit Jain]

Use int instead of long. long was deprecated in Python 3.

[Taras]

For dialect please refer to this page: docs.sqlalchemy.org/en/13/dialects

[Ping.Goblue]

this is the best answer.