Sqlite / SQLAlchemy: how to enforce Foreign Keys?

Question

The new version of SQLite has the ability to enforce Foreign Key constraints, but for the sake of backwards-compatibility, you have to turn it on for each database connection separately!

sqlite> PRAGMA foreign_keys = ON;

I am using SQLAlchemy -- how can I make sure this always gets turned on? What I have tried is this:

engine = sqlalchemy.create_engine('sqlite:///:memory:', echo=True)
engine.execute('pragma foreign_keys=on')

...but it is not working!...What am I missing?

EDIT: I think my real problem is that I have more than one version of SQLite installed, and Python is not using the latest one!

>>> import sqlite3
>>> print sqlite3.sqlite_version
3.3.4

But I just downloaded 3.6.23 and put the exe in my project directory! How can I figure out which .exe it's using, and change it?

Solution 1

I now have this working:

Download the latest sqlite and pysqlite2 builds as described above: make sure correct versions are being used at runtime by python.

import sqlite3   
import pysqlite2 
print sqlite3.sqlite_version   # should be 3.6.23.1
print pysqlite2.__path__       # eg C:\\Python26\\lib\\site-packages\\pysqlite2

Next add a PoolListener:

from sqlalchemy.interfaces import PoolListener
class ForeignKeysListener(PoolListener):
    def connect(self, dbapi_con, con_record):
        db_cursor = dbapi_con.execute('pragma foreign_keys=ON')

engine = create_engine(database_url, listeners=[ForeignKeysListener()])

Then be careful how you test if foreign keys are working: I had some confusion here. When using sqlalchemy ORM to add() things my import code was implicitly handling the relation hookups so could never fail. Adding nullable=False to some ForeignKey() statements helped me here.

The way I test sqlalchemy sqlite foreign key support is enabled is to do a manual insert from a declarative ORM class:

# example
ins = Coverage.__table__.insert().values(id = 99,
                                    description = 'Wrong',
                                    area = 42.0,
                                    wall_id = 99,  # invalid fkey id
                                    type_id = 99)  # invalid fkey_id
session.execute(ins) 

Here wall_id and type_id are both ForeignKey()'s and sqlite throws an exception correctly now if trying to hookup invalid fkeys. So it works! If you remove the listener then sqlalchemy will happily add invalid entries.

I believe the main problem may be multiple sqlite3.dll's (or .so) lying around.

Solution 2

For recent versions (SQLAlchemy ~0.7) the SQLAlchemy homepage says:

PoolListener is deprecated. Please refer to PoolEvents.

Then the example by CarlS becomes:

engine = create_engine(database_url)

def _fk_pragma_on_connect(dbapi_con, con_record):
    dbapi_con.execute('pragma foreign_keys=ON')

from sqlalchemy import event
event.listen(engine, 'connect', _fk_pragma_on_connect)

Solution 3

Building on the answers from conny and shadowmatter, here's code that will check if you are using SQLite3 before emitting the PRAGMA statement:

from sqlalchemy import event
from sqlalchemy.engine import Engine
from sqlite3 import Connection as SQLite3Connection

@event.listens_for(Engine, "connect")
def _set_sqlite_pragma(dbapi_connection, connection_record):
    if isinstance(dbapi_connection, SQLite3Connection):
        cursor = dbapi_connection.cursor()
        cursor.execute("PRAGMA foreign_keys=ON;")
        cursor.close()

Solution 4

From the SQLite dialect page:

SQLite supports FOREIGN KEY syntax when emitting CREATE statements for tables, however by default these constraints have no effect on the operation of the table.

Constraint checking on SQLite has three prerequisites:

• At least version 3.6.19 of SQLite must be in use
• The SQLite libary must be compiled without the SQLITE_OMIT_FOREIGN_KEY or SQLITE_OMIT_TRIGGER symbols enabled.
• The PRAGMA foreign_keys = ON statement must be emitted on all connections before use.

SQLAlchemy allows for the PRAGMA statement to be emitted automatically for new connections through the usage of events:

from sqlalchemy.engine import Engine
from sqlalchemy import event

@event.listens_for(Engine, "connect")
def set_sqlite_pragma(dbapi_connection, connection_record):
    cursor = dbapi_connection.cursor()
    cursor.execute("PRAGMA foreign_keys=ON")
    cursor.close()

Solution 5

As a simpler approach if your session creation is centralised behind a Python helper function (rather than exposing the SQLA engine directly), you can just issue session.execute('pragma foreign_keys=on') before returning the freshly created session.

You only need the pool listener approach if arbitrary parts of your application may create SQLA sessions against the database.

View more solutions

Comments

[Nick Perkins]

I already have SQLite 3.6.23 and pysqlite 2.6.0 ( and new SQLAlchemy ) The SQLite doc says that you must explicitly turn on FK enforcement. In your experience, when it did enforce, did you use that PRAGMA thing?

[Nick Perkins]

Thanks -- I tried the PoolListener, and it did allow me to execute the pragma for every database connection! Perfect! ...except that the pragma still does not work! The SQLite engine still does not enforce foreign keys!...I am still missing a piece of the puzzle. Maybe it's because I am on Windows? The SQLite docs say something about the "build options" that it was built with...but I just got the standard install for Windows...not sure if that matters?

[Nick Perkins]

Thanks, I got it working too. Indeed, the problem was multiple copies of SQLite on my machine...fixing that, and using the PoolListener have worked perfectly!

[David Parmenter]

conny's answer is perfect for newer versions of sqlalchemy. Use it! Moderator should really pick this one as correct.

[Matthew Moisen]

Thanks. This works for those of us who prefer the db = SQLAlchemy(app) approach as well.

[Steven]

This is a good and easy solution when controlling your session with a @contextmanager.

[toto_tico]

this also does the magic for pandas.to_sql as well, just copy it at the begining of the file that creates the session...

[Ron Kalian]

Thank you, this is the one to use. The reply by @CarlS (which I appreciate is from 2010) uses stuff that has now been deprecated (looking at SQLAlchemy v1.3) and hence does not work anymore.

[code_dredd]

Is there a way to do this in a backend-agnostic way? I use SQLite3 for development but intend to use PostgreSQL for production. Is there a way to know whether to execute the pragma or not, i.e. to check if you've connected to a PostgreSQL or SQLite back-end?

[amucunguzi]

If using Flask, you can call db.session.execute('pragma foreign_keys=on') before the queries in which you want FK constraint to be enforced. Works like a charm.

[Chen Lizi]

YES. This works. Foreign key constraint is now properly enforced by sqlite. Thank you!