How would you do the equivalent of preprocessor directives in Python?

Question

Is there a way to do the following preprocessor directives in Python?

#if DEBUG

< do some code >

#else

< do some other code >

#endif

Solution 1

There's __debug__, which is a special value that the compiler does preprocess.

if __debug__:
  print "If this prints, you're not running python -O."
else:
  print "If this prints, you are running python -O!"

__debug__ will be replaced with a constant 0 or 1 by the compiler, and the optimizer will remove any if 0: lines before your source is interpreted.

Solution 2

I wrote a python preprocessor called pypreprocessor that does exactly what you're describing.

The source and documentation is available on GitHub.

The package can also be downloaded/installed through the PyPI.

Here's an example to accomplish what you're describing.

from pypreprocessor import pypreprocessor

pypreprocessor.parse()

#define debug

#ifdef debug
print('The source is in debug mode')
#else
print('The source is not in debug mode')
#endif

pypreprocessor is capable of a lot more than just on-the-fly preprocessing. To see more use case examples check out the project on Google Code.

Update: More info on pypreprocessor

The way I accomplish the preprocessing is simple. From the example above, the preprocessor imports a pypreprocessor object that's created in the pypreprocessor module. When you call parse() on the preprocessor it self-consumes the file that it is imported into and generates a temp copy of itself that comments out all of the preprocessor code (to avoid the preprocessor from calling itself recursively in an infinite loop) and comments out all of the unused portions.

Commenting out the lines is, as opposed to removing them, is necessary to preserve line numbers on error tracebacks if the module throws an exception or crashes. And I've even gone as far as to rewrite the error traceback to report reflect the proper file name of the module that crashed.

Then, the generated file containing the postprocessed code is executed on-the-fly.

The upside to using this method over just adding a bunch of if statements inline in the code is, there will be no execution time wasted evaluating useless statements because the commented out portions of the code will be excluded from the compiled .pyc files.

The downside (and my original reason for creating the module) is that you can't run both python 2x and python 3x in the same file because pythons interpreter runs a full syntax check before executing the code and will reject any version specific code before the preprocessor is allowed to run ::sigh::. My original goal was to be able to develop 2x and 3x code side-by-side in the same file that would create version specific bytecode depending on what it is running on.

Either way, the preprocessor module is still very useful for implementing common c-style preprocessing capabilities. As well as, the preprocessor is capable of outputting the postprocessed code to a file for later use if you want.

Also, if you want to generate a version that has all of the preprocessor directives as well as any of the #ifdefs that are excluded removed it's as simple as setting a flag in the preprocessor code before calling parse(). This makes removing unwanted code from a version specific source file a one step process (vs crawling through the code and removing if statements manually).

Solution 3

I suspect you're gonna hate this answer. The way you do that in Python is

# code here
if DEBUG:
   #debugging code goes here
else:
   # other code here.

Since python is an interpreter, there's no preprocessing step to be applied, and no particular advantage to having a special syntax.

Solution 4

You can use the preprocessor in Python. Just run your scripts through the cpp (C-Preprocessor) in your bin directory. However I've done this with Lua and the benefits of easy interpretation have outweighed the more complex compilation IMHO.

Solution 5

You can just use the normal language constructs:

DEBUG = True
if DEBUG:
  # Define a function, a class or do some crazy stuff
  def f():
    return 23
else:
  def f():
    return 42

View more solutions

Comments

[Greg Hewgill]

Being an interpreter doesn't have anything to do with it. Nobody claims Java is interpreted, yet Java uses exactly the same technique (the D language is another example). Python is in fact a compiler, it compiles source code to bytecode and executes it in a VM, just like Java.

[intrepion]

I agree, that sounds a lot more trouble than its worth!

[user1066101]

@Greg Hewgill: There's no value in preprocessor directives to finesse things like static type declarations or conditional code, since the one doesn't exist and the other doesn't represent a significant cost.

[intrepion]

Wow this definitely answers my question!

[Moe]

The problem with the solution is that by default debug is true, it is only false if you run python with the -O command line switch. I find that this switch is typically not used, which is not necessarily what a user would expect.

[Bill the Lizard]

@Moe: It does seem that the logic of the flag is backwards. if debug evaluates to True I would expect that I am running in debug mode, which is not the case.

[Charlie Martin]

Greg, go back and think about that answer. (1) Java, unlike Python, has a separate compilation phase. (2) While they never became popular, there hae been several java preprocessors. ...

[Charlie Martin]

(con't) Now, as a quiz question, what makes a preprocessor more advantageous in C/C++ than Python?

[martineau]

FWIW I used exactly this technique long ago while working with another interpreted language, PostScript, and found it very useful -- mainly for #include'ing files and #ifdef'ing statements, not so much for macro substitution. It wasn't that much extra trouble to deal with -- just a few extra things in the make files which existed because C++ was also being used. @Evan Plaice's pypreprocessor sounds like something worth checking-out.

[martineau]

BTW: I had to use a special command-line argument with the C-Preprocessor to preserve comments because PostScript's use of "//" conflicted with C/C++'s. That would probably also need to be done to use it with Python which has the "//" integer divide operator.

[Evan Plaice]

@Charlie After the preprocessor runs (and it's written well) it should be able to remove all of the meta statements that aren't used. Therefore, IL (Intermediate Language) or ByteCode if you're using python will only contain the code that is used for that particular preprocessor condition. If you just sprinkle if/else statements, all of those statements still need to be sanity checked every time the code is run (even after optimizations are applied).

[Evan Plaice]

@Charlie (cont) There's the obvious advantage that compiled code doesn't need to be lexed/parsed post compilation, so the C++ preprocessor will run faster than the python one but only on the first execution. After the ByteCode is created, the lexer/parser stage is skipped altogether unless your preprocessor directives are constantly changing. The other major drawback to a python preprocessor is, you can't control what the lexer/parser stage does in the interpreter itself.

[Evan Plaice]

@Charlie (cont) IE, if the code found in the preprocessor meta data isn't valid python, it'll throw an error in the interpreter. Instead of doing 'on the fly' semantics checking, it does it all in one shot. If you could waylay this step, you could effectively have a single source file containing code capable of being run in multiple different languages (as long as that language contains a preprocessor that can follow the same functionality); or code that is capable of running in multiple versions of a language (IE, py2x and py3k in the same file).

[Evan Plaice]

@Charlie (cont) I originally wrote pypreprocessor to use as a tool to update python (pypi) libraries to be capable of running in both py3k and python2x. Unfortunately, I discovered the hard way that there's no way to create a true preprocessor in python alone because a true preprocessor actually executes in the interpreter before the source is lexed/parsed. The best option would be to include a true preprocessor in python but, due to the rampant abuse of macro replacements in C, the core devs won't touch the preprocessor subject with a 10ft pole.

[Evan Plaice]

@Charlie (cont) The sucky part about any preprocessor that isn't supported natively in the interpreter is... you still need to create a temporary file to contain the code post-transformation and pre-compilation/pre-execution because it still needs to be compiled/executed after the preprocessor is done doing its thing. If included in the interpreter, it would be a simple matter to add an additional preprocessor lexer/parser step before the actual source lexer/parser and TextStream the output to memory to be passed along the chain.

[Evan Plaice]

@Charlie (cont) the major disadvantage to using Java as a preprocessor is, Java requires a massive framework for such a simple task. The major disadvantage to using C/C++ is, the preprocessor executable will be platform specific. Not an issue if you're only working on one platform but it kinda defeats one of the main advantages to using python. The ideal would be a preprocessor in the interpreter, because the interpreter already abstracts away the platform specific details anyway.

[Evan Plaice]

@Charlie (cont) The major drawback to including it in the interpreter is, 99.9% of python scripts will never use the preprocessor so the additional lexer/processor necessary to make a preprocessor work just adds unnecessary overhead. It would be a simple matter to add an option for enabling the preprocessor in the script itself but python (which emphasizes simplicity) isn't exactly well known for adding meta options to their interpreter (because it's essentially an ugly hack).

[Evan Plaice]

@Charlie (cont) Sorry for the long responses. I've obviously spent a lot of time thinking/working on this issue because I see a need for a tool that makes coding in py2x/py3k side-by-side possible. With it, the source library migration to support py3k code could be a 2-3 year transition vs the currently expected 5-10 year transition. I'm talking about the PYPI (python package index) source, not the average Joe's one-off scripts. IMHO, Py3k support is lacking and will continue to suck as long as there's no easy way to support the transition.

[jonny]

But it allows you to have assertions in development that get stripped on a production server if you only use -O there. :-) I didn't think this was possible before reading this!

[jonny]

This also just made me learn that assert uses __debug__ which makes development assertions even more tangible: docs.python.org/2/reference/simple_stmts.html#assert

[victor n.]

i agree with @Moe. this is backward and confusing. even with knowing that there is a feature like that, i still had to Google what it was and get to this answer. happens every single time. no one usually runs python with -O set but oh well...

[Charlie Martin]

LONG time since I looked at this, @EvanPlaice, sorry. Here's my point: lots of preprocessors out there. Python doesn't have one. If you want to do what a p[reprocessor can do, use one of them. But if you want to do this in Python, you do it with Python code.

[ntg]

That would be perfect if one could use it across modules. However, it seems DEBUG will only be visible to a single file and you would use additional mechanisms to spread it across files...

[Ricardo Stuven]

How would you apply it to decorators in a simple way?

[David S.]

Sadly this packages does not seem to be supported anymore, and currently not even an install via pip install is working (someone else already documented this issue in the repo).

[lorenzo]

-O stands for optimize like in other languages, nothing backward here.

[MuhsinFatih]

The answer is simply wrong. See the accepted answer stackoverflow.com/a/482244/2770195 . I just tested a short script and got 3% average speedup using the flag in the accepted answer. Mind you, mine is a very short script. I can imagine if your program has a lot of debug-related checks, you will see a bigger difference