Is '# -*- coding: utf-8 -*-' also a comment in Python?

Question

As we use # for inserting comments in Python, then how does Python treat:

# -*- coding: utf-8 -*-

differently?

Solution 1

Yes, it is also a comment. And the contents of that comment carry special meaning if located at the top of the file, in the first two lines.

From the Encoding declarations documentation:

If a comment in the first or second line of the Python script matches the regular expression coding[=:]\s*([-\w.]+), this comment is processed as an encoding declaration; the first group of this expression names the encoding of the source code file. The encoding declaration must appear on a line of its own. If it is the second line, the first line must also be a comment-only line.

Note that it doesn't matter what codec should be used to read the file, as far as comments are concerned. Python would normally ignore everything after the # token, and in all accepted source code codecs the #, encoding declaration and line separator characters are encoded exactly the same as they are all supersets of ASCII. So all the parser has to do is read one line, scan for the special text in the comment, read another if needed, scan for the comment, then configure the parser to read data according to the given codec.

Given that the comment is required to be either the first or second in the file (and if it is the second line, the first line must be a comment too), this is entirely safe, as the configured codec can only make a difference to non-comment lines anyway.

Solution 2

See encoding declarations in the Python Reference Manual:

If a comment in the first or second line of the Python script matches the regular expression coding[=:]\s*([-\w.]+), this comment is processed as an encoding declaration; the first group of this expression names the encoding of the source code file.

_{(Emphasis mine)}

So yes, it is a comment, a special one. It is special in that the parser will try and act on it and not ignore it as it does for comments not in the first or second line. Take, for example, an unregistered encoding declaration in a sample file decl.py:

# # -*- coding: unknown-encoding -*-
print("foo")

If you try and run this, Python will try and process it, fail and complain:

python decl.py 
  File "decl.py", line 1
SyntaxError: encoding problem: unknown-encoding

Comments

[Martijn Pieters]

But if you were to register unkown-encoding as an encoding, say, with a .pth file, then that codec is actually loaded and used. This provides a very nice and interesting opportunity for pre-parse code processing.

[Dimitris Fasarakis Hilliard]

Indeed @MartijnPieters I mainly added that as a code example that Python processes the declaration, not to make any other claims for it.

[Łukasz Rogalski]

github.com/dropbox/pyxl would be an example of what @MartijnPieters is referring to.

[Jorge Leitao]

So the real question becomes: why do we use # -*- coding: X -*- instead of # coding: X?

[Martijn Pieters]

@J.C.Leitão: you don't have to. Anything that matches the regular expression would work. But if you are using Emacs as your editor, then that comment also informs that editor what codec to use.

[Nizam Mohamed]

` what codec the file is saved as` shouldn't be what encoding the file is saved as?

[Martijn Pieters]

@NizamMohamed: that sentence indeed had issues, edited it.

[ngShravil.py]

@MartijnPieters ... are there any other like this, which adds special meaning to the comment line?

[Martijn Pieters]

@ShravilPotdar: There's loads. There is the shebang line that Unix systems use, and the Windows py launcher will look at the same info. As mentioned, many editors can be configured using text in comments (not just what codec to use, but many other aspects as well, see the emacs and vim docs). There are probably more.

[Martijn Pieters]

@ShravilPotdar: many linters use config in comments too, see the pylint manual, or flake8.

[Martijn Pieters]

@ShravilPotdar: Since comments have no meaning to the program, they are easily hijacked by other systems that have to work with the code, basically.

[R.M.]

in all accepted source code codecs the #, encoding declaration and line separator characters are encoded exactly the same: Does Python not support UTF-16 source code?

[Martijn Pieters]

@R.M.: no, multi-byte codecs are not supported, for this very reason. From PEP 263: Any encoding which allows processing the first two lines in the way indicated above is allowed as source code encoding, this includes ASCII compatible encodings as well as certain multi-byte encodings such as Shift_JIS. It does not include encodings which use two or more bytes for all characters like e.g. UTF-16. The reason for this is to keep the encoding detection algorithm in the tokenizer simple.

[Mike Williamson]

@R.M. Does any programming language support multi-byte codecs? I don't mean that to sound aggressive... I've just never heard of such a thing. Since UTF-8 can handle all language characters, and since source code needn't worry about file size (source code is small compared to data), I would not think it necessary to support anything "more complete" than UTF-8.