What is the problem with shadowing names defined in outer scopes?

Question

I just switched to PyCharm and I am very happy about all the warnings and hints it provides me to improve my code. Except for this one which I don't understand:

This inspection detects shadowing names defined in outer scopes.

I know it is bad practice to access variable from the outer scope, but what is the problem with shadowing the outer scope?

Here is one example, where PyCharm gives me the warning message:

data = [4, 5, 6]

def print_data(data): # <-- Warning: "Shadows 'data' from outer scope
    print data

print_data(data)

Solution 1

There isn't any big deal in your above snippet, but imagine a function with a few more arguments and quite a few more lines of code. Then you decide to rename your data argument as yadda, but miss one of the places it is used in the function's body... Now data refers to the global, and you start having weird behaviour - where you would have a much more obvious NameError if you didn't have a global name data.

Also remember that in Python everything is an object (including modules, classes and functions), so there's no distinct namespaces for functions, modules or classes. Another scenario is that you import function foo at the top of your module, and use it somewhere in your function body. Then you add a new argument to your function and named it - bad luck - foo.

Finally, built-in functions and types also live in the same namespace and can be shadowed the same way.

None of this is much of a problem if you have short functions, good naming and a decent unit test coverage, but well, sometimes you have to maintain less than perfect code and being warned about such possible issues might help.

Solution 2

The currently most up-voted and accepted answer and most answers here miss the point.

It doesn't matter how long your function is, or how you name your variable descriptively (to hopefully minimize the chance of potential name collision).

The fact that your function's local variable or its parameter happens to share a name in the global scope is completely irrelevant. And in fact, no matter how carefully you choose you local variable name, your function can never foresee "whether my cool name yadda will also be used as a global variable in future?". The solution? Simply don't worry about that! The correct mindset is to design your function to consume input from and only from its parameters in signature. That way you don't need to care what is (or will be) in global scope, and then shadowing becomes not an issue at all.

In other words, the shadowing problem only matters when your function need to use the same name local variable and the global variable. But you should avoid such design in the first place. The OP's code does not really have such design problem. It is just that PyCharm is not smart enough and it gives out a warning just in case. So, just to make PyCharm happy, and also make our code clean, see this solution quoting from silyevsk's answer to remove the global variable completely.

def print_data(data):
    print data

def main():
    data = [4, 5, 6]
    print_data(data)

main()

This is the proper way to "solve" this problem, by fixing/removing your global thing, not adjusting your current local function.

Solution 3

A good workaround in some cases may be to move the variables and code to another function:

def print_data(data):
    print data

def main():
    data = [4, 5, 6]
    print_data(data)

main()

Solution 4

I like to see a green tick in the top right corner in PyCharm. I append the variable names with an underscore just to clear this warning so I can focus on the important warnings.

data = [4, 5, 6]

def print_data(data_):
    print(data_)

print_data(data)

Solution 5

It depends how long the function is. The longer the function, the greater the chance that someone modifying it in future will write data thinking that it means the global. In fact, it means the local, but because the function is so long, it's not obvious to them that there exists a local with that name.

For your example function, I think that shadowing the global is not bad at all.

View more solutions

Comments

[Framester]

Also I searched for the string "This inspection detects..." but found nothing in the pycharm online help: jetbrains.com/pycharm/webhelp/getting-help.html

[ChaimG]

To turn off this message in PyCharm: <Ctrl>+<Alt>+s (settings), Editor, Inspections, "Shadowing names from outer scopes". Uncheck.

[Admin]

I for one am not confused. It's pretty obviously the parameter.

[John Colanduoni]

@delnan You may not be confused in this trivial example, but what if other functions defined nearby used the global data, all deep within a few hundred lines of code?

[Admin]

@HevyLight I don't need to look at other functions nearby. I look at this function only and can see that data is a local name in this function, so I don't even bother checking/remembering whether a global of the same name exists, let alone what it contains.

[CodyF]

I don't think this reasoning is valid, solely because to use a global, you would need to define "global data" inside of the function. Otherwise, the global is not accessible.

[stanleyxu2005]

Yes. I think a good ide is able to handle local variables and global variables by refactoring. Your tip really helps to eliminate such potential risks for primitive ide

[dwanderson]

Well, sure, in a perfect world, you enver make a typo, or forget one of your search-replace when you change the parameter, but mistakes happens and that's what PyCharm is saying - "Warning - nothing is technically in error, but this could easily become a problem"

[dwanderson]

@CodyF False - if you don't define, but just try to use data, it looks up through scopes until it finds one, so it does find the global data. data = [1, 2, 3]; def foo(): print(data); foo()

[RayLuo]

@dwanderson The situation you mentioned is nothing new, it is clearly described in the currently chosen answer. However, the point I try to make, is that we should avoid global variable, not avoid shadowing global variable. The latter misses the point. Get it? Got it?

[bruno desthuilliers]

I wholefully agree on the fact that functions should be as "pure" as possible but you totally miss the two important points: there's no way to restrict Python from looking up a name in the enclosing scopes if it's not locally defined, and everything (modules, functions, classes etc) is an object and lives in the same namespace as any other "variable". In your above snippet, print_data IS a global variable. Think about it...

[wojtow]

Fortunately PyCharm (as used by the OP) has a very nice rename operation that renames the variable everywhere it is used in the same scope, which makes renaming errors less likely.

[Erik]

I know referencing non-local variables is a feature in Python, but for those who want to program in this answer's way (like me), it would never be a good idea to reference outer scopes. Therefore, the warning should not be 'This name shadows that name' for the same names in different scopes, but the warning could be issued for any variable referencing an outer scope. It's a warning, not an error, so still perfectly legal and valid at times. Or am I missing a different point?

[micseydel]

I ended up on this thread because I'm using functions defined in functions, to make the outer function more readable without cluttering the global namespace or heavy-handedly using separate files. This example here doesn't apply to that general case, of non-local non-global variables being shadowed.

[Leo]

In addition to PyCharm's renaming operation I would love to have special syntax highlights for variables that refer to outer scope. These two should render this time consuming shadow resolution game irrelevant.

[Felix D.]

Side note: You can use the nonlocal keyword to make outer score referring (like in closures) explicit. Note that this is different from shadowing, as it explicitly does not shadow variables from outside.

[CodeCabbie]

Agree. The problem here is Python scoping. Non explicit access to objects outside the current scope is asking for trouble. Who would want that! A shame because otherwise Python is a pretty well thought out language (not withstanding a similar ambiguity in module naming).

[FlorianH]

Neat, but: I like to debug my scripts by running it in Python Console within Pycharm, allowing me to access the variables from the global scope after the script terminates. After shifting all into function main(), the variables won't be readily available anymore. Any simple solution to that?

[RayLuo]

@florianH , I do not use PyCharm, perhaps you can somehow set a breakpoint at the end of main()?

[FlorianH]

@RayLuo, you don't know how much your small hint will help me throughout the coming years of my Py coding. Embarrassed & happy :-). Note to self (and if any others..): Breakpoint & Debugger to live interact with code at any point of interest; never again annoyingly write tons of data printing commands to check what's going on in the code. If others cannot get the Pycharm prompt to open: after debugger process connected, find the colorful Pycharm cross at the very bottom right of the debugger window (may first be hidden --> click the tiny arrows >>)

[Remko van Hierden]

@florianH , i put a pass row at the end and then put a breakpoint on the pass line. Pass does not do anything.

[o17t H1H' S'k]

prone to errors when forgetting the _

[Christopher Galpin]

I initially upvoted this, and did the same. I'm now reverting code across all of my projects to intentionally shadow the global (when I don't want or need it). I agree with @eyaler, this is extremely error-prone.

[Christopher Galpin]

# noinspection PyShadowingNames

[Hanan Shteingart]

I believe this is not the right answer and does not propose a solution. I believe this should be the answer: stackoverflow.com/a/40008745/2424587

[bruno desthuilliers]

@HananShteingart I already commented on why what you believe should be the answer is not, and it's also part of my own answer.

[pabouk - Ukraine stay strong]

I would not call this a workaround but a better design.

[pabouk - Ukraine stay strong]

1. The function has no parameter any more. 2. You do not need to declare a global variable using global if you need only read access.