Python is one of the most popular, beginner-friendly languages to learn. It’s super simple to read by being very direct in its syntax. As long as you know the basics, there really is no question as to what the language is doing at any given time.

However, just like any other language you might study, Python does have some quirks about it. This article will introduce you to some of the idiosyncrasies of Python by telling you what’s going on under the hood.

Note: for this article, we will only be referring to quirks that are relevant to Python 3.

We’ll take a look at:

• 1. Variables, Namespace, and Scope
• 2. Deleting a list item while iterating
• 3. Modifying the dictionary while iterating over it
• 4. Name resolution ignoring class scope
• 5. Beware of default mutable arguments
• 6. Same operands, different story
• 7. What’s wrong with Booleans?
• 8. Class attributes and instance attributes
• 9. split() method
• 10. Wild imports
• What to learn next

1. Variables, Namespace, and Scope

There are two things we need to talk about when it comes to looking at Python under the hood: namespace and scope.

Namespace

In Python, because it is an object-oriented programming language, everything is considered an object. A namespace is just a container for mapping an object’s variable name to that object.

function_namespace = { name_of_obj_a: obj_1, name_of_obj_b: obj_2 }

for_loop_namespace = { name_of_obj_a: obj_3, name_of_obj_b: obj_4 }

We can think of namespaces as just Python dictionaries, where the variable name for the object is the key, and the value is the object itself. We create a new, independent namespace every time we define a loop, a function, or a class. Each namespace has its own hierarchy called scope.

Scope

Scope, at a very high level, is the hierarchy at which the Python interpreter can “see” a defined object. The interpreter starts with the smallest scope, local, and looks outward if it can’t find the declared variable to the enclosed scope. If the interpreter can’t find it in the enclosed scope, it looks to the global scope.

Take this example:

i = 1

def foo():
   i = 5
   print(i, 'in foo()')
   print("local foo() namespace", locals())
   return i

print("global namespace", globals())
foo()

-->
global namespace {'i': 1, '__name__': '__main__', '__cached__': None, 'foo': <function foo at 0x7f974676af28>, '__file__': 'main.py', '__spec__': None, '__loader__': <_frozen_importlib_external.SourceFileLoader object at 0x7f974669bd30>, '__builtins__': <module 'builtins' (built-in)>, '__doc__': None, '__package__': None}
5 in foo()
local foo() namespace {'i': 5}

Here we have a global namespace and we have a foo() namespace. You can take a look at the individual namespaces by printing globals() and printing locals() at the given spots in the code.

The local namespace is pretty straightforward. You can clearly see i and its value. The global namespace is a little different in that it also includes some extraneous stuff from Python.

Here, it shows the foo function as a place in memory rather than the actual function value itself as well as the value for i in the global namespace.

That being said, you can alter a variable in the global namespace. Just use the global keyword in front of the variable name prior to your logic:

i = 1

def foo():
   global i
   i = 5
   print(i, 'in foo()')
   print("local namespace", locals())
   return i

print("global i before func invocation", globals()["i"])


foo()

print("global i after func invocation", globals()["i"])

-->
global i before func invocation 1
5 in foo()
local namespace {}
global i after func invocation 5

2. Deleting a list item while iterating

When working with lists in Python, we need to take a look at what happens when we remove items from a list when we loop over it. In general, it’s not a good idea to iterate and remove items from a list due to unintended consequences. Take these examples:

del keyword

The del keyword only deletes the instance of that item in the local namespace, but not the actual item itself in the global namespace. So, the globally defined list_1 is unaffected.

list_1 = ["apples", "oranges", "bananas", "strawberries"]

for item in list_1:
   del item

print("list_1: ",list_1); # ['apples', 'oranges', 'bananas', 'strawberries']

-->
list_1:  ['apples', 'oranges', 'bananas', 'strawberries']

remove() method

In the remove() method, once Python removes an item from the list, all of the other items will shift to the left once, but the iteration doesn’t happen until after everything has been moved.

list_2 = ["apples", "oranges", "bananas", "strawberries"]

for item in list_2:
   list_2.remove(item)

print("list_2: ",list_2)# ['oranges', 'strawberries']

-->
list_2:  ['oranges', 'strawberries']

Here is a step-by-step rundown of how it happens:

1. First iteration: remove apples. oranges moves to left and is now the current index. bananas moves to left and becomes the next index. strawberries movies to left, and loop goes to the next index.
2. Second iteration: bananas is at current index, so method removes bananas. strawberries moves to left and is now the current index. No more index values, so iteration is done.
3. Result: This leaves oranges and strawberries in the list.

pop(idx) method

For the same reason that we don’t use the remove method when looping over a list, we don’t use the pop(idx) method. When an index is not passed in as an argument, Python removes the last index in the list.

list_3 = ["apples", "oranges", "bananas", "strawberries"]

for item in list_3:
   list_3.pop()

print("list_3: ",list_3) # ['apples', 'oranges']

-->
list_3:  ['apples', 'oranges']

1. First iteration: Remove strawberries, so the list’s length is now three. Move to the next iteration.
2. Second iteration: Remove bananas, so the list’s length is now two. No more index values and iteration is done.
3. Result: This leaves apples and oranges in the list.

Note: If an index is passed into the pop() method and it doesn’t exist, it will raise an IndexError.

So, what does work?

The secret to iterating and manipulating a list in Python is by slicing or making a copy of the list. It’s as simple as using [:]:.

list_4 = ["apples", "oranges", "bananas", "strawberries"]


for item in list_4[:]: 
      list_4.remove()  #pop() would also work here. 

print("list_4: ",list_4) # [] 

list_4[:] This operator makes a copy of the list in memory. The original list is unaffected as we loop through it but does affect the original when all done.

3. Modifying the dictionary while iterating over it

Python dictionaries can be tricky objects to work with. One thing that is absolutely certain, though, is that these dictionaries cannot necessarily be modified at all when they are being looped over.

Depending on the Python version you have, you will either get a Runtime Error, or the loop will run a certain number of times (between 4 and 8) until the dictionary needs to be resized.

You can make a workaround by using list comprehensions, but it’s generally not in best practice.

for i in x:
   del x[i]
   x[i+1] = i + 1
   print(i)
   print(x)

4. Name resolution ignoring class scope

According to the creator of Python, Guido van Rossum, Python 2 had some “dirty little secrets” that allowed for certain leaks to happen. One of these leaks allowed for the loop control variable to change the value of a in the list comprehension.

That’s been fixed in Python 3 by giving list comprehensions their own enclosing scope. When the list comprehension doesn’t find a definition for a in the enclosing scope, it looks to the global scope to find a value. This is why Python 3 ignores a = 17 in the class scope.

a = 5
class Example:
   # global a
   a = 17
   b = [a for i in range(20)]

print(Example.y[0])

5. Beware of default mutable arguments

Default arguments in Python are fallback values that are set up as parameters if the function is invoked without arguments. They can be useful, but if you call the function several times in a row, there can be some unintended consequences.

def num_list(nums=[]):
   num = 1
   nums.append(num)
   return nums

print(num_list())
print(num_list())
print(num_list([]))
print(num_list())
print(num_list([4]))

-->
[1]
[1, 1]
[1]
[1, 1, 1]
[4, 1]

The first two times num_list() is invoked, a 1 will be appended to nums list both times. The result is [1, 1]. To reset the list, you have to pass in an empty list to the next invocation.

Trick! To prevent bugs where you use default arguments, use None as the initial default.

6. Same operands, different story

Reassignments in Python can be tricky if you are not sure of how they work. The = and the += operators carry two different meanings in Python when used in conjunction with lists.

# reassignment
a = [1, 2, 3, 4]
b = a
a = a + [5, 6, 7, 8]

print(a)
print(b)
# extends
a = [1, 2, 3, 4]
b = a
a += [5, 6, 7, 8]

print(a)
print(b)

-->
[1, 2, 3, 4, 5, 6, 7, 8]
[1, 2, 3, 4]
[1, 2, 3, 4, 5, 6, 7, 8]
[1, 2, 3, 4, 5, 6, 7, 8]

When manipulating lists, the = operator just means reassignment. When b is assigned as a, it created a copy of a as it was at the time. When a is reassigned to a + [5, 6, 7, 8], it concatenated the original a with [5, 6, 7, 8] to create [1, 2, 3, 4, 5, 6, 7, 8]. The b list remains unchanged from its original assignment.

With the += operator, when it pertains to lists, is a shortcut for the extends() method. This results in the list changing in place, giving us [1, 2, 3, 4, 5, 6, 7, 8] for both a and b.

7. What’s wrong with Booleans?

When it comes to Boolean values, it seems pretty straightforward. In this mixed array, how many Boolean values do we have and how many integer values do we have?

mixed_type_list = [False, 4.55, "educative.io", 3, True, [], False, dict()]
integers_count = 0
booleans_count = 0

for item in mixed_type_list:
   if isinstance(item, int):
       integers_count += 1
   elif isinstance(item, bool):
       booleans_count += 1


print(integers_count)
print(booleans_count)

-->
4
0

Why is the output 4-0? In short, a Boolean value in Python is a subclass of integers. True in Python equates to 1, and False equates to 0.

8. Class attributes and instance attributes

In object-oriented Python, a class is a template, and an instance is a new object based on that template. What would happen if we were to try to change or mix up the assignments to class variables and instance variables?

class Animal:
   x = "tiger"

class Vertebrate(Animal):
   pass

class Cat(Animal):
   pass

print(Animal.x, Vertebrate.x, Cat.x)

Vertebrate.x = "monkey"
print(Animal.x, Vertebrate.x, Cat.x)

Animal.x = "lion"
print(Animal.x, Vertebrate.x, Cat.x)

a = Animal()
print(a.x, Animal.x)

a.x += "ess"
print(a.x, Animal.x)

-->
tiger tiger tiger
tiger monkey tiger
lion monkey lion
lion lion
lioness lion

Here we have three classes: Animal, Vertebrate, and Cat. When we assign a variable in the Animal class, and the other classes are extensions of the Animal class, those other classes have access to the variable created in the Animal class.

Be certain of your reassignment when working with classes and instances. If you want to alter the template, use the class name, and when you want to alter the instance, use the variable you assigned to the new instance of the class name.

9. split() method

The split() method has some unique properties in Python. Take a look at this example:

print('         foo '.split(" ")) # ['', '', '', '', '', '', '', '', '', 'foo', '']
print(' foo        bar   '.split()) # ['foo', 'bar']
print(''.split(' ')) #['']

--> 
['', '', '', '', '', '', '', '', '', 'foo', '']
['foo', 'bar']
['']

When we give the split method a separator, in this case (" "), and use it on a string of any length, it’ll split on the whitespace. No matter how many whitespace characters you have in a row, it’ll split on each one.

If there is no separator indicated, the Python interpreter will compress all of the repeating whitespace characters into one, and split on that character, leaving only the groups of non-whitespace characters separated.

An empty string split on a whitespace character will return a list with an empty string as its first index.

10. Wild imports

Wildcard imports can be useful when you know how to use them. They have some idiosyncrasies that can make them more often confusing than not. Take this example:

helpers.py:

def hello_world(str):
   return str;

def _hello_world(str):
   return str

main.py:

from helpers import *
hello_world("hello world -- WORKS!")
_hello_world("_hello_world -- WORKS!")

If we were to try to run this in the directory these files were in, the first invocation of the hello_world function would work fine. The second, not so much. When using wildcard imports, the functions that start with an underscore do not get imported.

For those methods, you will either have to directly import the function or use the __all__ list to use your wildcard import.

helpers.py:

__all__ = [hello_world, _private_hello_world]
def hello_world(str):
   return str;

def _private_hello_world(str):
   return str

main.py:

from helpers import *
hello_world("hello world -- WORKS!")
_private_hello_world("__all__ -- WORKS!")

Note: The __all__ variable is surrounded by two underscores on either side.

What to learn next

Congrats! You've now learned about ten common quirks in Python that can improve your code. It's important to understand what's going on under the hood of Python to get the most out of the language. But there is still more to learn to truly master Python.

Next you should learn about:

• del operation
• Tricks with strings
• Subclass relationships
• Bloating instance dicts
• Non-reflexive class method

To get started with these quirks and more, check out Educative's course Python FTW: Under the Hood Instructor Satwik Kansal shares more about how Python works and the reasons for certain errors or responses in the interpreter. You can think of the course as a “Python hacks” handbook. Mind-bending and fun!

Happy learning!

Continue reading about Python on Educative

• Python 3.9 Updates: topographical sort and string manipulation
• Dynamic Programming Tutorial: making efficient programs in Python
• The Python FAQ: quick answers to common Python questions

Start a discussion

What is your favorite use case of Python? Was this article helpful? Let us know in the comments below!