Z. QIU
Posted on December 10, 2020
Yesterday I was busy helping a young engineer in our startup to solve a technique problem about strangely high RAM consumption. He has recently implemented an API in our Back-end project for uploading data from json/jsonl file to MongoDB. I spent some time yesterday on digging into the RAM consumption of Python. I would like to share what I've learnt for this subject.
Check RAM status of Linux
Above all, I want to note down how I check the usage of RAM in our Ubuntu 18.04 running on server.
free -h
Output in my case:
Some explanation of the output of free command from this page:
Free memory is the amount of memory which is currently not used for anything. This number should be small, because memory which is not used is simply wasted.
Available memory is the amount of memory which is available for allocation to a new process or to existing processes.
Memory allocation in C++
I'd like to recall firstly how C++ allocates memories for its variables. In C++ (prior to C++ 11), all variables are declared with a predefined type. Thus the compiler can easily decide the size of the variable and where to store it (heap, stack or static area). See this example I wrote yesterday (I have a 64-bit CPU and the compiler is g++ for x86_64) as file testMem.cpp:
#include <iostream>
/* I set here alignment to 1 byte.
One can remove this line to see RAM consumption with default alignment setting /*
#pragma pack(1)
using namespace std;
int g1 = 4;
class c1
{
};
class c2
{
int x = 1;
int y = 2;
char z = 12;
char* name;
};
int main()
{
cout << " ================= " << endl;
cout << " Sizeof g1: " << sizeof(g1) << endl;
cout << " Address of g1: " << &(g1) << endl;
cout << " ================= " << endl;
int a = 100;
double b = 20.0;
c1* myC1 = new c1(); // heap
c2* myC2 = new c2(); // heap
char c = 55;
short d = 122;
cout << " Sizeof a: " << sizeof(a) << endl;
cout << " Address of a: " << &(a) << endl;
cout << " Sizeof b: " << sizeof(b) << endl;
cout << " Address of b: " << &(b) << endl;
cout << " Sizeof c: " << sizeof(c) << endl;
cout << " Address of c: " << static_cast<void *>(&c) << endl;
cout << " Sizeof d: " << sizeof(d) << endl;
cout << " Address of d: " << static_cast<void *>(&d) << endl;
cout << " ================= " << endl;
cout << " Sizeof c1: " << sizeof(c1) << endl;
cout << " Sizeof c2: " << sizeof(c2) << endl;
cout << " Sizeof myC1: " << sizeof(myC1) << endl;
cout << " Sizeof myC2: " << sizeof(myC2) << endl;
cout << " ================= " << endl;
cout << " Address of ptr myC1: " << static_cast<void *>(&myC1) << endl;
cout << " Address of ptr myC2: " << static_cast<void *>(&myC1) << endl;
cout << " Address value of myC1: " << static_cast<void *>(myC1) << endl; // heap
cout << " Address value of myC2: " << static_cast<void *>(myC1) << endl; // heap
cout << " ================= " << endl;
int arr[10] = {1};
cout << " Sizeof arr: " << sizeof(arr) << endl; // array of 10 integers
cout << " Address of arr: " << arr << endl;
}
Compile this file and execute it:
> g++ testMem.cpp -o testMem
> ./testMem
Below is the output:
In C++, it's quite clear for us to predict in which memory area (stack/heap/static) a variable is stored by only reading the code. The size of a simple variable in C++ is exactly the number of bytes in which its data has been stored, and it's also straight forward to calculate the size of a compound data type variable. As shown in this example, one can calculate the size of a class/struct by summing up the sizes of its non-static data members (one can search google for a further explanation).
Memory allocation in Python
Now let's do some tests in Python. We can use sys.getsizeof() and id() to get the size and address of an object in Python, however, the real calculation of RAM consumption in Python is a little more complicated than expected.
import sys
import time
def testMem():
a1, a2 = 1, 1.0
print("++ a1 has size: {}, address: {}".format( sys.getsizeof(a1), id(a1) ))
print("-- a2 has size: {}, address: {}".format( sys.getsizeof(a2), id(a2) ))
b1, b2 = 256, 257
print("++ b1 has size: {}, address: {}".format( sys.getsizeof(b1), id(b1) ))
print("-- b2 has size: {}, address: {}".format( sys.getsizeof(b2), id(b2) ))
c1, c2 = -5, -6
print("++ c1 has size: {}, address: {}".format( sys.getsizeof(c1), id(c1) ))
print("-- c2 has size: {}, address: {}".format( sys.getsizeof(c2), id(c2) ))
d1 = {"x":12}
d2 = {"x1":100000, "x2":"abcdefg", "x3":-100000000000, "x4":0.00000005, "x5": 'v'}
print("++ d1 has size: {}, address: {}".format( sys.getsizeof(d1), id(d1) ))
print("-- d2 has size: {}, address: {}".format( sys.getsizeof(d2), id(d2) ))
e1 = (1, 2, 3)
e2 = [1, 2, 3]
print("++ e1 has size: {}, address: {}".format( sys.getsizeof(e1), id(e1) ))
print("-- e2 has size: {}, address: {}".format( sys.getsizeof(e2), id(e2) ))
if __name__ =="__main__":
testMem()
Execution output:
As we can see in the picture above, variables' sizes in Python are larger than in C++. Reason for this fact is that everything in Python is an Object (i.e instance of a class-type). Some interesting facts seen in this example:
- addresses of integers in [-5, 256] are far away from that of other integers (-6, 257 in this example)
- a short Dict has the same size as a long Dict
- size of a tuple/list is not the sum of all its items
To understand the details of memory management in Python, one can refer to this article. Here I want to emphasize two important things :
- The management of Python is quite different from C++, Python objects have a huge fixed overhead regarding C++.
- For Python containers,
sys.getsizeof()does not return the sum of its containing objects, however, it returns only the memory consumption of the container itself and the pointers to its objects.
Below is an example for calculating the "real size" of a container. This function total_size() will go over all items in the container and sum up their sizes to give a total size of the container. See the code:
from __future__ import print_function
from sys import getsizeof, stderr
from itertools import chain
from collections import deque
try:
from reprlib import repr
except ImportError:
pass
import sys
def total_size(o, handlers={}, verbose=False):
""" Returns the approximate memory footprint an object and all of its contents.
Automatically finds the contents of the following builtin containers and
their subclasses: tuple, list, deque, dict, set and frozenset.
To search other containers, add handlers to iterate over their contents:
handlers = {SomeContainerClass: iter,
OtherContainerClass: OtherContainerClass.get_elements}
"""
dict_handler = lambda d: chain.from_iterable(d.items())
all_handlers = {tuple: iter,
list: iter,
deque: iter,
dict: dict_handler,
set: iter,
frozenset: iter,
}
all_handlers.update(handlers) # user handlers take precedence
seen = set() # track which object id's have already been seen
default_size = getsizeof(0) # estimate sizeof object without __sizeof__
def sizeof(o):
if id(o) in seen: # do not double count the same object
return 0
seen.add(id(o))
s = getsizeof(o, default_size)
if verbose:
print(s, type(o), repr(o), file=stderr)
for typ, handler in all_handlers.items():
if isinstance(o, typ):
s += sum(map(sizeof, handler(o)))
break
return s
return sizeof(o)
def testMemory():
a = {"x":12}
b = {"x1":1, "x2":"hello", "x3":1.2, "x4":-3, "x5":2000000}
print("memory of a: {}".format( total_size(a) ))
print("memory of b: {}".format( total_size(b) ))
print('Done!')
if __name__ == '__main__':
testMemory()
Execution output:
Our issue
To analyze our technique problem, I have used a very handy tool memory_profiler for displaying memory consumption status. Here is my test code (function total_size() is needed but not shown here):
from pymongo import MongoClient
from memory_profiler import profile
import sys
@profile
def testMemory():
client = MongoClient("mongodb://xxxx:tttttt@ourserver:8917")
db = client["suppliers"]
col = db.get_collection("companies_pool")
re_query_filter = {"domain": {'$regex': "科技"} }
docs = col.find(re_query_filter)
print(type(docs))
docs = docs[10:]
l = list(docs)
print("memory of l: {}".format( total_size(l) ))
f = open("D:\\concour_s2\\Train\\dd.zip", "br") // a large file in my test, in server case, it shall load only json/jsonl file
s = f.read()
print("memory of s: ", sys.getsizeof(s))
del l
del s
print('Done!')
if __name__ == '__main__':
testMemory()
Below is the execution output:
The code in this part is not what our engineer has written in his project, but he has implemented some similar operations in his API method. With this simple example, he has now understood why his method devoured surprisingly so much RAM at times. Then the problem has been quickly solved.
Posted on December 10, 2020
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.
Related