Python Numpy Memory Leaks Due to PYTHONMALLOC (C Python and Address Sanitizer)

A simple C/C++ calling Python script to import numpy package will simply be reported Leaking Memory:

1
2
3
4
5
6
7
8

include <Python.h>
 
int main() {
    Py_Initialize();
    PyRun_SimpleString("import numpy");
    Py_Finalize();
    return 0;
}

include <Python.h>

int main() {
    Py_Initialize();
    PyRun_SimpleString("import numpy");
    Py_Finalize();
    return 0;
}

This will be caught leaking memory by Address Sanitizer which is a tool to inspect the potential memory leaks and other memory issues such as Heap Overflow, Double Free, Heap Use after Free in the code.

==7==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 412 byte(s) in 38 object(s) allocated from:
    #0 0x7fa330131808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7fa32a6d2d19 in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/usr/local/lib/python3.8/dist-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so+0x39cd19)

Direct leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7fa330131c3e in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:163
    #1 0x7fa32a6d3218 in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/usr/local/lib/python3.8/dist-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so+0x39d218)

Direct leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7fa330131c3e in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:163
    #1 0x7fa32a6d3200 in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/usr/local/lib/python3.8/dist-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so+0x39d200)

Direct leak of 23 byte(s) in 1 object(s) allocated from:
    #0 0x7fa330131808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7fa32a6d2d58 in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/usr/local/lib/python3.8/dist-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so+0x39cd58)

Direct leak of 12 byte(s) in 1 object(s) allocated from:
    #0 0x7fa330131c3e in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:163
    #1 0x7fa32a6d3233 in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/usr/local/lib/python3.8/dist-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so+0x39d233)

Direct leak of 12 byte(s) in 1 object(s) allocated from:
    #0 0x7fa330131808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7fa32a6d2dbe in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/usr/local/lib/python3.8/dist-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so+0x39cdbe)

Direct leak of 12 byte(s) in 1 object(s) allocated from:
    #0 0x7fa330131808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7fa32a6d2dad in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/usr/local/lib/python3.8/dist-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so+0x39cdad)

SUMMARY: AddressSanitizer: 519 byte(s) leaked in 44 allocation(s).

Python uses the default memory allocator, which seems to be causing problems issue. Fortunately, we can replace it by setting Environmental Variable PYTHONMALLOC to malloc – the C library version.

1

export PYTHONMALLOC=malloc

export PYTHONMALLOC=malloc

“import numpy” is reported as memory leaks by Address Sanitizer. How can I ignore this?

Address Sanitizer (ASan) is a tool used to detect memory errors such as buffer overflow, use-after-free, and memory leaks. Memory leaks occur when memory is allocated but not freed, leading to a gradual accumulation of memory usage over time.

If you have determined that the memory leak is not a critical issue and you want to ignore it, you can use ASan’s suppression mechanism to ignore the leak. To do this, you need to create a suppression file that specifies the functions or modules to ignore.

Here’s an example suppression file that ignores memory leaks in the libpython (which is Python module for C/C++):

1
2

# Ignore memory leaks in the Python for C Module
leak:libpython

# Ignore memory leaks in the Python for C Module
leak:libpython

To use this suppression file with ASan, you can pass the path to the file as a command-line argument to your program, like this:

1
2

$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so ASAN_OPTIONS=detect_leaks=1 \
   ASAN_OPTIONS=suppressions=/path/to/suppression/file ./your_program

$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so ASAN_OPTIONS=detect_leaks=1 \
   ASAN_OPTIONS=suppressions=/path/to/suppression/file ./your_program

This command sets the ASAN_OPTIONS environment variable to enable leak detection and load the ASan library, and then specifies the suppression file to use with the suppressions option.

Note that ignoring memory leaks is generally not recommended as it can lead to a gradual buildup of memory usage, which can eventually cause performance issues or even crashes. It’s usually better to identify and fix the root cause of the leak if possible.

Address Sanitizer (ASan) is a tool used for detecting memory errors in C and C++ programs. It works by instrumenting the program’s code to track memory usage and identify issues such as buffer overflows, use-after-free errors, and memory leaks.

Numpy is a popular Python library for numerical computing, which uses C and C++ code under the hood to achieve high performance. Since it uses C and C++ code, it is possible for Numpy to have memory errors that ASan can detect.

To use ASan with Numpy, you will need to compile Numpy with ASan support. This can be done by adding the -fsanitize=address flag to the compiler options when building Numpy. Once Numpy is compiled with ASan support, you can run your Python code using the ASan-enabled Numpy library to detect memory errors.

To detect memory leaks specifically, you can use ASan’s leak detection feature. This feature tracks all dynamically allocated memory and reports any memory that is not freed before the program exits. To use this feature, you can add the -fsanitize=leak flag to the compiler options when building Numpy.

It is important to note that ASan is a tool for detecting memory errors and leaks, but it does not fix them. Once you have identified memory errors, you will need to fix them in your code. This could involve rewriting code to use memory more efficiently, fixing buffer overflows or use-after-free errors, or ensuring that all allocated memory is properly freed when it is no longer needed.

–EOF (The Ultimate Computing & Technology Blog) —