How to hook memory allocation failures by malloc()?

14 05 2008

C have malloc() and C++ have more advanced “new” for allocating memory. One of the advantages of new over malloc is that its tailored to report memory allocation failures where malloc() is not. For instance, in C++ we can set a “Allocation failure handler” function by calling _set_new_handler(). When ever “new” fails to allocate function, this function will be called. Its a nice feature that malloc() don’t have by default.

There are several maintenance projects which are actually done in C but ported to C++. Usually instead of rewriting entire source, the legacy code exists in C itself and new feature additions will be done in C++, since they co-exist with each other. So in the old C code base, the malloc() will be used extensively. So hooking memory failures by malloc() are necessary and how can we do it?

By default malloc() won’t call the New Handler function during memory failure. But you can specify to do so by calling _set_new_mode(). See the sample code snippet below.

#include "new.h"

// Memory Allocation Failure Handler.
int AllocationFailureHandler( size_t )
    // Do what ever you wish.
    return 0;

// The one and only Main.
void main()
    // Set the NewHandler and try "new" allocation failures.
    // The NewHandler will be called.
    _set_new_handler( AllocationFailureHandler);
    char* pAllocatedByNew = new char[1000000000000];

    // Check malloc. By default it won't call
    // newHandler on failure.
    char* pAllocatedByMalloc = (char*)malloc( 1000000000000 );

    // Enable allocation failure reporting for malloc.
    // and try once again. This time, NewHandler is called!
    _set_new_mode( 1 );
    pAllocatedByMalloc = (char*)malloc( 1000000000000 );

Now hook the memory allocation failures from you legacy code too… 😉

If you have MFC support, then the NewHandler provided by you will be overridden and that of MFC will handle the allocation failure. So take care…

And _set_new_mode() is not the part of C++ language standard. Its Microsoft specific.

Targeted Audiance – Intermediate.