Python, releasing memory for real, possible? Could it be because I'm using named tuples?

The code I've inherited is too nested to try and paraphrase here. Basically I have a class method defined that makes a copy of a complex graph structure (e.g. graph=NetworkX_graph.copy()) and returns it as part of a named tuple.

The returned named tuple is compared against a max value, and if it's higher, it's kept. e.g.

if value > max_value:
    best_value = {"index": index, "value": value, "graph": graph}

How do I release the memory allocated for my copied object?!? I've tried everything I can think of. I'm currently using memory_profiler and attaching the @profile decorator to the method that include the .copy(). That copy in the test case I'm working with increments by 7.8MB, (could be higher or lower depending on the case) and is never released. It just keeps climbing until the application itself exceed the available system memory and it starts swapping to disk. (ugly...)

I've tried setting the no longer required tuple to None, the 'del' it, then gc.collect(), and gc.collect(2). The memory usage keeps growing.

BTW, I'm stuck with Python 2.6, I could force a move to 2.7.

Could it be because I'm using tuples?

Answers


Ok, I solved it. It required a little refactoring. When I was passing in a object reference (as a parameter to a method/function call) any changes to that reference would result in new memory being allocated. The memory wouldn't be released/gc.collect() until after the reference was out of scope. e.g. the code pointer returned to the function/method that passed the reference in question in the first place, and that function/method exited.

The following is the most time I can spend trying to illustrate the problem:

_compute(self, graph):
    maxValue = 5
    values = {}
    keeper = {}
    values ["graph"] = graph.copy()
    for i in range(1,1000):
        self._process(values )
        if values ["value"] > maxValue:
            keeper = {"graph":values ["graph"], "value":values ["value"]}

_process(self, values):
    graph = values["graph"]
    # Do some graph processing, like make a copy, allocate some memory, add some vertex values, etc... 
    values["value"] = <some value, like 0 to 10>
    values["graph"] = graph

Any changes to the "values" object passed into _process as a reference will result in the Python runtime holding old and new versions of the changed data. It won't get released until you'r back in _compute and it's exited.

My fix was to modify the _process method to actually return the graph object. Once I was returning objects instead of modifying passed in references, the garbage collection was back to release the allocated memory properly.

_compute(self, graph):
    maxValue = 5
    values = {}
    keeper = {}
    values ["graph"] = graph.copy()
    for i in range(1,1000):
        newGraph = self._process(values )
        if values ["value"] > maxValue:
            keeper = {"graph":newGraph, "value":values ["value"]}

_process(self, values):
    graph = values["graph"]
    # Do some graph processing, like make a copy, allocate some memory, add some vertex values, etc... 
    values["value"] = <some value, like 0 to 10>
    return graph

I'm not saying my answer is the best or that I even have a clue what's going on as I'm not a Python expert, but hopefully this helps someone looking for clues on memory consumption like I was.


Need Your Help

Git altering binary files when cloning

windows git github

I'm just getting started with git and am running into a problem when attempting to clone a respository from github. There are several binary files (.exe and .dll) contained in the repo; when I make a

How to tell if two arrays are permutations of each other (without the ability to sort them)

arrays algorithm permutation

If I have two different arrays and all I can do is check whether two elements in the arrays are equal (in other words, there is no comparison function (beyond equals) for the elements to sort them)...

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.