Python os.walk memory issue

I programmed a scanner that looks for certain files on all hard drives of a system that gets scanned. Some of these systems are pretty old, running Windows 2000 with 256 or 512 MB of RAM but the file system structure is complex as some of them serve as file servers.

I use os.walk() in my script to parse all directories and files.

Unfortunately we noticed that the scanner consumes a lot of RAM after some time of scanning and we figured out that the os.walk function alone uses about 50 MB of RAM after 2h of walk over the file system. This RAM usage increases over the time. We had about 90 MB of RAM after 4 hours of scanning.

Is there a way to avoid this behaviour? We also tried "betterwalk.walk()" and "scandir.walk()". The result was the same. Do we have to write our own walk function that removes already scanned directory and file objects from memory so that the garbage collector can remove them from time to time?

Thanks

Answers


have you tried the glob module?

import os, glob

def globit(srchDir):
    srchDir = os.path.join(srchDir, "*")
    for file in glob.glob(srchDir):
        print file
        globit(file)

if __name__ == '__main__':
    dir = r'C:\working'
    globit(dir)

If you are running in the os.walk loop, del() everything that you don't need anymore. And try running gc.collect() at the end of every iteration of os.walk.


Need Your Help

Android MenuItem setActionView(null) crash

android android-4.0 android-actionbar actionbarsherlock

On android 4.0.3 if you call MenuItem.setActionView(null) and then rotate the screen, my application will crash.

Object literal property and method location

javascript object methods properties object-literal

Do all methods have to be stated after all properties in JavaScript when defining objects in object literal form? I have tried voiding this idea and it seems that all properties stated afterwards a...

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.