ImageMagick: how to achieve low memory usage while resizing a large number of image files?

I would like to resize a large number (about 5200) of image files (PPM format, each 5 MB in size) and save them to PNG format using convert.

Short version:

convert blows up 24 GB of memory although I use the syntax that tells convert to process image files consecutively.

Long version:

Regarding more than 25 GB of image data, I figure I should not process all files simultaneously. I searched the ImageMagick documentation about how to process image files consecutively and I found:

It is faster and less resource intensive to resize each image it is read:

$ convert '*.jpg[120x120]' thumbnail%03d.png

Also, the tutorial states:

For example instead of...

montage '*.tiff' -geometry 100x100+5+5 -frame 4 index.jpg

which reads all the tiff files in first, then resizes them. You can instead do...

montage '*.tiff[100x100]' -geometry 100x100+5+5 -frame 4 index.jpg

This will read each image in, and resize them, before proceeding to the next image. Resulting in far less memory usage, and possibly prevent disk swapping (thrashing), when memory limits are reached.

Hence, this is what I am doing:

$ convert '*.ppm[1280x1280]' pngs/%05d.png

According to the docs, it should treat each image file one by one: read, resize, write. I am doing this on a machine with 12 real cores and 24 GB of RAM. However, during the first two minutes, the memory usage of the convert process grows to about 96 %. It stays there a while. CPU usage is at maximum. A bit longer and the process dies, just saying:

Killed

At this point, no output files have been produced. I am on Ubuntu 10.04 and convert --version says:

Version: ImageMagick 6.5.7-8 2012-08-17 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2009 ImageMagick Studio LLC
Features: OpenMP 

It looks like convert tries to read all data before starting the conversion. So either there is a bug in convert, an issue with the documentation or I did not read the documentation properly.

What is wrong? How can I achieve low memory usage while resizing this large number of image files?

BTW: a quick solution would be to just loop over the files using the shell and invoke convert for each file independently. But I'd like to understand how to achieve the same with pure ImageMagick.

Thanks!

Answers


Without having direct access to your system it's really hard to help you debugging this.

But you can do three things to help yourself narrowing down this problem:

  1. Add -monitor as the first commandline argument to see more details about what's going on.

  2. (Optionally) add -debug all -log "domain: %d +++ event: %e +++ function: %f +++ line: %l +++ module: %m +++ processID: %p +++ realCPUtime: %r +++ wallclocktime: %t +++ userCPUtime: %u \n\r"

  3. Temporarily, don't use '*.ppm[1280x1280]' as an argument, but use 'a*.ppm[1280x1280]' instead. The purpose is to limit your wildcard expansion (or some other suitable way to achieve the same) to only a few matches, instead of all possible matches.

If you do '2.' you'll need to do '3.' as well otherwise you'll be overwhelmed by the mass of output. (Also your system does seem to not be able to process the full wildcard anyway without having to kill the process...)

If you do not find a solution, then...

  1. ...register a username at the official ImageMagick bug report forum.
  2. ...report your problem there to see if they can help you (these guys are rather friendly and responsive if you ask politely).

Need Your Help

What is the reason for rotating an array?

c++ arrays data-structures

I see these programming questions everywhere:

Comparing strings with C

c string compare

I am trying to make a basic text game in C. I am trying to compare a set string with user input but am failing miserably at it. I am rather new to C (I have worked with javascript, php, and C++ in ...

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.