Writing to file doesn't flush content automatically and cause out of memory in Python

I made simple python program to generate big text file:

import sys
import random

f = open('data.txt', 'w')
for i in range(100000000):
        f.write(str(i) + "\t" + str(random.randint(0,1000)) + "\n")
f.close()

When I launch it using CPython it eat all available OS memory and write nothing to the file.

When I launch it on Jython I get OutOfMemoryException.

As far as I understand it stores everything in memory buffer and never did flush before close() call.

My question is: how to limit the file buffer and trigger autoflush? I don't want to callflush() manually, I think it's wrong from performance point of view. I want flush() to be called automatically when file buffer is overloaded or so.

Thanks!

Answers


Buffering is not the problem. The problem is calling the range() function with a giant argument, which will attempt to allocate an array with lots of elements. You will get the same error if you just say

r = range(100000000)

Have you tried passing in a buffer size to the open function?

f = open('data.txt', 'w', 5000)

Need Your Help

SQL Server 2000 to SQL Server 2008 R2 migration

sql sql-server sql-server-2000 sql-server-2008-r2

I am using SQL Server 2000 including 77 databases, and I want to migrate them to the new SQL Server 2008 R2.

How do I use cookies for sessions in mod_perl?

perl mod-perl

What are the exact requirements for using cookies? How do I use cookies for maintaining a session in mod_perl?

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.