Efficient way to read large tab delimited txt file?

I have a tab delimited txt file with 500K records. I'm using the code below to read data to dataset. With 50K it works fine but 500K it gives "Exception of type 'System.OutOfMemoryException' was thrown."

What is the more efficient way to read large tab delimited data? Or how to resolve this issue? Please give me an example

public DataSet DataToDataSet(string fullpath, string file)
{
    string sql = "SELECT * FROM " + file; // Read all the data
    OleDbConnection connection = new OleDbConnection // Connection
                  ("Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + fullpath + ";"
                   + "Extended Properties=\"text;HDR=YES;FMT=Delimited\"");
    OleDbDataAdapter ole = new OleDbDataAdapter(sql, connection); // Load the data into the adapter
    DataSet dataset = new DataSet(); // To hold the data
    ole.Fill(dataset); // Fill the dataset with the data from the adapter
    connection.Close(); // Close the connection
    connection.Dispose(); // Dispose of the connection
    ole.Dispose(); // Get rid of the adapter
    return dataset;
}

Answers


Use a stream approach with TextFieldParser - this way you will not load the whole file into memory in one go.


Need Your Help

Save image from URL by paperclip

ruby-on-rails ruby upload paperclip

Please suggest me a way to save an image from an URL by Paperclip.

Searchable Plugin Unable to Resolve Class in Grails 2.2.4

grails searchable grails-searchable

Whenever I run my Grails application for the first time, using searchable 0.6.4, I get the following error:

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.