Efficient way to read large tab delimited txt file?

I have a tab delimited txt file with 500K records. I'm using the code below to read data to dataset. With 50K it works fine but 500K it gives "Exception of type 'System.OutOfMemoryException' was thrown."

What is the more efficient way to read large tab delimited data? Or how to resolve this issue? Please give me an example

public DataSet DataToDataSet(string fullpath, string file)
{
    string sql = "SELECT * FROM " + file; // Read all the data
    OleDbConnection connection = new OleDbConnection // Connection
                  ("Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + fullpath + ";"
                   + "Extended Properties=\"text;HDR=YES;FMT=Delimited\"");
    OleDbDataAdapter ole = new OleDbDataAdapter(sql, connection); // Load the data into the adapter
    DataSet dataset = new DataSet(); // To hold the data
    ole.Fill(dataset); // Fill the dataset with the data from the adapter
    connection.Close(); // Close the connection
    connection.Dispose(); // Dispose of the connection
    ole.Dispose(); // Get rid of the adapter
    return dataset;
}

Answers


Use a stream approach with TextFieldParser - this way you will not load the whole file into memory in one go.


Need Your Help

Getting below error while executing update or delete methods for repeated rows in database by using hibernate

java hibernate

I am very new to hibernate. I am using a code which will insert data into database, retrive, update or delete rows for the table using hibernate annotations. But update and delete methods are not w...

Can you use NodeJs Passport Sessions in a RESTFUL environment?

node.js rest session cookies passport.js

I'm trying to get passport set up in a RESTful environment and can't quite get the user to get serialized into the session data. Login and signup work great, though.

Drawing 100+ circles with different position using UIBezierPaths

ios drawrect uibezierpath

I am trying to draw 100+ UIBezierPath circles in the drawRect function of an UIView. I am creating and adding the UIBezierPaths to an array:

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.