migrating large data between systems

I need a suggestion to implement a functionality for my application. My application is a legacy JSP , JDBC application installed in client environments used by internal employees of the client. I have multiple client and each client have their own environment of my software installed in their system.

A a requirement, In one module we need to have an interface to their existing external application modules to synchronize some data between their system and our system (eg. user info, office info).

As first step, Right now we have some module where the client can upload CSV,EXCEL files and upload to our system's temporary tables.

Then in second step, they use another module where the trigger the instruction to get data from the temporary table which is processed according to our data format and then inserted to our mail tables to be used into our system.

Now, we are planning to provide them some API where they an directly insert data from their system/module to the temporary tables instead of uploading the CSV,EXCEL files. So that it will not be our responsibility to format data and load into temporary tables.

They will access our API and executes specific methods.

My question is how this can be done? We already have a web service where we expose a small amount of our system just to modify/insert data into specific table. But I am afraid if inserting / synchronizing 40k-50k of rows with web service POST call will be a good idea.

Final point is how can we develop a API that can be used from any client (PHP,.net, java) which I think web service is the solutions.

I hope I did explain clearly. Help appreciated.


Yes, you should provide a bulk capability to your web service. Doing 50k or more calls it's not efficient. So, since you already defined a format for a file and it looks that people is familiar with it, you can write a WS that receives the file as multipart. See http://cxf.apache.org/docs/jax-rs-multiparts.html

Additionally, I'd suggest to do this call async. You can return a "JobId" for future reference.

NOTE1: If you think you an provide a new format, use Json. You can provide some RESTFul API. SOAP with XML might have some overhead. NOTE2: Since you are now using HTTP. Be aware of the security risks.

Need Your Help

Java 7 installer for Mac/Windows

osx installer java java-7

I am programming software in Java 7. The users of the software are not all up to date (especially the Mac users). Therefore I need an installer (Windows/OSX) that can be easily integrated with the

Read from list and get errno 9 python

python image list csv screen-scraping

i try to download images from the google api everything worked fine but on download image 75 sometime 52 51 it throw me errno 9 i have 1 csva.py class and 1 main.py here the main.py imp...

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.