CakePHP High-Availability Server Farm setup

I am currently working on configuring my CakePHP (1.3) based web app to run in a HA Setup. I have 4 web boxes running the app itself a MySQL cluster for database backend. I have users uploading 12,000 - 24,000 images a week (35-70 GB). The app then generates 2 additional files from the original, a thumbnail and a medium size image for preview. This means a total of 36,000 - 72,000 possible files added to the repositories each week.

What I am trying to wrap my head around is how to handle large numbers of static file request coming from users trying to view these images. I mean I can have have multiple web boxes serving only static files with a load-balancer dispatching the requests.

But does anyone on here have any ideas on how to keep all static file servers in sync?

If any of you have any experiences you would like to share, or any useful links for me, it would be very appreciated.

Thanks,

serialk

Answers


It's quite a thorny problem.

Technically you can get a high-availability shared directory through something like NFS (or SMB if you like), using DRBD and Linux-HA for an active/passive setup. Such a setup will have good availability against single server loss, however, such a setup is quite wasteful and not easy to scale - you'd have to have the app itself decide which server(s) to go to, configure NFS mounts etc, and it all gets rather complicated.

So I'd probably prompt for avoiding keeping the images in a filesystem at all - or at least, not the conventional kind. I am assuming that you need this to be flexible to add more storage in the future - if you can keep the storage and IO requirement constant, DRBD, HA NFS is probably a good system.

For storing files in a flexible "cloud", either

Tahoe LAFS

Or perhaps, at a push, Cassandra, which would require a bit more integration but maybe better in some ways.

MySQL-cluster is not great for big blobs as it (mostly) keeps the data in ram; also the high consistency it provides requires a lot of locking which makes updates scale (relatively) badly at high workloads.

But you could still consider putting the images in mysql-cluster anyway, particularly as you have already set it up - it would require no more operational overhead.


Need Your Help

Why would you quote a LambdaExpression?

c# linq lambda linq-expressions custom-linq-providers

I've read this answer and understood from it the specific case it highlights, which is when you have a lambda inside another lambda and you don't want to accidentally have the inner lambda also com...

Proper way of sharing data and methods in Java

java oop class

So I'm finally starting learn efficient programming (Used to use one class for everything), and this new game I'm making has the following classes so far:

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.