What is the current state of forum software?

Just a question regarding ndbcluster, are you running the replication in the same datacenter/network?
How does it handle datacenter->datacenter? I suppose its more a question of latency tolerance although the datacenter's have a very very low ping between them.
I have played with Galera but it had some serious issues with drop outs triggering rebuilds.

 

@digitalpoint have you ever tried benchmarking your boards with HHVM? (Facebook's HipHop Virtual Machine). Or perhaps the real question is can XenForo run on it?

 

There is no "replication" at least not in the sense of how you think about MySQL replicating everything to every slave (or every node in the case of Galera).

The underlying issue with Galera is that it's built upon technology that absolutely was not designed to scale out linearly. With Galera are doing every SQL write on every node (no choice because all data resides on every node). You are also going to bottleneck at the disk IO because the way InnoDB works and how it needs to write transactions to disk for every commit (normally anyway). Your cluster is also going to be as slow as your slowest node since again... everything is replicated everywhere.

Now think about this on a larger scale... say you were something huge like Facebook and you had 1,000,000 servers in your database cluster. Now imagine trying to replicate everything to all 1,000,000 servers (you of course would have disk space issues too), and InnoDB trying to write all transactions to disk in realtime for every SQL write. The entire cluster would collapse in on itself because again... InnoDB itself wasn't designed to scale out at that level, nor was MySQL replication designed to scale out like to that many slaves.

In my experience, Galera is good up to about 4 nodes/servers in it's cluster... beyond that and MySQL Cluster is a better choice. MySQL Cluster shards data, so every node is only dealing with SOME of the data, all tables are 100% memory resident by default with only the transaction logs needing to be written to disk (which only needs sequential disk IO to append files... while InnoDB is doing a ton of random IO on various places on the drive). And with data split between each node, you don't need as much memory per node. Say you have 20GB of database files, spread across 8 nodes... the per node memory requirements to keep it all in-memory (and have all data exist in on 2 nodes in case any node fails) is 20/8*2... so 5GB of stuff per node... vs. 20GB on every Galera server.

Ndbcluster was designed from the ground up to be high performance and fault tolerant/redundant.

You can run multiple independent clusters in geographically different locations (I believe that's what you are asking)...

http://www.mysql.com/products/cluster/faq.html#16

For that, it supports a more traditional bidirectional replication as well where latency isn't an issue.

I haven't messed with HipHop since they moved to their virtual machine implementation. I think their HHVM setup solved the issue of not being able to use eval() (XenForo doesn't use it much since it's generally a good idea to avoid when you can anyway, but it does use it for it's event listener system).

But either way... no, I haven't messed with it since they went to HHVM.

 

Thanks for all the info, it has given me plenty of food for thought.

 

You can also do some pretty great stuff with MySQL Cluster straight out of the box... 100% non-blocking backups (users don't know a backup is going on, and everything is fully write capable while it's happening). And it's as easy as issuing the command START BACKUP.

You can also do some other interesting things that lead to zero downtime. For example, I upgraded all my nodes to MySQL 5.6 a couple months ago (from 5.5, but it could have been older) without a second of downtime for users. The hot MySQL upgrades and backups are pretty nice.

 

Thanks again, I am just trying to size up hardware requirements and just exactly how to handle a migration to it if we went down that path. The primary reason I am curious about it is minimising downtime for HA etc rather than server load currently. We have servers in several datacenters and having replication across would be great in case one DC gets particularly stormed with ddos we can just use DC2 etc. Galera was nice but yes I did notice the slowest node issue you mentioned and in the end I was not happy with reliability which made it useless for our HA setup we wanted.
I like the idea of minimal ram requirements as that can give quite a few options for hosting light use nodes on lower hardware for failover etc.

We also have to decide if we are going down Xenforo or IPB....
Both have their merits but we dont want to decide until we see where IPB is going in v4.