MariaDB Galera Cluster: Three Nodes, No Single Point of Failure
Galera replicates every transaction synchronously across three MariaDB nodes, so a node failure loses no data and stops nothing, at the cost of an extra millisecond or two per write and some discipline in how the application writes.
The short answer
MariaDB Galera Cluster ties three or more MariaDB nodes together with synchronous multi-writer replication: every transaction is sent to all nodes before COMMIT completes, each node checks that it does not conflict (certification), and then applies it. The data on all three nodes is identical at every moment, so if one node dies the system carries on with the remaining two, losing no committed transaction and needing no manual failover.
Why three and not two? Galera runs on quorum: only the group holding more than half the votes may accept writes. With two nodes, losing one leaves 50%, which is not a majority, so the surviving node stops serving queries to protect the data. With three, losing one leaves 66% and work continues.
The price is higher write latency, at least one network round trip per COMMIT, which is 1 to 2 milliseconds between nodes in the same data center, plus some discipline in the application: InnoDB tables only, a primary key on every table, and retries for transactions rejected by certification. A cluster earns its keep when an hour of ERP downtime costs more than two extra servers.
When your system deserves a cluster
The simpler alternative is a primary with an asynchronous replica and manual or semi-automatic failover. That covers most systems, but it carries two risks: the replica may lag by seconds, so the last transactions disappear when the primary dies, and the failover itself takes anywhere from one to 15 minutes depending on how ready the team is. Galera removes both risks.
A cluster suits an ERP or CRM that staff use all day, with reads outnumbering writes 4 to 1 or more, and that cannot afford to lose an invoice or a journal entry. It does not suit workloads that write heavily to the same rows, such as a single stock counter updated by a hundred requests a second, because certification conflicts pile up and throughput collapses. Nor does it suit databases where single transactions touch hundreds of thousands of rows, since each transaction is replicated whole in one go.
| Nodes | Quorum needed | Failures tolerated |
|---|---|---|
| 2 | 2 | 0 |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
Four nodes give you nothing over three, which is why odd numbers are the rule. If the budget cannot stretch to three database servers, the third can be garbd, the Galera arbitrator, which votes but stores no data and runs on any existing application server.
What synchronous writes cost in latency
On COMMIT the node sends its write-set to every node through ordered broadcast, waits until it has been received everywhere, and then commits locally. Network latency between the two most distant nodes is therefore added to every write transaction. Inside one Riyadh data center that is 1 to 2 milliseconds, so a transaction that took 4ms now takes about 6ms. Reads are untouched because they stay local.
The second factor is flow control. If a node falls behind applying transactions and its queue fills, it asks everyone to pause until it catches up. The cluster moves at the speed of its slowest node, so all three nodes need identical resources. One node with a slower disk or less memory drags the other two down with it.
The third factor is conflict. If two transactions on different nodes modify the same row at the same instant, the first in order wins and the second is rejected with deadlock error 1213 at COMMIT time rather than earlier. The application must retry, and most modern frameworks can, but you need to confirm yours does.
Core wsrep configuration
On every node, the essential settings in the MariaDB configuration are:
- wsrep_on = ON, with wsrep_provider pointing at libgalera_smm.so.
- wsrep_cluster_address = gcomm://10.0.0.11,10.0.0.12,10.0.0.13 listing all three nodes.
- A shared wsrep_cluster_name and wsrep_node_address set to the node's own address.
- wsrep_sst_method = mariabackup, so full state transfers do not lock the donor node.
- binlog_format = ROW, default_storage_engine = InnoDB and innodb_autoinc_lock_mode = 2.
Open ports 3306, 4567, 4568 and 4444 between the nodes only, and none of them to the internet. Start the first node with galera_new_cluster, then start the service normally on the second and third, which join and pull the data automatically. Confirm that wsrep_cluster_size reads 3.
Before moving an existing system, audit the tables. Any MyISAM table must be converted to InnoDB, and any table without a primary key needs one, because Galera relies on the key to locate rows when applying changes, and a keyless table slows the whole cluster.
ProxySQL or HAProxy in front
The application should not know the addresses of all three nodes. Put a load balancer in front that routes connections only to healthy nodes. HAProxy is the simpler choice: TCP balancing with a health check that asks each node whether it is Synced and drops any node that is not. ProxySQL speaks the MySQL protocol itself, so it can split reads from writes, cache repeated query results, and watch wsrep status directly.
Our practical recommendation with either one is a "single writer" pattern: all writes go to one node and reads spread across all three, and if the writer fails the balancer promotes another node within seconds. This pattern nearly eliminates certification conflicts while keeping every benefit of synchronous replication. Run ProxySQL on the application servers themselves, one instance per server, so it never becomes the single point of failure you built the cluster to avoid.
Split scenarios and how to recover
One node isolated
The isolated node sees itself as one third of the cluster, switches to non-Primary and refuses queries, while the other two continue. When the network returns it rejoins automatically and catches up through IST, an incremental transfer from gcache, provided gcache is large enough. Set gcache.size to 2GB or more so you are not forced into a full SST.
Power loss on all three
When everything comes back the cluster will not start by itself, because no node knows whether it is the most recent. Check grastate.dat on each node for the highest seqno, or run mariadbd with the wsrep-recover option if the value is -1. Set safe_to_bootstrap to 1 on the most advanced node only, start it with galera_new_cluster, then start the rest normally. Bootstrapping from the wrong node means losing the latest transactions.
A full 1-1-1 split
If the network between all three nodes breaks at once, every node becomes non-Primary and the cluster stops accepting writes. That is the correct behavior, since the alternative is conflicting writes that can never be merged. Once you know which node is the most recent, run SET GLOBAL wsrep_provider_options='pc.bootstrap=YES' on it to make it Primary, and the others join it when the network returns.
Daily monitoring
Four variables tell you whether the cluster is healthy: wsrep_cluster_size should be 3, wsrep_cluster_status should be Primary, wsrep_local_state_comment should be Synced, and wsrep_flow_control_paused should stay under 0.1, since anything above means a node is slowing everyone. Wire these to immediate alerts, because a cluster quietly running on two nodes is one failure away from stopping.
Estimating the yearly cost of three servers
The nodes must match. For a mid-sized ERP with a database up to 30GB, three servers on the Growth plan (2 vCPU, 4GB RAM, 40GB SSD) at 6,900 SAR each per year come to 20,700 SAR a year for the whole cluster. For heavier loads, three servers on the Business plan (4 vCPU, 8GB RAM) at 11,500 SAR each come to 34,500 SAR a year, plus a +100GB SSD add-on at 500 SAR per node if you need the space. Prices include VAT.
All three nodes sit in the Riyadh data center with sub-millisecond internal latency, which keeps the cost of synchronous writes at its minimum. Remember that a cluster is not a backup: a table dropped by mistake is dropped on all three nodes instantly, so add daily backup at 800 SAR a year on at least one node. Resources are on the Cloud Server page, and every add-on is on pricing and plans.
The practical decision
- If an hour of downtime costs you more than two extra servers, the cluster is justified.
- Audit your tables today: InnoDB only and a primary key on every table.
- Start with a single writer behind ProxySQL, and only enable writes on all three nodes for a proven need.
- Rehearse shutting down all three nodes and bootstrapping from the most recent one before you need it in production.
