ThisCalculator

Everyday

GlusterFS Redundant Storage Pool Setup Guide

Updated · Standard formula, shown below

GlusterFS Redundant Storage Pool Setup Guide

This guide walks you through creating a redundant storage pool with GlusterFS, a distributed filesystem that combines multiple servers’ disks into one unified volume. You’ll set up two or more servers, each with a spare disk or directory, and configure them to replicate data so that if one server fails, your files remain available on the others. By the end, you’ll have a working GlusterFS volume that tolerates hardware failures without downtime.

What You’ll Need

  • At least two Linux servers (CentOS, RHEL, Ubuntu, or Debian) with static IP addresses and hostnames configured.
  • Each server should have a dedicated disk or partition for GlusterFS (e.g., /dev/sdb) or a directory with enough free space. Do not use the OS disk.
  • Root or sudo access on all servers.
  • Network connectivity between servers on the ports GlusterFS uses (24007/tcp and 49152-49162/tcp for brick communication).
  • Firewall rules allowing those ports, or temporarily disable the firewall for testing.
  • Consistent time synchronization (NTP or chrony) across servers to avoid metadata conflicts.

Step-by-Step

  1. Update and install GlusterFS on all servers. On each server, run the package manager update, then install the glusterfs-server package. For CentOS/RHEL: sudo yum install centos-release-gluster then sudo yum install glusterfs-server. For Ubuntu/Debian: sudo apt update && sudo apt install glusterfs-server. Start and enable the service: sudo systemctl start glusterd && sudo systemctl enable glusterd.
  2. Configure hostname resolution. Edit /etc/hosts on every server to include the hostnames and IPs of all nodes. For example, if you have server1 (192.168.1.10) and server2 (192.168.1.11), add lines like 192.168.1.10 server1 and 192.168.1.11 server2. Ensure each server’s hostname matches its entry (check with hostname).
  3. Probe the peers. From the first server, add the other nodes to the trusted pool. Run sudo gluster peer probe server2 (replace with actual hostname). Repeat for each additional server. Verify with sudo gluster peer status – all peers should show as “Connected”.
  4. Prepare the bricks. On each server, create a directory that will serve as the brick. For example, if you have /dev/sdb formatted as ext4 or xfs and mounted at /data, create sudo mkdir -p /data/brick1. If you’re using a directory on the OS disk, ensure it has enough space. The brick path is the full path to this directory.
  5. Create the replicated volume. On the first server, run the following command to create a volume with 2 replicas (adjust the replica count to match your server count). For two servers: sudo gluster volume create vol-replica replica 2 server1:/data/brick1 server2:/data/brick1. For three servers, use replica 3 and list all three bricks. This sets up mirroring across the bricks.
  6. Start the volume. After creation, start it with sudo gluster volume start vol-replica. Check the status with sudo gluster volume info – it should show “Status: Started”.
  7. Mount the volume on a client. On a client machine (can be one of the servers or a separate host), install the glusterfs-client package (e.g., sudo apt install glusterfs-client). Create a mount point, e.g., sudo mkdir /mnt/glusterfs, and mount using the FUSE client: sudo mount -t glusterfs server1:/vol-replica /mnt/glusterfs. You can also add this to /etc/fstab for persistence.
  8. Test redundancy. Write a test file to the mount point, then stop one of the servers (or unmount its brick) and verify the file is still accessible from the client. This confirms replication is working.

Common Mistakes

  • Using the root filesystem for bricks. If your brick directory is on the OS partition, it can fill up and crash the server. Always use a separate disk or partition.
  • Mismatched replica count. The replica count must equal the number of bricks you specify. If you have 2 servers, you must use replica 2 and list exactly 2 bricks.
  • Forgetting to open firewall ports. GlusterFS uses port 24007 for management and a range for brick communication. If firewalls block these, peer probing fails.
  • Hostname mismatches. If the hostname in /etc/hosts doesn’t match the actual hostname, peer probing fails. Use the same hostname that the hostname command returns.
  • Not formatting the disk. If you use a raw disk without a filesystem, the brick will fail. Format it (e.g., sudo mkfs.xfs /dev/sdb) and mount it first.

Tips

  • Use XFS filesystem for bricks – it’s the recommended filesystem for GlusterFS due to its performance and support for extended attributes.
  • Set the volume’s performance cache to improve read speed: sudo gluster volume set vol-replica performance.cache-size 256MB (adjust as needed).
  • Enable quota if you want to limit usage: sudo gluster volume quota vol-replica enable then set limits.
  • Use a separate network for GlusterFS traffic to avoid saturating your production network.
  • Monitor brick health with sudo gluster volume heal vol-replica info to see if any files need self-healing.

Troubleshooting

  • Peer probe fails with “Host not found” – Check /etc/hosts entries and ensure the hostname resolves. Try pinging the hostname from the probing server.
  • Volume creation fails with “Brick path does not exist” – Verify the brick directory exists on every server. Create it if missing, and check permissions (should be owned by root).
  • Volume start hangs or times out – Check that all peers are connected (gluster peer status). If a peer is disconnected, re-probe it. Also check firewall ports.
  • Mount fails with “mount: /mnt/glusterfs: failed to resolve server” – Ensure the client can resolve the server hostname. Also verify the volume name is correct.
  • Files are not replicating – Run sudo gluster volume heal vol-replica info to see pending heal operations. If the volume is in a split-brain state, you may need to resolve it manually (this is advanced).
  • Performance is slow – Check network latency, ensure the bricks are on fast disks (SSD), and tune the volume’s read/write cache settings.

Frequently Asked Questions

Can I add more servers to an existing replicated volume?

Yes, you can add a new brick to a volume, but for a replicated volume, you must add bricks in multiples of the replica count. For example, if you have replica 2, you must add 2 new bricks at once. Use gluster volume add-brick vol-replica server3:/data/brick1 server4:/data/brick1.

What happens if one server goes down?

With replication, the volume remains available as long as at least one replica of each file is online. The client can still read and write; the volume will self-heal when the server comes back.

Can I use GlusterFS on Windows or macOS clients?

Officially, GlusterFS supports Linux clients via FUSE. There are third-party clients for Windows (e.g., via SMB gateway) but native support is limited. For mixed environments, consider using Samba on top of GlusterFS.

How do I back up a GlusterFS volume?

You can use standard tools like rsync on a mounted client, or use gluster volume geo-replication for continuous asynchronous replication to another cluster.

Is GlusterFS suitable for high-availability databases?

GlusterFS is a general-purpose distributed filesystem, not designed for high-performance database workloads like MySQL or PostgreSQL. For databases, consider using a clustered filesystem like GFS2 or a distributed database.

Can I expand the size of a brick after creation?

Yes, you can add a new brick to the volume (as above) to increase total capacity. You can also grow the underlying filesystem if it’s on LVM, but the brick size itself is fixed at creation.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.