Mastering CloudLinux Node Rebalancing: A Step‑by‑Step Guide for Multi‑WHM Bare Metal Environments

Managing a fleet of WHM/cPanel servers can feel like juggling flaming torches—especially once CloudLinux enters the picture. As your client base expands, resource demand spikes, and without a smart‑balancing strategy you’ll quickly encounter bottlenecks, uneven load distribution, and wasted capacity. CloudLinux’s built‑in node‑rebalancing feature automates the redistribution of containers (LVE‑based cages) across multiple bare‑metal WHM nodes, keeping each server operating at optimal utilization while your customers stay happy.

Why Automated Node Rebalancing Matters

Before diving into the technical steps, consider the business value of an automated rebalancing workflow:

• Maximized hardware ROI: Evenly spread containers prevent a scenario where one server is maxed out while another sits idle.
• Improved performance: Users enjoy consistent CPU, RAM, and I/O allocations, reducing latency spikes caused by overloaded nodes.
• Reduced manual overhead: Automation eliminates daily “ssh‑and‑move” sessions, freeing your staff to focus on higher‑value tasks.
• Scalable growth: As you add new bare‑metal nodes, the rebalancer instantly incorporates them into the pool, keeping the ecosystem balanced from day one.

Prerequisites: Setting the Stage for Success

1. Uniform WHM & CloudLinux Versions

All nodes must run the same major version of WHM/cPanel and CloudLinux (e.g., WHM 112 and CloudLinux 8). Mismatched versions can cause API incompatibilities and unpredictable migration failures.

2. Centralized Root Access & SSH Keys

Configure password‑less SSH between the “master” node (the one that will run the rebalancing script) and every other WHM server. A typical approach:

1. Generate a key pair on the master:
   ssh-keygen -t ed25519 -f /root/.ssh/cloudlinux_rebalancer
2. Copy the public key to each node:
   ssh-copy-id -i /root/.ssh/cloudlinux_rebalancer.pub root@nodeX.example.com
3. Test connectivity:
   ssh -i /root/.ssh/cloudlinux_rebalancer root@nodeX.example.com "hostname"

3. API Tokens with Sufficient Privileges

On each WHM server, generate an API token (WHM → Development → Manage API Tokens) that includes at least the following scopes:

• list_cages
• move_cage
• get_server_stats

Store these tokens securely on the master node—preferably in /etc/cloudlinux/rebalancer.conf with restrictive permissions (600).

Step‑by‑Step: Configuring Automated Rebalancing

1. Install Required Packages

The rebalancer relies on the cloudlinux-rebalancer utility (available from CloudLinux’s repository) and a few standard tools:

yum install -y cloudlinux-rebalancer python3-pip
pip3 install requests

2. Create the Configuration File

Below is a sample /etc/cloudlinux/rebalancer.conf that defines the node pool and thresholds. Adjust the values to match your environment’s typical usage patterns.

[global]
# Minimum free RAM (in MB) before a node is considered “overloaded”
min_free_ram=2048
# Maximum CPU load average (1‑minute) before triggering a move
max_load=4.0
# Desired balance tolerance (percentage difference between most and least loaded nodes)
tolerance=15

[node1.example.com]
api_token=YOUR_TOKEN_FOR_NODE1

[node2.example.com]
api_token=YOUR_TOKEN_FOR_NODE2

[node3.example.com]
api_token=YOUR_TOKEN_FOR_NODE3

3. Write the Rebalancing Script

The following Python script pulls statistics from each node, identifies “hot” cages, and migrates them to the “coldest” node. Save it as /usr/local/sbin/cloudlinux_rebalance.py and make it executable.

#!/usr/bin/env python3
import configparser, subprocess, json, requests, sys

CONF_PATH = '/etc/cloudlinux/rebalancer.conf'
cfg = configparser.ConfigParser()
cfg.read(CONF_PATH)

def get_node_stats(host, token):
    url = f'https://{host}:2087/json-api/get_server_stats'
    headers = {'Authorization': f'whm {token}'}
    resp = requests.get(url, headers=headers, verify=False, timeout=10)
    data = resp.json()['data']
    return {
        'load': data['load_average'],
        'free_ram': data['free_memory'] / 1024,   # MB
        'cages': data['cagefs']['cages']
    }

def list_cages(host):
    cmd = [
        'ssh', '-i', '/root/.ssh/cloudlinux_rebalancer',
        f'root@{host}', 'cldetect', '--list', '--json'
    ]
    out = subprocess.check_output(cmd, text=True)
    return json.loads(out)

def move_cage(src, dst, cage_id):
    cmd = [
        'ssh', '-i', '/root/.ssh/cloudlinux_rebalancer',
        f'root@{src}', 'cldetect', '--move',
        f'--cage-id={cage_id}', f'--target={dst}'
    ]
    subprocess.run(cmd, check=True)

def main():
    nodes = {}
    for section in cfg.sections():
        if section == 'global':
            continue
        token = cfg[section]['api_token']
        stats = get_node_stats(section, token)
        nodes[section] = {'token': token, **stats}

    # Identify overloaded nodes
    overloaded = [
        n for n, v in nodes.items()
        if v['free_ram'] < cfg['global'].getint('min_free_ram')
        or v['load'][0] > cfg['global'].getfloat('max_load')
    ]

    if not overloaded:
        print('All nodes within thresholds – nothing to do.')
        return

    # Find the coolest node (most free RAM)
    target = min(nodes.items(), key=lambda i: i[1]['free_ram'])[0]

    for src in overloaded:
        if src == target:
            continue
        cages = list_cages(src)
        cages.sort(key=lambda c: c['memory'], reverse=True)   # biggest first
        for cage in cages:
            if nodes[src]['free_ram'] >= cfg['global'].getint('min_free_ram'):
                break   # node is now under threshold
            print(f'Migrating cage {cage["id"]} from {src} to {target}')
            move_cage(src, target, cage['id'])
            # Refresh stats after each move
            nodes[src] = get_node_stats(src, nodes[src]['token'])
            nodes[target] = get_node_stats(target, nodes[target]['token'])

if __name__ == '__main__':
    main()

Key points:

• The script uses WHM’s get_server_stats API to fetch load and free RAM.
• Cage listings are retrieved with cldetect --list --json, which returns JSON containing each container’s memory usage.
• Migration is performed with cldetect --move, which handles the underlying LVE transfer and updates the cPanel account mapping.

4. Schedule the Script with Cron

Run the rebalancer every hour (or more frequently if you have a high churn rate). Add the following line to crontab -e for the root user:

0 * * * * /usr/local/sbin/cloudlinux_rebalance.py >> /var/log/cloudlinux_rebalance.log 2>&1

Monitor the log file for errors. A healthy run logs messages such as “All nodes within thresholds – nothing to do.” or “Migrating cage 12345 from node1 to node3”.

Best Practices & Troubleshooting Tips

1. Test in a Staging Environment First

Before deploying to production, spin up a small test cluster (e.g., three VMs) and run the script manually. Verify that:

• Cages retain their data after migration.
• DNS and email routing remain intact (CloudLinux updates /etc/hosts automatically).
• cPanel accounts stay accessible from the client’s perspective.

2. Fine‑Tune Thresholds

If migrations cause short‑term spikes, raise min_free_ram or max_load. Conversely, if some nodes stay under‑utilized for long periods, lower the thresholds to trigger earlier rebalancing.

3. Guard Against “Ping‑Pong” Moves

Without a tolerance check, a cage could bounce back and forth between two nodes. The tolerance setting in the config file prevents this by ensuring the percentage difference between the most and least loaded nodes stays within the defined range before any move is executed.

4. Keep an Eye on Licensing

CloudLinux licenses are tied to physical CPUs. When you add a new bare‑metal node, purchase and assign the appropriate license key via cldetect --license-add. An unlicensed server will cause the rebalancer to abort migrations to that node.

5. Use Monitoring Tools for Visibility

Integrate the rebalancer with your existing monitoring stack (e.g., Zabbix, Grafana). Export metrics such as “cages migrated per hour” and “average node load” via a simple JSON endpoint or by parsing the log file.

Conclusion

Automated CloudLinux node rebalancing transforms a potentially chaotic collection of WHM servers into a harmonious, self‑optimizing infrastructure. By standardizing WHM/CloudLinux versions, establishing secure API and SSH communication, and deploying a lightweight Python‑driven rebalancer, you can keep each bare‑metal node operating near its sweet spot—maximizing hardware ROI, delivering consistent performance, and freeing your team from manual migrations.

Remember that automation is only as good as the parameters you feed it. Regularly review your thresholds, test migrations in a sandbox, and keep licensing up to date. With these safeguards in place, your multi‑node WHM environment will scale gracefully, letting you focus on what truly matters: growing your business and delighting your customers.