Node Transition (Hot Swap)

Transition from a voting validator to another validator without downtime.

How to run node transition

We offer a script called node-transition.sh that enables you to relocate your voting validator to a different machine without experiencing any downtime. This script is integrated into the solana home directory via our ansible playbook installation process

Take time to read the script to fully understand how it works before running it.

Initial Setup

Copying the Public SSH Key of solana user into the remote host

Before running the script, you must be able to ssh between your nodes with the Solana user. The content of your id_rsa.pub file will have to be added to a file ~/.ssh/authorized_keys on your remote machine somehow.

Define remote host IP in node-transition.sh

#!/bin/bash

# Define here the IP of the node to transition the voting operation
TRANSITIONAL_NODE_IP=
...

Validator keypair setup

Each validator must have alternative identities that can be used when they are not actively voting. You can create these fictitious identities on each of your validators as follows:

solana-keygen new -s --no-bip39-passphrase -o unfunded-validator-keypair.json

Assuming that you have executed the init_validator.sh script on both nodes, the unfunded-validator-keypair.json file should already exist.

Your real validator-keypair.json have to be named funded-validator-keypair.json and copied on each node.

Create a symlink validator-keypair.json pointing to funded-validator-keypair.json your primary node.

ln -sf /home/solana/.secrets/funded-validator-keypair.json validator-keypair.json

Create a symlink validator-keypair.json pointing to unfunded-validator-keypair.json your secondary node.

ln -sf /home/solana/.secrets/unfunded-validator-keypair.json validator-keypair.json

Modifying your solana-validator.service

The identity flag and authorized voter flags should be modified on both validators.

...
--identity /home/solana/.secrets/validator-keypair.json \
--vote-account /home/solana/.secrets/vote-account-keypair.json \
--authorized-voter /home/solana/.secrets/funded-validator-keypair.json \
...

Assuming that you have executed the init_validator.sh script on both nodes, the unfunded-validator-keypair.json file should already exist.

Start your nodes

Your primary and secondary nodes should be running.

The primary node is voting and validator-keypair.json point to funded-validator-keypair.json

The secondary node is started, as catches up with the network but it is not voting and validator-keypair.json point to unfunded-validator-keypair.json

You can check that the second is not voting by tailing the log file. You should see Unable to vote.

[2023-03-21T22:52:26.226528450Z INFO  solana_core::replay_stage] Vote account node_pubkey mismatch: FnpP7TK6F2hZFVnqSUJagZefwRJ4fmnb1StS1NokpLZM (expected: DCbiSAQwvSrWkDhiqTfcTUWacY4jxDZWG1s1GSPQ5TwZ).  Unable to vote

Run node-transition.sh on primary to switch to secondary node

At this stage, you can run the script and observe your logs on both nodes.

./node-transition.sh