Before adding nodes into your cluster, you first need to make sure that the cluster token (property CLUSTER_TOKEN in the configuration file) has the same value in each node. For better security, we recommend that you change the predefined value of the token by generating a new one and pasting the same value into each node configuration file. You can generate a new token with the following command:
> grb_rs token GRBTK-6o4xujs59WJO5508nmaNwc1TtjZJAL1UcwN4vTD4qK4nata8oLr9GnubyXrLTkggc/aw2A==
Similarly, the passwords used for client, administrator and cluster administrator must be the same in all nodes. For better security, it is recommended to change the predefined value of the passwords by choosing a new password, generating a hash value for that password, and then pasting the result into each node configuration file. You can generate a hash of your chosen password (e.g., mynewpass) with the following command:
> grb_rs hash mynewpass $$v0UBWkM_9kpY_v2RECV2LBGnlr8qzaGHzf0fMJvrMYwPnJap
Adding nodes to your cluster
Once you've started a single-node cluster, you can add nodes using the --join flag to grb_rs or the JOIN configuration property. For example, if you've already started a cluster on the default port of server1, you would run the following command on the new node (call it server2) to create a two-node cluster:
> grb_rs --join=server1
In the log output for server2, you should see the result of the handshake between the servers:
info : Node server1, transition from JOINING to ALIVE
Similarly, the log output of server1 will include the line:
info : Node server2, added to the cluster
If you are using a non-default port, you can specify the target node port as part of the node URL in the --join flag and you can specify the port of the current node using the --port flag. You could use different ports on the different machines, but it is a good practice to use the same one, for example 61000. The command would look like this instead:
> grb_rs --join=server1:61000 --port=61000
The JOIN property can also be set through the configuration file in the same way:
JOIN=server1:61000 PORT=61000When starting grb_rs as a service, you won't have the opportunity to provide command-line options, so you'll need to provide this information through the configuration file.
Once you've created a multi-node cluster, you can add additional nodes to that cluster by doing a JOIN using the name of any member node. Furthermore, the --join flag or the JOIN property can take a comma-separated list of node names, so a node can still join a cluster even if one of the member nodes is unavailable. Note that when a list of nodes is specified, the joining node will try to join all the specified nodes at the same time. Joining nodes is an asynchronous process, if some target nodes are not reachable, the joining node will retry before giving up on joining. If all the nodes are reachable, they will all join and form a single cluster.
Checking the status of your cluster
Using grbcluster, you can check the status of the cluster:
> grbcluster --server=server1 --password=pass nodes --long ADDRESS STATUS TYPE LICENSE PROCESSING #Q #R JL IDLE %MEM %CPU STARTED RUNTIMES VERSION server1 ALIVE COMPUTE VALID ACCEPTING 0 0 2 1h55m 2.35 3.69 2019-04-07 11:41:25 [8.0.1 8.1.1] 8.1.1-v8.1.1rc0 server2 ALIVE COMPUTE VALID ACCEPTING 0 0 2 1h55m 2.38 1.30 2019-04-07 11:41:33 [8.0.1 8.1.1] 8.1.1-v8.1.1rc0
The nodes of the cluster are constantly sharing information about their status. When using grbcluster, you can use any of the nodes in the --server flag for all global commands.
Each node can be in one of the following states:
- The node is up and running.
- The node failed to respond to recent communications.
The node could return to the ALIVE state if it can be reached
again. The node will stay in this state until a timeout (controlled
by the configuration property DEGRADED_TIMEOUT), at which
point it is considered as FAILED
- The node has been in DEGRADED state for too long,
and has been flagged as FAILED.
A node will remaine in the FAILED state for a short time, and it
will eventually be removed from the cluster. If the node comes back
online, it will not re-join the cluster automatically.
- The node is in the process of joining the cluster.
- The node left the cluster. It will stay in that state for short time period and then it will be removed from the cluster.
You can dynamically add or remove a node from a cluster using the grbcluster join or grbcluster leave commands. The join command can be useful when you want a node to rejoin the cluster after a network issue without having to restart the node. For example, if server2 left the cluster after a failure, it could rejoin using the following command:
> grbcluster --server=server2 --password=cluster join server1