®
Splunk Search Head Clustering (SHC) Cheat Sheet (v2.0) https://www.aplura.com/cheatsheets
Search Head Clustering
A Search Head Cluster is a group of Search Heads that work together to create high availability and
horizontal scaling by sharing configurations, job schedules, and search artifacts
Captain is the Member whom delegates any Deployer is the instances that pushes apps,
activity on the Members that is not an ad-hoc user data, and configurations to the members
search through out the SHC effectively
spreading the work load. This includes Members are the instances which searches
controlling replication and pushing knowledge and jobs are preformed on
bundles to Search Peers (Indexers.)
The Captain is a roll that rotates between Note: It is always recommended to use a load
Members using an election process. To win the
election the Member must receive majority votes
balancer with “sticky sessions” enabled to allow for
from all the other Members continuity of which system a user is using
The Captain election process starts when either Note: Enterprise Security should only be used on a
the current Captain restarts, some of the SHC if the number of concurrent users and/or
Members are separated on the network, and/or concurrent searches exceeds the capacity of a
the current Captain steps down due to missing a maximum speced stand alone instance.
majority of the other Members
Deployment Steps
1 Create the To create a Deployer configure server.conf Conf [shclustering]
Deployer pass4SymmKey = YurPwd
Restart the Member after this command shcluster_label = YurName
To Initialize the Search Heads run the follow command after you modify it with your chosen
2 Initialize the information on each Search Head Member
Members /splunk init shcluster-config -mgmt_uri https://
CLI
<LocalSearchheadAddress>:8089 -replication_port <yourPort> -
conf_deploy_fetch_url https://<DeployerAddressHere>:8089 -secret
shc1node -shcluster_label shcluster1
3 Boot-Strap Boot-strapping initializes the Captain role for the SHC joining the Members
the Members together. Only run this command on 1 Member
/splunk bootstrap shcluster-captain -servers_list "https://spl-
CLI
srch01:8089,https://spl-srch02:8089,https://spl-srch03.edu:
8089"
Restart the member after this command
4 Apply the It does not matter which Member is used in the command
First Search Always add the “preserve-lookups true” flag
Head Cluster CLI
/splunk apply shcluster-bundle --answer-yes -target https://
Bundle <AnyMemberNameInSHC>:8089 -preserve-lookups true
The “boot-strap” command and the first “apply shcluster-bundle” command will take a short while to run
TIPS Their will be a different version of the initialize command for each Member.
The order of initializing Members does not matter
Always build a SHC from a new install of Splunk CLI Splunk CLI command
Conf Splunk .conf file configuration
Provided by Aplura, LLC. Splunk Consulting and Application Development Services sales@aplura.com • https://www.aplura.com
Splunk is a registered trademark of Splunk, Inc. This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License.
v2.0
Many Solutions, One Goal.
Management Tricks Gotchas
You have the option to remotely load the The setup guide on back is meant for a new SHC
deployment only. A SHC migration involves more steps
Deployer from another Splunk Server. and is more complicated.
CLI /splunk apply shcluster-bundle -- The “Deployer” is != the “Deployment Server”
answer-yes -target https://
The “KV store replication” is != the “SHC replication”
<Any1MemberNameInSHC>:8089 -preserve-
lookups true -uri https:// A SHC has a limit of 5000 active and/or unexpired alerts
<deployeraddress>:8089 Once the SHC is created, avoid modifying conf files
directly on the members themselves. The SHC will not
This link discusses how you can run replicate changes made directly to any conf file.
TIPS most other Splunk commands remotely The Captain role is chosen by election and will not always
be the same server. Use the Monitoring Console or Splunk
Troubleshooting command “show shcluster-status” to find the Captain
The majority of the SHC Members must be online for
Replication errors: If a Seach Head failes to sync more
dynamic Captain selection (Captain election) to occur.
than 20 consecutive times then it becomes an Error and
requires a forced sync. Make sure to have necessary network ports for the KV
store, SHC replication, and management open before you
CLI /splunk resync shcluster-replicated-config
initialize the SHC Members
Replication errors: Make sure the Replication Factor is the
same on all Search Head Cluster Members.
Replication
Replication errors: Check the Members to see if an
extremely large Lookup file exists or if the members are
timing out while pushing/pulling configurations.
What causes SHC Replication?
Changes made through: Splunk Web, Splunk CLI
If a SHC looses its majority members for a prolonged commands, and REST API
period of time and can not elect a Captain by itself.
Manually set a temporary Captain as static. Revert the
SHC to dynamic after the majority of Members are
restored. Do not leave the SHC in a static Captain state.
What gets Replicated (by Default)?
alert_actions manager searchscripts
KV store authentication models segmenters
Once the SHC is created the Members also authorize multikv tags
cluster the KV store. A single instance assumes datamodels nav times
the role KV store Captain. event_renderers panels transforms
The KV store Captain handles all write requests for the eventtypes passwd transactiontypes
cluster’s KV store. Any other KV store request is handled fields passwords ui-prefs
locally on the instance. html props user-prefs
All instances in the SHC sync from the KV store Captain. literals quickstart views
lookups savedsearches viewstates
To view the KV store status: macros searchbnf workflow_actions
CLI /splunk show kvstore-status
If the KV store becomes out of sync you can resync the KV lookup table, datamodel JSON, nav XML, meta files
store manually from the SHC Captain:
CLI /splunk rsync kvstore
Change what is replicated in the SHC by modifying the
Add the flag below to the command above if you wish to replication whitelist in server.conf on all members
use a different Member as the source for the sync instead
of the SHC Captain
CLI -source sourceId CLI Splunk CLI command
Conf Splunk .conf file configuration
Provided by Aplura, LLC. Splunk Consulting and Application Development Services sales@aplura.com • https://www.aplura.com
Splunk is a registered trademark of Splunk, Inc. This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License.
v2.0
Many Solutions, One Goal.