Splunk Enterprise Certified Admin
Exam code SPLK-1003. Validates that you can install, configure, and operate a Splunk Enterprise deployment: license management, configuration files and precedence, indexes and buckets, users/roles/authentication, getting data in through forwarders and inputs, the parsing pipeline, and distributed search. This page is a fast overview plus a thorough, topic-by-topic guide matched to the official exam blueprint.
56 questions 60 minutes Professional-level $130 USD Prereq: Power User Valid 3 yearsPart 1 — High-Level Overview
What & Why
The Splunk Enterprise Certified Admin credential is the third and final rung of the Splunk Core track, reached after Core Certified User (SPLK-1001) and Core Certified Power User (SPLK-1002). Where Power User proves you can build search-time knowledge objects, Admin proves you can stand up and run the platform itself: licensing, indexing, configuration precedence, user/role/auth management, and — the largest single chunk of the exam — getting data in through forwarders, inputs, and the parsing pipeline.
Exam at a Glance
| Attribute | Value |
|---|---|
| Exam code | SPLK-1003 |
| Questions | 56 multiple choice / multiple response |
| Time | 60 minutes total (includes ~3 min to review the exam agreement) |
| Level | Professional |
| Passing score | Not publicly published by Splunk — see §1 for what is/isn't confirmed |
| Cost | $130 USD per attempt |
| Delivery | Pearson VUE — test center or online proctored |
| Prerequisite | Splunk Core Certified Power User (hard requirement, per official blueprint) |
| Validity | 3 years from the date of the highest certification exam passed |
The 17 Blueprint Topics
Splunk's blueprint doesn't group this exam into a handful of big weighted domains the way the Power User exam does — instead it lists 17 numbered topics, mostly at 5%, with three topics bumped to 10%: Splunk Indexes, Distributed Search, and Forwarder Management. Getting Data In is really one giant theme split across six of the seventeen topics (Getting Data In, Staging, Configuring Forwarders, Forwarder Management, Monitor Inputs, Network/Scripted Inputs, Agentless Inputs, Fine Tuning, Parsing, and Manipulating Raw Data) — together that's roughly half the exam.
| # | Topic | Weight | Deep dive |
|---|---|---|---|
| 1.0 | Splunk Admin Basics | 5% | → §2 |
| 2.0 | License Management | 5% | → §3 |
| 3.0 | Splunk Configuration Files | 5% | → §4 |
| 4.0 | Splunk Indexes | 10% | → §5 |
| 5.0 | Splunk User Management | 5% | → §6 |
| 6.0 | Splunk Authentication Management | 5% | → §7 |
| 7.0 | Getting Data In | 5% | → §8 |
| 8.0 | Distributed Search | 10% | → §9 |
| 9.0 | Getting Data In — Staging | 5% | → §10 |
| 10.0 | Configuring Forwarders | 5% | → §11 |
| 11.0 | Forwarder Management | 10% | → §12 |
| 12.0 | Monitor Inputs | 5% | → §13 |
| 13.0 | Network and Scripted Inputs | 5% | → §14 |
| 14.0 | Agentless Inputs | 5% | → §15 |
| 15.0 | Fine Tuning Inputs | 5% | → §16 |
| 16.0 | Parsing Phase and Data | 5% | → §17 |
| 17.0 | Manipulating Raw Data | 5% | → §18 |
Weights and titles above are quoted exactly from splunk-test-blueprint-enterprise-admin.pdf, retrieved directly from splunk.com. They sum to 100%.
Configuration File Snapshot
Admin lives and dies by configuration files. If you master where each of these lives and what it controls, you cover a large share of the exam.
Core precedence
indexes.conf, server.conf, authorize.conf, authentication.conf — layered by directory precedence, resolved with btool. → detailsRecommended Materials (official + community consensus)
Splunk's own blueprint names its preparation sources directly: the Splunk How-To YouTube channel, Splunk Docs, your own hands-on Splunk experience, and — as a non-exhaustive suggestion — the Splunk Enterprise System Administration and Splunk Enterprise Data Administration courses from the official Admin learning path.
Study Plan (Quick Version)
- Map to the blueprint: treat Splunk Indexes, Distributed Search, and Forwarder Management as your three "big" topics (10% each); everything else is 5%.
- Learn by doing: a home lab beats reading. Install Splunk Enterprise, add a universal forwarder, wire up a deployment server, and watch data move through the indexing pipeline.
- Active recall: after each topic, close the docs and reproduce the relevant conf-file stanza from memory. Drill the confusable pairs (hot vs warm bucket, universal vs heavy forwarder, license master vs license peer).
- Practice + docs: verify every practice question's answer against current Splunk Docs before trusting it — third-party dump sites for this exam are common and unreliable.
→ Detailed retention-based plan (§22) · ⚡ Cram Jam one-pager · → Exam-day tips & traps (§23)
Part 2 — Thorough Study Guide
1 · Exam Details & Logistics
SPLK-1003 is a 56-question, 60-minute, Professional-level exam delivered by Pearson VUE. Splunk's blueprint notes the 60 minutes "includes 3 minutes to review the exam agreement," so budget closer to 57 minutes of actual answering time — roughly 61 seconds per question.
Prerequisite — the evidence
- The official blueprint PDF (
splunk-test-blueprint-enterprise-admin.pdf) states on its final page, under the heading "The prerequisite exam for this certification is:" — Splunk Core Certified Power User. This is stated as a fact, not a recommendation. - The Splunk certification-track marketing page lists "Splunk Core Certified Power User" under the certification's prerequisites, consistent with the blueprint.
- Community threads on r/Splunk corroborate this from the registration side: multiple posters independently describe Power User as a "mandatory prerequisite" you must hold before Pearson VUE will let you book the Admin exam.
Passing score & question format
- Splunk does not publish an exact passing percentage for SPLK-1003 on its public pages or in the blueprint. Older, retired Splunk exams (e.g. the legacy Fundamentals track) had a published 75% cut score, but that does not carry over to the current Core/Admin track — treat any specific number you see on third-party sites as unverified.
- Questions are multiple choice and multiple response (some require selecting more than one correct answer).
Logistics
| Item | Detail |
|---|---|
| Price | $130 USD per exam attempt |
| Delivery | Pearson VUE test center or online proctored |
| Recertification | All Splunk certifications share a three-year life cycle from the date of the highest certification exam passed, per Splunk's official Training & Certification FAQ. Recertify by re-passing a current exam or by earning a higher certification in the same track before expiry. |
| Splunk version alignment | Splunk does not publish an explicit "this exam targets version X.Y" statement on the blueprint or certification page. Current third-party courses and books (Udemy, Packt's Splunk 9.x Enterprise Certified Admin Guide) target the Splunk Enterprise 9.x line. Treat this as the best available signal, not an official commitment, and verify version-specific behavior against current docs.splunk.com before the exam. |
2 · Splunk Admin Basics 5%
Blueprint item 1.1: identify Splunk components. This is the smallest topic but underlies everything else — know each component's job before you study how they're configured.
| Component | Role |
|---|---|
| Forwarder (UF/HF) | Collects and ships data to indexers. Doesn't index locally by default. |
| Indexer | Parses, transforms, and stores data in indexes as buckets. Handles the searches sent to it by a search head. |
| Search head | Coordinates searches across one or more indexers ("search peers") and renders results, dashboards, and reports. |
| License manager (was: license master) | Tracks daily indexing volume against your license across all peers in a license pool. |
| Deployment server | Pushes apps/configuration to forwarders and other Splunk instances registered as deployment clients. |
| Cluster manager (was: master node) | Coordinates indexer clustering — bucket replication and search factor across a peer cluster (out of deep scope for this exam, but you should recognize the term). |
| Monitoring Console | Built-in app for observing the health of your whole deployment — see §19. |
First-time setup checklist
Conceptual steps every fresh Splunk Enterprise install goes through, useful for orienting yourself in "what order does this happen in" style questions:
- Install and start Splunk; set the initial admin password (recent versions require this at first login rather than shipping a default).
- Apply a license (or start on the bundled trial), then set up license pools/stacks if you have more than one Enterprise license.
- Enable a receiving port on the indexer(s) so forwarders have somewhere to send data.
- Create indexes for your data before pointing inputs at them — Splunk won't silently create a named index you haven't defined.
- Install forwarders on data sources, configure inputs and outputs, and confirm data is arriving with a simple search.
- Set up users/roles (or wire up LDAP/SAML) before onboarding a wider set of users.
- Stand up a deployment server once you have more than a handful of forwarders to manage by hand.
Common deployment topologies
| Topology | Shape | When |
|---|---|---|
| Single instance | One Splunk Enterprise instance does everything — collects, indexes, searches. | Labs, tiny deployments, proof of concept. |
| Distributed, non-clustered | Separate search head(s) and indexer(s), forwarders feeding data in. | Small-to-medium production, no HA requirement on the indexing tier. |
| Indexer cluster | Multiple indexers replicate each other's buckets, coordinated by a cluster manager, for data resilience. | Production deployments needing indexing HA / no data loss on indexer failure. |
| Search head cluster | Multiple search heads share configuration/scheduling via a captain, coordinated app pushes via a deployer. | Production deployments needing search HA and horizontal search scaling. |
3 · License Management 5%
Blueprint items: 2.1 identify license types, 2.2 understand license violations.
License types
- Enterprise license — the standard paid license, metered by daily indexing volume (GB/day), not by data retained or events.
- Free license — legacy single-instance, no auth/clustering/forwarding-to-multiple-indexers restrictions vary by version; largely superseded by short-lived trial licenses in current releases.
- Trial license — time-boxed, full-featured evaluation license.
- Developer license — for non-production development use.
- No-enforcement license — an Enterprise license variant (available since Splunk 6.5) that keeps search running even after license violations accumulate; see below.
Violations and enforcement — version-sensitive, verify against current docs
- A license violation warning is issued when a license pool's indexed volume exceeds its licensed daily quota. Under the classic (pre-6.5) enforcement model, accumulating 5 warnings within a rolling 30-day window would disable search on the license manager until the condition cleared.
- Since Splunk 6.5, Splunk Enterprise stacks can run under a no-enforcement license: search is not blocked even after 5 warnings in 30 days — the license manager still records itself as "in violation," but users can keep searching. This is a real behavior change from earlier versions and is exactly the kind of detail worth re-checking against current docs.splunk.com before the exam, since licensing policy is one of the areas Splunk has continued to adjust release over release.
- What counts against the license: indexed data volume — the uncompressed size of raw data written to an index. Search activity, retained history, and the number of users do not themselves consume license.
License pools and stacks
- A license stack groups licenses of the same type (e.g. all your Enterprise licenses) so their volumes add together into one total daily quota.
- A license pool subdivides a stack's total volume among specific groups of indexers (license peers) — e.g. reserving 50GB/day for a security team's indexers and 100GB/day for the rest, out of a single 150GB stack.
- Every indexer in a pool reports its indexed volume back to the license manager, which aggregates usage against the pool's quota and issues warnings when the pool exceeds it.
Checking license usage
Settings ▸ Licensing in Splunk Web shows current pools, stacks, peer status, and usage history. The CLI equivalent for a quick read is splunk list licenser-pools / splunk list licenser-groups.
Walking through a violation timeline (conceptual)
- Day 1: license pool exceeds its daily quota → warning 1 of 5 recorded.
- Warnings accumulate independently; if 5 occur inside a rolling 30-day window, the pool is "in violation."
- On a classic enforcement license: search on peers in that pool is disabled until the violation count drops out of the window or the license is fixed (more capacity purchased, or usage reduced).
- On a no-enforcement license (default behavior since 6.5 for eligible license types): step 3 doesn't happen — search stays available, but the manager still logs the violation for visibility/compliance.
4 · Splunk Configuration Files 5%
Blueprint items: 3.1 directory structure, 3.2 configuration layering, 3.3 configuration precedence, 3.4 use btool.
Directory structure
- All configuration lives in
.conffiles insideetc/directories, organized by app:$SPLUNK_HOME/etc/apps/<app>/{default,local}/, plus system-wide$SPLUNK_HOME/etc/system/{default,local}/. default/holds an app's shipped configuration — never hand-edit it.local/holds your customizations and always wins overdefault/within the same app/layer.- User-specific overrides live under
etc/users/<user>/<app>/local/— the most specific layer of all.
Directory layout at a glance
$SPLUNK_HOME/etc/
├── system/
│ ├── default/ # shipped baseline, never edit
│ └── local/ # your global overrides — highest precedence
├── apps/
│ └── <app_name>/
│ ├── default/ # app's shipped config
│ ├── local/ # your app-scoped overrides
│ └── bin/ # scripts for scripted inputs, etc.
├── users/
│ └── <user>/<app>/local/ # per-user, most specific
└── deployment-apps/ # apps a deployment server distributes
Layering and precedence (know this cold)
When the same stanza/attribute is defined in multiple places, Splunk merges files by combining all definitions and, for conflicts, applies a strict precedence order from highest to lowest:
etc/system/local/— highest precedence, always wins.etc/apps/<app>/local/for apps in the current user's context, ordered alphabetically by app name (later app names lose to earlier ones for the same setting) — with any app the user's active context belongs to considered before others.etc/users/<user>/<app>/local/— user-level, layered in relative to app precedence.etc/apps/<app>/default/— shipped app defaults.etc/system/default/— lowest precedence, the system-wide baseline shipped with Splunk.
system/local is truly global; app-level local only applies to instances/contexts running that app.btool
splunk btool <conf-file> list [--debug]shows the effective, merged configuration Splunk actually uses after layering — the single most useful troubleshooting command for "why isn't my setting taking effect."--debugadditionally prints, for each line, which file it came from — essential for finding a stray override.- Example:
splunk btool inputs list --debugshows every effectiveinputs.confstanza and its source file.
Worked example — resolving a conflict
Say indexes.conf sets frozenTimePeriodInSecs for the same index in three places:
# etc/system/default/indexes.conf
frozenTimePeriodInSecs = 188697600 # ~6 years, shipped default
# etc/apps/my_app/local/indexes.conf
frozenTimePeriodInSecs = 2592000 # 30 days
# etc/system/local/indexes.conf
frozenTimePeriodInSecs = 7776000 # 90 days
The effective value is 90 days — system/local always wins, regardless of what any app's local or any default file says. btool --debug would show all three lines and mark the system/local one as the winner.
default/ directory (gets clobbered on the next app upgrade — always edit local/). Assuming a change takes effect without a restart or reload — most conf-file changes need one or the other. Forgetting that a setting defined in multiple apps' local/ directories is resolved by app name precedence, not by which one you edited most recently.5 · Splunk Indexes 10%
Blueprint items: 4.1 index structure, 4.2 bucket types, 4.3 check index data integrity, 4.4 indexes.conf options, 4.5 the fishbucket, 4.6 apply a data retention policy. This is the single largest topic on the exam alongside Distributed Search and Forwarder Management.
Index structure
An index is a directory of buckets under $SPLUNK_HOME/var/lib/splunk/<index>/. Each bucket holds raw data plus the index files (.tsidx) Splunk builds over it. Indexes are defined and tuned in indexes.conf.
Bucket lifecycle — the core exam fact
Every bucket ages through five states. Only frozen is unsearchable by default:
| State | Description | Searchable? | Default location |
|---|---|---|---|
| Hot | Actively written to. An index can have several hot buckets open at once. | Yes | <index>/db/ |
| Warm | Rolled from hot; no longer written to, but many warm buckets can exist. | Yes | <index>/db/ |
| Cold | Rolled from warm, typically moved to cheaper/slower storage. | Yes | <index>/colddb/ |
| Frozen | Rolled from cold. Deleted by default, or archived if you configure a frozen path/script. | No | Deleted, or your configured archive location |
| Thawed | An archived frozen bucket restored back into the index. | Yes | <index>/thaweddb/ |
Key indexes.conf settings
| Setting | Controls |
|---|---|
homePath | Location for hot and warm buckets. |
coldPath | Location for cold buckets. |
thawedPath | Location for thawed buckets (required even if unused). |
maxDataSize | Max size per hot bucket before it rolls to warm (e.g. auto, auto_high_volume). |
maxHotSpanSecs | Max time span of data a hot bucket may cover before rolling. |
homePath.maxDataSizeMB / coldPath.maxDataSizeMB | Size ceiling for hot+warm and for cold buckets respectively. |
frozenTimePeriodInSecs | Age (default ~6 years) at which buckets roll to frozen — the primary retention-policy knob. |
maxTotalDataSizeMB | Size ceiling for the whole index; oldest data freezes first when exceeded. |
coldToFrozenDir / coldToFrozenScript | Archive frozen data instead of deleting it. |
frozenTimePeriodInSecs or the size-based maxTotalDataSizeMB. Set both deliberately — don't rely on defaults for a production retention policy.Data integrity
splunk check-integrity -index <name> -bucketid <id>verifies a bucket's data hasn't been tampered with, using per-bucket integrity hashes (feature must be enabled at index time viaenableDataIntegrityControlinindexes.conf).
The fishbucket
The fishbucket (internal index _thefishbucket, stored under $SPLUNK_HOME/var/lib/splunk/fishbucket/) tracks which files a forwarder/indexer has already read and how far into them (CRC + byte offset), so file-monitor inputs don't re-index the same data on restart. Clear it (via the fishbucket Splunk Web/REST controls or by deleting the splunk_private_db content) to force a full re-read of monitored files.
Creating and applying a retention policy — example
[app_logs]
homePath = $SPLUNK_DB/app_logs/db
coldPath = $SPLUNK_DB/app_logs/colddb
thawedPath = $SPLUNK_DB/app_logs/thaweddb
maxTotalDataSizeMB = 500000
frozenTimePeriodInSecs = 7776000 # 90 days
This index freezes data older than 90 days or once it exceeds 500GB total, whichever happens first. Create it in Splunk Web (Settings ▸ Indexes ▸ New Index) or by editing indexes.conf directly and restarting/reloading.
Internal indexes to know
| Index | Holds |
|---|---|
_internal | Splunk's own operational logs — splunkd.log, metrics.log, and related diagnostics. |
_audit | User activity and search auditing records. |
_introspection | OS-level resource usage data collected by Splunk itself (CPU, disk, etc.). |
main | The default index for data with no index explicitly set — not itself "internal," but the default landing spot worth naming explicitly in production so you don't end up dumping everything into it by accident. |
srchIndexesAllowed = * does not include internal indexes, since they all start with an underscore. Grant _* explicitly if a role (e.g. an admin/troubleshooting role) needs to search them.6 · Splunk User Management 5%
Blueprint items: 5.1 describe user roles, 5.2 create a custom role, 5.3 add Splunk users.
Built-in roles
| Role | Typical capabilities |
|---|---|
| admin | Nearly all capabilities: manage users/roles, install apps, edit all configuration. |
| power | Edit all shared objects, schedule searches/alerts, run real-time searches — but not full server administration. |
| user | Run searches, create and own their own private knowledge objects, limited by search quotas. |
Custom roles — authorize.conf
Roles are stanzas in authorize.conf. A role is defined by a set of enabled capabilities plus optional search-scoping attributes:
| Attribute | Meaning | Default |
|---|---|---|
<capability> = enabled | Grants that capability to the role. | disabled |
importRoles | Semicolon-list of roles this role inherits capabilities from. | none |
srchFilter | A search string automatically appended to filter every search this role runs. | none |
srchIndexesDefault | Indexes searched when the user doesn't specify one. | not set |
srchIndexesAllowed | Indexes the role is permitted to search at all (supports wildcards; * excludes internal indexes — use _* for those). | not set |
srchDiskQuota | Max MB of disk a role's search jobs may consume. | 100 |
srchJobsQuota | Max concurrently running historical searches. | 3 |
Role inheritance
importRolesmakes a role inherit every capability of the roles it imports — e.g. a custom "analyst" role built withimportRoles = userstarts from everythingusercan do and layers on extra capabilities.- When a user holds multiple roles, they get the union of capabilities and the most permissive quota values across those roles (e.g. the largest
srchJobsQuota).
Common capabilities reference
Capabilities in authorize.conf are individually toggled per role. A handful come up repeatedly in Admin scenarios:
| Capability | Grants |
|---|---|
admin_all_objects | Access to all knowledge objects regardless of owner/sharing — a hallmark of the built-in admin role. |
edit_user | Create/edit/delete users. |
edit_roles | Create/edit/delete roles. |
schedule_search | Schedule saved searches/alerts to run automatically. |
rtsearch | Run real-time searches. |
indexes_edit | Create/edit index definitions. |
edit_forwarders | Configure forwarding (outputs.conf) via Splunk Web. |
edit_deployment_client | Configure this instance as a deployment client. |
edit_deployment_server | Configure and manage a deployment server. |
delete_by_keyword | Required (with deleteIndexesAllowed) to delete indexed data. |
Adding users
- Splunk Web: Settings ▸ Users and Authentication ▸ Users ▸ New User — assign one or more roles, set a password, optionally force a password reset.
- CLI:
splunk add user <name> -role <role> -password <pw>. - Users can also be provisioned centrally via LDAP or SAML — see §7.
Example custom-role stanza
[role_analyst]
importRoles = user
srchIndexesAllowed = security;network_*
srchIndexesDefault = security
srchDiskQuota = 500
srchJobsQuota = 8
This role inherits everything user can do, is restricted to the security index and any index starting with network_, defaults its searches to security when no index is specified, and gets a larger disk/job quota than the built-in user role's defaults.
7 · Splunk Authentication Management 5%
Blueprint items: 6.1 integrate with LDAP, 6.2 list other authentication options, 6.3 describe steps to enable MFA.
LDAP integration
- Configured under Settings ▸ Users and Authentication ▸ Authentication Method ▸ LDAP, backed by
authentication.conf. - You map LDAP groups to Splunk roles — Splunk roles themselves are still defined locally in
authorize.conf; LDAP only supplies identity and group membership. - Requires a bind DN/password to query the directory, plus mapping of LDAP attributes (user, group, member) to how your directory is structured.
Other authentication options
| Method | Notes |
|---|---|
| Splunk built-in auth | Local users/passwords stored and managed by Splunk itself — the default. |
| LDAP | Centralized directory auth (Active Directory, OpenLDAP, etc.). |
| SAML | SSO federation with an external Identity Provider; supports IdP-initiated and SP-initiated flows. |
| Scripted authentication | Custom auth via a scripted API for identity systems with no native support. |
| Multifactor authentication (MFA) | Layered on top of one of the above; see below. |
Enabling MFA
- MFA is configured as an additional layer alongside an existing authentication scheme (Splunk built-in, LDAP, or SAML) — it is not itself a standalone auth method.
- General steps: install/enable the relevant MFA authenticator (e.g. Duo or another supported provider), configure it in
authentication.confunder an MFA stanza, associate it with the authentication scheme in use, and assign it per role/user as required. - MFA availability and specific supported providers vary by Splunk Enterprise version and licensing — verify current supported providers against docs.splunk.com before relying on specifics.
SAML specifics worth knowing
| Concept | Meaning |
|---|---|
| Identity Provider (IdP) | External system that authenticates the user and asserts identity to Splunk (the Service Provider). |
| SP-initiated vs IdP-initiated | Whether the login flow starts at the Splunk login page or at the IdP's own portal. |
| Role mapping | Like LDAP, SAML maps IdP group/attribute values to Splunk roles defined locally in authorize.conf. |
authorize.conf, regardless of how the user authenticated.8 · Getting Data In 5%
Blueprint items: 7.1 basic input settings, 7.2 list forwarder types, 7.3 configure the forwarder, 7.4 add an input to a UF using CLI. This is the first of nine Getting-Data-In-flavored topics — treat §8, §10–§18 as one connected story: collect → forward → parse → transform.
Basic input settings
Every input, regardless of type, can set: index (destination index), sourcetype (data format identity), host (event host value), and optionally source overrides — all definable in inputs.conf or via Splunk Web.
Forwarder types
| Type | What it does | When to use |
|---|---|---|
| Universal Forwarder (UF) | Lightweight agent: collects and forwards raw data. No parsing, no indexing, minimal CPU/memory footprint. | Default choice on every data source — install broadly. |
| Heavy Forwarder (HF) | A full Splunk Enterprise instance running in forwarding mode: can parse, index-time transform, and route data before forwarding. | When you need pre-forwarding parsing/filtering/routing, or to talk to sources a UF can't reach directly (e.g. some HEC/modular-input-only sources). |
Configuring the forwarder
- A forwarder needs an
outputs.confpointing at one or more receiving indexers (or an intermediate HF), and one or moreinputs.confstanzas describing what to collect. - Enable the forwarding role on a full Splunk instance to make it act as a heavy forwarder:
splunk enable app SplunkForwarderdisables local indexing so it behaves like a forwarder, or manage forwarding directly viaoutputs.conf.
Adding an input to a UF via CLI
splunk add monitor /var/log/messages -index main -sourcetype syslog
splunk add tcp 9997 -sourcetype my_tcp_data
splunk add forward-server indexer1.example.com:9997
splunk list forward-server
splunk list monitor
9 · Distributed Search 10%
Blueprint items: 8.1 how distributed search works, 8.2 roles of search head & search peers, 8.3 configure a distributed search group, 8.4 search head scaling options.
How it works
- A search head dispatches a search to all (or a scoped subset of) its search peers (indexers). Each peer runs the search locally over its own data and streams partial results back; the search head merges the results.
- The search head distributes the knowledge bundle — a replicated snapshot of the search head's knowledge objects (props/transforms, lookups, saved searches, etc.) — to each peer so peers can apply the same field extractions and knowledge at search time.
What's in the knowledge bundle
| Contents | Why peers need it |
|---|---|
| Field extractions, aliases, calculated fields | So each peer parses/enriches its local data identically to what the search head expects. |
| Lookups & lookup tables | Peers evaluate | lookup commands locally against their portion of the data. |
| Event types & tags | So eventtype=/tag= filters match consistently across peers. |
| Macros | Peers need the macro definition to expand it the same way the search head would. |
Bundle replication happens automatically whenever knowledge objects change; a large or frequently-changing bundle can itself become a performance bottleneck on very large peer counts — one reason bundle size is a metric worth watching in the Monitoring Console.
Roles
| Role | Job |
|---|---|
| Search head | Accepts search requests, dispatches to peers, merges/post-processes results, renders dashboards. |
| Search peer (indexer) | Stores data and executes the peer-local portion of a distributed search against it. |
Configuring a distributed search group — distsearch.conf
- Add a search peer: Settings ▸ Distributed Search ▸ Search peers ▸ New Search Peer, or CLI
splunk add search-server -host <indexer:mgmt-port> -auth <user:pass>. distsearch.confalso lets you define named server classes/groups of peers and tune bundle replication behavior.
Search head scaling options
| Option | Use when |
|---|---|
| Single search head | Small deployments, low concurrent-user count. |
| Multiple independent search heads | Different user populations pointed at the same indexer tier, no shared scheduling/config needed. |
| Search Head Cluster (SHC) | High availability and horizontal scaling for search — a captain-coordinated group of search heads sharing configuration and scheduling, with a separate deployer pushing apps to the cluster. |
Common distributed search failure symptoms
| Symptom | Likely cause |
|---|---|
| Peer missing from results, no error shown | Peer disconnected or knowledge bundle replication failed silently — check Settings ▸ Distributed Search ▸ Search peers status column. |
| Field extractions inconsistent across peers | Stale or unreplicated knowledge bundle on one peer. |
| Search hangs / very slow | One overloaded or network-degraded peer holding up the merge; check that peer's metrics.log. |
10 · Getting Data In — Staging 5%
Blueprint items: 9.1 the three phases of Splunk indexing, 9.2 list Splunk input options.
The three phases
| Phase | What happens |
|---|---|
| 1 · Input | Data is read from its source (file, network, script, agentless) and given basic metadata: host, source, sourcetype. |
| 2 · Parsing | Data is broken into events, timestamps extracted, line-breaking applied, and index-time transforms (mask/route/reject) run. Happens once, on the first full Splunk instance the data reaches (a heavy forwarder or the indexer). |
| 3 · Indexing | Parsed events are written to disk as index files (.tsidx) inside buckets, and become searchable. |
Input options
Files/directories (monitor), network ports (TCP/UDP), scripted inputs, Windows inputs (WMI, Event Log, Registry, Perfmon), and agentless inputs (HTTP Event Collector). Each is detailed in its own topic below (§13–§15).
11 · Configuring Forwarders 5%
Blueprint items: 10.1 configure forwarders, 10.2 identify additional forwarder options.
outputs.conf essentials
[tcpout]
defaultGroup = my_indexers
[tcpout:my_indexers]
server = idx1.example.com:9997, idx2.example.com:9997
autoLB = true
autoLBFrequency = 30
useACK = true
| Setting | Meaning |
|---|---|
autoLB | Automatic load balancing across the listed indexers — the only supported forwarding mode; plain round-robin is not. |
autoLBFrequency | Seconds between switching the active receiver (default 30s); the forwarder switches at a safe logical event boundary, not mid-stream. |
forceTimebasedAutoLB | Forces a hard switch to a new indexer every autoLBFrequency cycle even mid-stream (default false). |
useACK | Enables indexer acknowledgment — the forwarder waits for confirmation that data was received, indexed, and written to disk (or relayed and acked, for an intermediate HF) before considering it delivered. Increases delivery reliability at some throughput cost. |
maxQueueSize | Output queue size; sized relative to autoLBVolume to avoid blocking. |
Additional forwarder options
- Indexer discovery — instead of hardcoding indexer addresses, a forwarder can ask a cluster manager for the current list of available peers, which simplifies configuration in clustered/elastic indexer tiers.
- Compression (
compressed = true) reduces bandwidth between forwarder and receiver. - SSL settings encrypt forwarder-to-indexer traffic.
- Volume-based load balancing (
autoLBVolume) switches receivers based on bytes sent rather than (or in addition to) elapsed time.
Indexer discovery example
# On the forwarder, outputs.conf
[indexer_discovery:my_cluster]
pass4SymmKey = <shared secret>
master_uri = https://cluster-manager.example.com:8089
[tcpout:my_cluster_group]
indexerDiscovery = my_cluster
autoLB = true
The forwarder authenticates to the cluster manager with a shared key, then receives and maintains its own list of current indexer peers — so adding or removing indexers from the cluster doesn't require touching every forwarder's outputs.conf.
12 · Forwarder Management 10%
Blueprint items: 11.1 explain deployment management, 11.2 describe the Splunk Deployment Server, 11.3 manage forwarders using deployment apps, 11.4 configure deployment clients, 11.5 configure client groups, 11.6 monitor forwarder management activities. Tied with Splunk Indexes and Distributed Search as the heaviest single topic.
Deployment management, conceptually
The deployment server (DS) centrally pushes configuration bundles ("deployment apps") to a fleet of deployment clients (typically universal forwarders), grouped into server classes so different fleets get different app sets — e.g. all Linux web servers get one syslog-monitoring app, all Windows hosts get a different WinEventLog app.
Server-side: serverclass.conf
[serverClass:linux_webservers]
whitelist.0 = *.web.example.com
[serverClass:linux_webservers:app:syslog_inputs]
restartSplunkd = true
- Deployment apps live under
$SPLUNK_HOME/etc/deployment-apps/on the DS. - Server classes match clients by hostname, IP, DNS name, or a client-supplied
clientName— viawhitelist/blacklistpatterns. - Whichever apps a matching server class lists get pushed into the client's
$SPLUNK_HOME/etc/apps/.
Client-side: deploymentclient.conf
| Setting | Meaning | Default |
|---|---|---|
targetUri (under target-broker:deploymentServer) | Address of the deployment server. | — |
clientName | Name the DS can filter server classes on, independent of hostname/IP. | deploymentClient |
phoneHomeIntervalInSecs | How often the client checks in with the DS for new/changed content. | 60 seconds |
repositoryLocation | Where downloaded apps land locally. | $SPLUNK_HOME/etc/apps |
Monitoring forwarder management activity
- Splunk Web: Settings ▸ Forwarder Management shows connected clients, their phone-home status, server class membership, and app deployment status per client.
- The Monitoring Console (§19) has dedicated Forwarder Management health views.
reloadDSOnAppInstall(client-side) can trigger a hierarchical DS reload when a deployment client that is itself also a DS installs an app — an advanced, rarely-needed setting.
Troubleshooting a client that won't check in
- Confirm
deploymentclient.confexists and is enabled (disabled = false) on the client, with a correcttargetUripointing at the DS's management port (default 8089). - Verify network reachability from client to DS on that port — a firewall block here is the single most common cause of a client never appearing.
- Check whether
phoneHomeIntervalInSecshas actually elapsed yet (default 60s) — a client that just started may not have checked in on your first look. - On the DS, confirm the client actually matches a defined server class's
whitelist/blacklistpattern — a client that never matches any class won't receive any apps, though it may still show as connected. - Use
splunk reload deploy-serverafter editingserverclass.confso the DS picks up the change without a full restart.
13 · Monitor Inputs 5%
Blueprint items: 12.1 create file/directory monitor inputs, 12.2 optional settings, 12.3 deploy a remote monitor input.
Basic stanza
[monitor:///var/log/app/*.log]
index = app_logs
sourcetype = app_log
disabled = false
Common optional settings
| Setting | Purpose |
|---|---|
whitelist / blacklist | Regex to include/exclude specific files within a monitored directory. |
recursive | Whether to descend into subdirectories (default true). |
followTail | Skip existing content and only index new data going forward — useful for very large pre-existing files you don't want to backfill. |
crcSalt | Adds extra uniqueness to the fishbucket's file-identity hash — needed when multiple files can share identical starting content (e.g. rotated logs) so Splunk doesn't treat them as already-seen. |
ignoreOlderThan | Skip files whose modtime is older than the given age — avoids reprocessing stale archives. |
Deploying a remote monitor input
Push a monitor stanza to remote forwarders by packaging it as a deployment app's inputs.conf and distributing it via the deployment server (§12) — you don't hand-configure each forwarder individually at scale.
14 · Network and Scripted Inputs 5%
Blueprint items: 13.1 create TCP/UDP inputs, 13.2 optional network input settings, 13.3 create a basic scripted input.
Network inputs
[tcp://9514]
sourcetype = syslog
index = network
[udp://514]
sourcetype = syslog
index = network
| Setting | Purpose |
|---|---|
connection_host | Whether the event host field comes from DNS, IP, or is left as configured (dns/ip/none). |
no_priority_stripping | For syslog-style TCP/UDP data, whether to strip the leading syslog priority marker. |
queueSize | Input-side queue sizing for network listeners under load. |
useACK, not raw UDP.| Common default port | Use |
|---|---|
| 514 | Traditional syslog (UDP or TCP) |
| 9997 | Splunk-to-Splunk forwarding (indexer receiving port) |
| 8089 | Splunk management/REST port |
| 8000 | Splunk Web (default, HTTP unless SSL enabled) |
| 8088 | HTTP Event Collector |
Scripted inputs
[script://./bin/my_script.sh]
sourcetype = my_script_data
interval = 300
index = main
A scripted input runs an executable/script on the configured interval (seconds, or cron syntax) and indexes whatever it writes to stdout. The script must live in an app's bin/ directory and be referenced with a relative path.
15 · Agentless Inputs 5%
Blueprint items: 14.1 create WMI inputs, 14.2 describe HTTP Event Collector.
Windows Management Instrumentation (WMI) inputs
- Lets a Splunk instance (typically a heavy forwarder) pull Windows event logs and performance data remotely from other Windows hosts, without installing a forwarder on each of them — hence "agentless."
- Configured via
wmi.conf; requires appropriate Windows credentials/permissions and WMI connectivity (DCOM/RPC) between the collecting instance and the target hosts. - Tradeoff versus a UF on every host: fewer agents to manage, but higher load on the collector and dependency on WMI/network reachability.
WMI troubleshooting checklist
| Symptom | Check |
|---|---|
| No data from a target host | Firewall allowing DCOM/RPC (typically TCP 135 + dynamic high ports) between collector and target. |
| Access denied | Collecting account has the right local/WMI permissions on the target host. |
| Partial data / gaps | Target host's Windows Event Log service or WMI service itself stopped or overloaded. |
HTTP Event Collector (HEC)
- A token-authenticated HTTP(S) endpoint (default port 8088) that lets applications send events (JSON or raw) directly into Splunk without any forwarder at all.
- Enabled under Settings ▸ Data Inputs ▸ HTTP Event Collector; each token can be scoped to a specific index/sourcetype and enabled/disabled independently.
- Common use: cloud services, containers, and lightweight app instrumentation that can make an HTTP call but can't run a forwarder process.
Example HEC request
curl -k https://splunk.example.com:8088/services/collector/event \
-H "Authorization: Splunk 1234-5678-ABCD-EFGH" \
-d '{"event": "hello world", "sourcetype": "my_hec_data", "index": "app_events"}'
| Concept | Detail |
|---|---|
| Token | Per-source credential; can be individually enabled/disabled and scoped to a default index/sourcetype. |
| Event vs raw endpoint | /services/collector/event expects Splunk's JSON envelope; /services/collector/raw accepts arbitrary raw text. |
| Indexer acknowledgment | HEC supports its own ack mechanism so a sender can confirm an event was durably received. |
16 · Fine Tuning Inputs 5%
Blueprint items: 15.1 default input-phase processing, 15.2 configure input-phase options (sourcetype fine-tuning, character-set encoding).
- At input time, Splunk does minimal processing: it tags data with default
host/source/sourcetypemetadata based on the input stanza and, where possible, source-based sourcetype heuristics. - Sourcetype fine-tuning — override the auto-detected sourcetype per input (
sourcetype = my_type) or dynamically per event later at parse time (§17) withprops.conf/transforms.confrules keyed on source or content. - Character-set encoding — set
CHARSETinprops.conf(e.g.UTF-8,ISO-8859-1) when a source's encoding isn't auto-detected correctly, to avoid garbled/mojibake events.
Example sourcetype override by source
# props.conf on the collecting instance
[source::/var/log/app/*.json]
sourcetype = app_json
CHARSET = UTF-8
Stanzas can key off [sourcetype], [host::<pattern>], or [source::<pattern>] — source and host stanzas let you fine-tune settings for a subset of data that shares a sourcetype but needs different handling (e.g. one host's clock is in a different time zone).
17 · Parsing Phase and Data 5%
Blueprint items: 16.1 default parsing processing, 16.2 event line breaking, 16.3 timestamp/timezone extraction, 16.4 use Data Preview.
Default parsing behavior
Parsing turns a raw stream into discrete, timestamped events: applying line-breaking rules, extracting or assigning a timestamp, and running any configured index-time transforms — all before the event is written to a bucket.
Event line breaking — props.conf
| Setting | Purpose |
|---|---|
SHOULD_LINEMERGE | Whether Splunk may merge multiple raw lines into one event (e.g. multi-line stack traces). Default true, but for well-formed one-line-per-event data, set to false for performance. |
LINE_BREAKER | Regex marking where one event ends and the next begins. |
BREAK_ONLY_BEFORE / BREAK_ONLY_BEFORE_DATE | Alternative line-breaking anchored to a pattern, often a timestamp format, at the start of a new event. |
MAX_EVENTS | Hard cap on lines merged into a single event, to bound runaway merges from malformed data. |
Timestamps and time zones
- Splunk tries to auto-detect a timestamp in each event; you can pin this down with
TIME_PREFIX,TIME_FORMAT, andMAX_TIMESTAMP_LOOKAHEADinprops.conf. - If no timestamp is found or extraction is disabled, Splunk falls back to using the time the event was indexed.
- Time zone is taken from the event itself (if present in the timestamp), otherwise from the
TZsetting on the input/sourcetype, otherwise from the forwarder/indexing host's system time zone — set this explicitly for any source whose local clock/timezone context Splunk can't infer.
Data Preview
Settings ▸ Data Inputs ▸ [add a new input] ▸ ... or the standalone Data Preview tool lets you see how Splunk will break a sample of raw data into events and extract fields before committing an input configuration — the intended way to validate line-breaking and timestamp settings without indexing real data first.
Example line-breaking & timestamp props.conf
[my_multiline_app_log]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{2}-\d{2})
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 25
TZ = America/New_York
Here LINE_BREAKER starts a new event only at a line beginning with a date stamp, so multi-line stack traces stay attached to their originating log line — a very common real-world pattern worth being able to read at a glance.
18 · Manipulating Raw Data 5%
Blueprint items: 17.1 how data transformations are defined/invoked, 17.2 use props.conf/transforms.conf to mask/route/override, 17.3 use SEDCMD.
How transformations work
An index-time transform is defined in transforms.conf (a regex + action) and then invoked from a sourcetype/source/host stanza in props.conf via a TRANSFORMS-<name> attribute. Because these run at parse time, they affect what actually gets written to disk — unlike search-time knowledge objects, which only affect what a search sees.
What transforms can do
| Goal | Mechanism |
|---|---|
| Mask or delete raw data before indexing | transforms.conf regex with DEST_KEY = _raw and a FORMAT that replaces the matched text. |
| Override sourcetype or host based on event content | DEST_KEY = MetaData:Sourcetype or MetaData:Host. |
| Route events to a specific index by content | DEST_KEY = _MetaData:Index. |
| Prevent unwanted events from being indexed entirely | DEST_KEY = queue, FORMAT = nullQueue — the classic "drop this data" pattern. |
# transforms.conf
[drop_debug_events]
REGEX = DEBUG
DEST_KEY = queue
FORMAT = nullQueue
# props.conf
[my_sourcetype]
TRANSFORMS-drop = drop_debug_events
SEDCMD
SEDCMD-<name> = s/regex/replacement/flags in props.conf runs a sed-style substitution directly against raw event text at index time — a lighter-weight way to redact or reshape text (e.g. masking credit card numbers) than a full transforms.conf stanza, for simple substitutions.
Second worked example — routing by content
# transforms.conf
[route_security_events]
REGEX = severity=(critical|high)
DEST_KEY = _MetaData:Index
FORMAT = security_high
# props.conf
[syslog]
TRANSFORMS-route = route_security_events
Any syslog event matching the regex gets routed into the security_high index instead of wherever the input's default index setting would have sent it — useful for separating high-priority data into an index with tighter access controls or different retention.
19 · Reference: CLI & Monitoring
CLI reference card
| Command | Does |
|---|---|
splunk start / splunk stop / splunk restart | Control the splunkd process. |
splunk reload deploy-server | Have a deployment server re-read serverclass.conf and app content without a full restart. |
splunk add monitor <path> -index <idx> -sourcetype <st> | Add a file/directory monitor input. |
splunk list forward-server | Show configured forwarding destinations and their connection status. |
splunk add forward-server <host:port> | Add a receiving indexer to a forwarder's outputs.conf. |
splunk btool <conf> list [--debug] | Show effective merged configuration (and its source file with --debug). |
splunk apply cluster-bundle | Push a configuration bundle change out to an indexer cluster (cluster-manager side). |
splunk edit user <name> -roles <role> | Change a user's role assignment from the CLI. |
splunk add search-server -host <host:port> | Add a distributed search peer. |
splunk check-integrity -index <idx> -bucketid <id> | Verify a bucket's data-integrity hashes. |
splunk list monitor | List all currently configured file/directory monitor inputs. |
splunk add tcp <port> -sourcetype <st> | Add a TCP network input. |
splunk enable listen <port> | Enable a receiving port on an indexer for forwarders to send to. |
splunk list licenser-pools | Show current license pools and their volumes. |
splunk show deploy-poll | Show which deployment server this instance is configured to poll, if any. |
Splunk Web menu quick reference
| Task | Path |
|---|---|
| Add/edit an index | Settings ▸ Indexes |
| Add/edit a role | Settings ▸ Users and Authentication ▸ Roles |
| Configure LDAP/SAML | Settings ▸ Users and Authentication ▸ Authentication Method |
| View/add search peers | Settings ▸ Distributed Search ▸ Search peers |
| Manage deployment clients | Settings ▸ Forwarder Management |
| Configure HEC tokens | Settings ▸ Data Inputs ▸ HTTP Event Collector |
| View licensing | Settings ▸ Licensing |
Monitoring and troubleshooting
| Tool / index | Use for |
|---|---|
| Monitoring Console | Built-in app aggregating health across the whole deployment — indexing performance, resource usage, forwarder connectivity, search head/indexer cluster status. |
_internal index | Splunk's own operational logs (splunkd.log, metrics.log, etc.) indexed as searchable data — the first place to search for "what went wrong." |
_audit index | Records of user actions and search activity for auditing/compliance. |
splunkd.log | Main daemon log — startup errors, license issues, input/forwarding failures. |
metrics.log | Periodic performance counters — queue fill levels, throughput, thread pool stats. First stop for "why is ingestion slow." |
Monitoring Console — key panels
| Panel area | What it shows |
|---|---|
| Indexing Performance | Indexing throughput, queue fill levels, and indexing latency per indexer. |
| Search Activity | Concurrent searches, search performance, skipped scheduled searches. |
| Resource Usage | CPU, memory, disk usage per instance. |
| Forwarder Monitoring | Which forwarders are connected, their last-seen time, and data volume sent. |
| License Usage | Daily volume against license quota, historical trend, and violation history. |
The Monitoring Console must be told which instances are indexers vs. search heads vs. forwarders (instance roles), either by asset discovery or manual configuration, before its panels populate meaningfully in a distributed deployment.
Common failure symptoms — first thing to check
| Symptom | Check first |
|---|---|
| Forwarder shows "not connected" in Forwarder Management | Network reachability to DS management port; phoneHomeIntervalInSecs hasn't elapsed yet; deployment client is actually enabled/configured. |
| Data isn't showing up in search | Fishbucket state (already-read?), input disabled flag, index value mismatch between input and search, license violation blocking indexing, queue blocking in metrics.log. |
| Config change "isn't taking effect" | btool to see the effective merged value and which file actually won precedence. |
| Search is slow across peers | Knowledge bundle size/replication lag; a single slow/overloaded search peer; unbounded search time range. |
20 · Decision Guides & Tables
Which component when
| Need | Use |
|---|---|
| Collect from thousands of endpoints cheaply | Universal Forwarder |
| Parse/filter/route data before it leaves the source tier | Heavy Forwarder |
| App/service can POST but can't run an agent | HTTP Event Collector |
| Pull Windows logs from hosts you can't install a UF on | WMI input |
| Push config to a whole fleet consistently | Deployment server + server classes |
| Scale search concurrency & HA for search | Search Head Cluster |
| See fleet-wide operational health at a glance | Monitoring Console |
Which conf file when
| Task | File |
|---|---|
| Define/tune an index and its retention | indexes.conf |
| Define a data input (monitor, TCP/UDP, scripted, WMI) | inputs.conf |
| Point a forwarder at receiving indexers, tune load balancing | outputs.conf |
| Control line breaking, timestamps, sourcetype rules, invoke transforms | props.conf |
| Define regex-based masking/routing/rejection rules | transforms.conf |
| Define roles, capabilities, search scoping/quotas | authorize.conf |
| Configure LDAP/SAML/scripted auth | authentication.conf |
| Define deployment-server server classes and app assignment | serverclass.conf |
| Point a deployment client at its deployment server | deploymentclient.conf |
| Manage distributed search peers/groups | distsearch.conf |
Bucket state quick check
| Question on exam | Answer |
|---|---|
| "Actively being written to" | Hot |
| "No longer written, still searchable, on primary storage" | Warm |
| "Rolled to cheaper storage, still searchable" | Cold |
| "Not searchable, deleted or archived" | Frozen |
| "Restored from archive, searchable again" | Thawed |
Universal Forwarder vs Heavy Forwarder, side by side
| Capability | UF | HF |
|---|---|---|
| Parses/transforms data before forwarding | No | Yes |
Runs index-time props.conf/transforms.conf | No | Yes |
| Resource footprint | Minimal | Full Splunk instance |
| Typical scale | Thousands of endpoints | Small number of tiered collectors |
| Can locally index data | No (forwards only) | Optional |
Which built-in role fits
| Need | Role (or custom role built from it) |
|---|---|
| Full server administration | admin |
| Build/schedule shared dashboards & alerts, no server admin | power |
| Run searches, own private objects only, scoped to specific indexes | user plus a custom role with srchIndexesAllowed set |
| Search everything including internal indexes for troubleshooting | Custom role importing user/power with srchIndexesAllowed = *;_* |
Terminology cheat sheet
| Term | Definition |
|---|---|
| Sourcetype | The format/data-type identity Splunk assigns to an event (e.g. access_combined); drives default parsing rules. |
| Source | Where the data physically came from — a file path, a network port, a script name. |
| Host | The name of the machine the data originated on. |
| Index | The named data store a bucket belongs to; the unit users scope searches to. |
| Knowledge bundle | Search head's knowledge objects (field extractions, lookups, saved searches, etc.) replicated to search peers. |
| Search peer | An indexer participating in a distributed search. |
| Deployment client | Any Splunk instance configured to receive apps/config from a deployment server. |
| Server class | A named group of deployment clients that receive the same set of deployment apps. |
| Stanza | A [bracketed] section header in a .conf file grouping related settings. |
| Capability | A single named permission (e.g. edit_user) that can be enabled on a role. |
Search-time vs index-time — where does a change apply
| Change | Applies at | Reversible? |
|---|---|---|
| Field extraction, tag, event type, alias (Power User knowledge objects) | Search time | Yes — just remove the object |
SEDCMD, TRANSFORMS-* masking/routing/dropping | Index time | No — raw event is altered/dropped permanently |
Sourcetype/host override via MetaData:* | Index time | No |
21 · Study Materials (Community Consensus)
What Splunk itself points you to
The official blueprint's "Exam Preparation" section names, verbatim: the Splunk How-To YouTube Channel, Splunk Docs, "your own Splunk experience," and — as a non-exhaustive suggested list — the Splunk Enterprise System Administration and Splunk Enterprise Data Administration courses from the Admin learning path. These are paid, instructor-led (or self-paced eLearning) courses; Splunk's education datasheets describe System Administration as covering license manager, indexers, and search heads, and Data Administration as covering forwarders and methods for getting remote data into Splunk indexers.
What the community adds
- Free path: Splunk's free eLearning modules (education.splunk.com), Splunk Docs — especially the Admin Manual and Getting Data In manual — and hands-on practice in a personal/free-tier Splunk instance.
- Paid third-party courses: multiple Udemy options exist; a two-course "System Admin 101 / Data Admin 201" track from Ableversity (instructor Hailie Shaw, developed under Michael Bentley) is one frequently mentioned option, and standalone SPLK-1003-branded Udemy prep courses also exist. Community opinion on Udemy depth is mixed — some posters found it too shallow on hard topics compared with official ILT and recommend it as a supplement, not a substitute.
- Reference while studying:
community.splunk.com(the official community forum) for real troubleshooting threads, and Quizlet decks for terminology drilling. - Home lab: repeatedly described as close to essential, not optional, for this specific exam — Admin topics like deployment management and forwarder configuration are hard to internalize from reading alone.
Contrast with Power User's prerequisite situation
It's worth stating the difference explicitly, since it's easy to carry the wrong assumption from one Splunk exam to the next. For SPLK-1002 (Power User), community and official sources agree there is no prerequisite exam — SPLK-1001 is only a recommended stepping stone. For SPLK-1003 (Admin), both the official blueprint and multiple independent r/Splunk posters agree the opposite: Power User is required, and Pearson VUE registration reportedly enforces it. Don't assume the two exams work the same way.
Known-bad sources — avoid
Sources
- splunk.com/en_us/training/certification-track/splunk-enterprise-certified-admin.html — certification-track overview page.
- splunk.com/en_us/pdfs/training/splunk-test-blueprint-enterprise-admin.pdf — official exam blueprint (topics, weights, prerequisite statement, exam prep list).
- splunk.com/en_us/training/faq.html — official Training & Certification FAQ (3-year certification life cycle).
- splunk.com/en_us/pdfs/training/splunk-enterprise-system-administration-course-description.pdf and the equivalent Data Administration course description PDF.
- r/Splunk thread "Preparing for Splunk 1003 Admin Certification - Seeking Free Resources" — home-lab and free-resource discussion.
- r/Splunk thread "Splunk Enterprise Certified Admin" (System Admin / Data Admin course + labs recommendation).
- r/Splunk thread "Splunk Enterprise Certified Admin - online training? Udemy or others?" — Power User prerequisite confirmation, docs.splunk.com Admin/Getting Data In manuals, community forum and Quizlet recommendations, mixed Udemy-quality opinions.
- r/Splunk thread "Splunk Enterprise System / Data Administration - Instructor Led, Worth it?" — ILT vs. self-study comparison, Ableversity course mention.
- docs.splunk.com / help.splunk.com pages for
authorize.conf,deploymentclient.conf, and bucket lifecycle ("How the indexer stores indexes") — used to verify the specific settings and defaults cited throughout this guide. - Splunk Community threads on no-enforcement licensing behavior (community.splunk.com) — used to verify the post-6.5 license violation behavior described in §3.
22 · Full Study Plan
Assumes you already hold Power User and have some Splunk search experience. Compress to ~2 weeks if you have prior admin/sysadmin background, or stretch to 5–6 weeks for a first-time admin. Each phase should end with hands-on lab work, not just reading.
| Phase | Focus | Actions |
|---|---|---|
| Week 1 Foundations | Admin basics, licensing, config files/precedence, indexes | Install Splunk Enterprise in your lab. Walk through btool on a fresh install. Create a custom index and watch a bucket roll hot→warm by tuning maxDataSize down for a test. Read the Admin Manual's licensing chapter. |
| Week 2 Users, auth, search | User management, authentication, distributed search | Create a custom role with scoped srchIndexesAllowed. If possible, wire up LDAP against a test directory (or at least read the exact config flow). Stand up a second indexer and configure it as a search peer. |
| Weeks 3–4 Getting Data In (the big half) | Forwarders, forwarder management, all input types, parsing, transforms | Install a UF on a second VM, forward to your indexer. Stand up a deployment server, define a server class, and push an app with a monitor input. Build a scripted input and a HEC token. Practice props.conf/transforms.conf masking and index routing; verify with Data Preview before applying. |
| Week 5 Consolidate | All topics, weak-area drilling | Re-read the blueprint topic by topic and self-rate 1–5 confidence on each; spend remaining time only on anything below a 4. Reproduce every conf-file example in this guide from memory. |
| Final days | Review only | Skim the Cram Jam, re-verify any version-sensitive fact (especially licensing enforcement) against current docs.splunk.com, sleep well. |
outputs.conf/serverclass.conf/deploymentclient.conf flow for getting a new log source from a fresh VM into a search — end to end, no notes.A repeatable daily routine
- 5–10 min warm-up: from memory, sketch yesterday's conf-file diagram before opening any notes.
- 30–40 min new learning: one blueprint topic at a time, in your lab, not just in docs.
- 10–15 min retrieval: close everything and write out the key stanza/settings for what you just learned; check against the real file.
- Weekend: a full end-to-end build (new VM → UF → DS-managed app → data flowing → search) under a timer.
23 · Exam-Day Tips & Common Traps
Technique
- ~61 seconds per question. 56 questions in a real ~57 minutes of answering time. Flag hard ones and return — don't sink 5 minutes into one.
- This exam is configuration-file-heavy. Where Power User tests SPL syntax, Admin tests "which conf file, which stanza, which attribute" — memorize the decision table in §20 cold.
- Read for the component doing the action: a question about pushing config to many hosts is a deployment-server question even if it doesn't say "deployment server" directly.
- Distinguish "identify/describe" from "configure/create" verbs in the blueprint wording — several topics (e.g. Splunk Admin Basics, Getting Data In — Staging) are conceptual-only ("identify," "list," "describe"), so don't over-invest in memorizing exact CLI syntax for those; save the syntax cramming for topics whose blueprint verbs are "configure" or "create."
Recurring traps
- Hot/warm/cold are all searchable; only frozen isn't (until thawed). A common wrong-answer bait is claiming cold buckets aren't searchable.
- Universal forwarder does not parse. Parsing/index-time transforms require a heavy forwarder or the indexer itself.
local/beatsdefault/, andsystem/localbeats everything — reversed answers are a classic distractor.- SCPs/roles: inheritance is additive, not restrictive.
importRolesadds capabilities; a role never loses capabilities by importing another role. - Index-time transforms are permanent — there's no search-time way to recover data masked or dropped via
transforms.conf/SEDCMD. - Phone-home default is 60 seconds, not instant — don't assume a deployment client picks up changes immediately.
- License violations don't necessarily block search on modern no-enforcement stacks — this is a frequently outdated assumption baked into older third-party questions; verify against current docs if a question hinges on it.
- WMI vs. UF for Windows: WMI is agentless/remote-pull; a UF installed on the Windows host itself is the more common, more reliable production pattern. Know both exist and when each applies.
- HEC needs no forwarder at all — don't reach for "install a UF" when the scenario says an application can already make an HTTP call.
- Deployment server ≠ indexer cluster manager. The DS pushes apps/config to forwarders and other clients; the cluster manager coordinates bucket replication between indexers. They're often confused because both "manage" other instances, but they solve different problems.
- autoLBFrequency default is 30 seconds, not the same as the deployment client's 60-second
phoneHomeIntervalInSecs— don't mix the two numbers up. - A custom role's quotas take the most permissive value across all roles a user holds — a common wrong answer assumes the most restrictive role wins.
- srchIndexesAllowed with
*excludes internal indexes. To let a role search_internal/_audit, you must explicitly add_*.
btool · "mask/drop/reroute before indexing" = transforms.conf/SEDCMD · "confirm delivery" = useACK.