Capstone: Investigating a Real Incident Start to Finish
Graylog
Chapter 10 (Capstone) · Investigating a Real Incident Start to Finish
No new fields, no new syntax — one realistic incident, traced from a vague first report through to a confirmed root cause, using nothing but the real tools Chapters 1 through 9 already covered.
The Ticket
"Account bullhorn-27515 says their Bullhorn sync stopped working overnight, and separately a couple of their recruiters say search results look off. Might be related, might not."
Step 1 — Confirm Real Activity (Chapter 2)
Real activity is present — the account and service are genuinely correct, ruling out a Chapter 2-style separator mistake before anything else.
Step 2 — Narrow by Severity, Correctly (Chapters 3 & 7)
Explicit OR-grouping, not a range comparison — deliberately avoiding Chapter 3's own off-by-one trap. A handful of real error-level messages turn up.
Step 3 — Rule Out Baseline Noise (Chapter 9)
Before trusting that error as the cause, check it against a known-healthy account:
It doesn't appear there — this error is genuinely specific to this account, not background noise.
Step 4 — Follow One Request (Chapter 4)
The full sequence around that one request shows a downstream call that itself failed — pointing toward something happening outside the search service entirely.
Step 5 — Pivot to the Integration Cluster (Chapters 2, 4 & 6)
That downstream failure looks Flux-shaped, not search-shaped. Widening scope from a single request to the whole integration cluster:
Several real failures show up on this cluster overnight. Picking one and following its own
correlationId (not requestId — this is a cross-service job, per Chapter 4's
own scope distinction) surfaces the same real Flux fields from Chapter 6 —
datumType:Candidate, a specific rmsId — confirming this is the same real sync
job the ticket originally described as broken.
Step 6 — Confirm It's Not a Candidate-Content False Positive (Chapter 9)
Since this investigation touched search directly, it's worth double-checking that nothing
here was actually just a free-text keyword coincidentally matching resume content rather than a genuine
code-path error — Chapter 9's own scout/search gotcha. Every query used in this capstone has been
structured (service, account, level, correlation IDs), not
generic free text, so this concern doesn't apply here — a real, deliberate design choice, not an
afterthought.
Building a Shareable Link
| Capstone step | Built using |
|---|---|
| Confirm activity | Chapter 2 — service, account |
| Narrow by severity | Chapter 3, Chapter 7 — level:(2 OR 3), correct grouping |
| Rule out noise | Chapter 9 — baseline noise check |
| Follow one request | Chapter 4 — requestId |
| Pivot to cluster | Chapter 2, Chapter 6 — daxtraResource, correlationId, Flux fields |
| Confirm no false positive | Chapter 9 — structured vs. free-text discipline |
| Shareable link | Real, explicitly unverified recipe from the cheat sheet |
Hands-On Exercises
Explain, in your own words, why Step 5 switches from requestId to
correlationId — what specifically changed about the nature of what's being traced?
Build the query for Step 3 (the baseline-noise check) as a full, real query,
substituting a literal exact error phrase, "upstream request failed", and a literal
known-healthy account, bullhorn-40021.
Explain why this chapter recommends Graylog's own built-in share/save-search feature over hand-building a URL using the recipe given here, even though the recipe is presented in full.
📄 View solutionChapter 10 (Capstone) Quick Reference
- A full incident investigation built from every real field and technique in Chapters 1-9, with no new syntax introduced
- The shareable-link URL recipe is explicitly unverified — Graylog's own built-in share/save-search feature is the safer real alternative
Course Complete
Graylog is now complete — 10/10 chapters, from the shape of the search box itself through to a full, real incident investigation, grounded throughout in genuine day-to-day usage at logger.daxtra.io.