Capstone: Investigating a Real Incident Start to Finish

Graylog

Chapter 10 (Capstone) · Investigating a Real Incident Start to Finish

No new fields, no new syntax — one realistic incident, traced from a vague first report through to a confirmed root cause, using nothing but the real tools Chapters 1 through 9 already covered.

The Ticket

"Account bullhorn-27515 says their Bullhorn sync stopped working overnight, and separately a couple of their recruiters say search results look off. Might be related, might not."

Step 1 — Confirm Real Activity (Chapter 2)

service:search AND account:bullhorn-27515

Real activity is present — the account and service are genuinely correct, ruling out a Chapter 2-style separator mistake before anything else.

Step 2 — Narrow by Severity, Correctly (Chapters 3 & 7)

service:search AND account:bullhorn-27515 AND level:(2 OR 3)

Explicit OR-grouping, not a range comparison — deliberately avoiding Chapter 3's own off-by-one trap. A handful of real error-level messages turn up.

Step 3 — Rule Out Baseline Noise (Chapter 9)

Before trusting that error as the cause, check it against a known-healthy account:

"<the exact error text>" AND account:<known-healthy account>

It doesn't appear there — this error is genuinely specific to this account, not background noise.

Step 4 — Follow One Request (Chapter 4)

requestId:"<id from the error line>"

The full sequence around that one request shows a downstream call that itself failed — pointing toward something happening outside the search service entirely.

Step 5 — Pivot to the Integration Cluster (Chapters 2, 4 & 6)

That downstream failure looks Flux-shaped, not search-shaped. Widening scope from a single request to the whole integration cluster:

daxtraResource:"rms-integration-cluster-eu-west-1" AND level:(2 OR 3)

Several real failures show up on this cluster overnight. Picking one and following its own correlationId (not requestId — this is a cross-service job, per Chapter 4's own scope distinction) surfaces the same real Flux fields from Chapter 6 — datumType:Candidate, a specific rmsId — confirming this is the same real sync job the ticket originally described as broken.

Step 6 — Confirm It's Not a Candidate-Content False Positive (Chapter 9)

Since this investigation touched search directly, it's worth double-checking that nothing here was actually just a free-text keyword coincidentally matching resume content rather than a genuine code-path error — Chapter 9's own scout/search gotcha. Every query used in this capstone has been structured (service, account, level, correlation IDs), not generic free text, so this concern doesn't apply here — a real, deliberate design choice, not an afterthought.

Building a Shareable Link

Unverified — treat as a starting guess, not a confirmed recipe
Nobody has confirmed that a link built this way actually loads the expected search when clicked. The real, safer alternative is Graylog's own built-in share/save-search feature in the UI, which generates a working link without any manual encoding. What follows is recorded as a real starting point only, not a trusted shortcut.
# Base URL https://logger.daxtra.io/search # Query parameters ?q=<URL-encoded query> # space -> +, : -> %3A, " -> %22, < -> %3C &rangetype=relative&from=<seconds-ago> # from=300 (5 min), from=900 (15 min), from=3600 (1 hour) # Worked example, encoding "service:scout AND daxtraResource:"scout-cluster-eu-west-2" AND level:<3" https://logger.daxtra.io/search?q=service%3Ascout+AND+daxtraResource%3A%22scout-cluster-eu-west-2%22++AND+level%3A%3C3&rangetype=relative&from=900
Capstone stepBuilt using
Confirm activityChapter 2 — service, account
Narrow by severityChapter 3, Chapter 7 — level:(2 OR 3), correct grouping
Rule out noiseChapter 9 — baseline noise check
Follow one requestChapter 4 — requestId
Pivot to clusterChapter 2, Chapter 6 — daxtraResource, correlationId, Flux fields
Confirm no false positiveChapter 9 — structured vs. free-text discipline
Shareable linkReal, explicitly unverified recipe from the cheat sheet

Hands-On Exercises

Exercise 1

Explain, in your own words, why Step 5 switches from requestId to correlationId — what specifically changed about the nature of what's being traced?

📄 View solution
Exercise 2

Build the query for Step 3 (the baseline-noise check) as a full, real query, substituting a literal exact error phrase, "upstream request failed", and a literal known-healthy account, bullhorn-40021.

📄 View solution
Exercise 3

Explain why this chapter recommends Graylog's own built-in share/save-search feature over hand-building a URL using the recipe given here, even though the recipe is presented in full.

📄 View solution

Chapter 10 (Capstone) Quick Reference

  • A full incident investigation built from every real field and technique in Chapters 1-9, with no new syntax introduced
  • The shareable-link URL recipe is explicitly unverified — Graylog's own built-in share/save-search feature is the safer real alternative

Course Complete

Graylog is now complete — 10/10 chapters, from the shape of the search box itself through to a full, real incident investigation, grounded throughout in genuine day-to-day usage at logger.daxtra.io.