Capstone: A Full EDA Project on a Real Dataset

Data Science Fundamentals

Chapter 10 · Capstone: A Full EDA Project on a Real Dataset

A third dataset, unrelated to the coffee-shop sales table (ds1-3ds1-8) or the used-car listings (ds1-9): a small employee-attrition table for a fictional company, asking a genuinely different kind of question — not "what predicts a price," but "what's associated with an employee leaving." ds1-9's own seven-step methodology, applied start to finish, unchanged.

employee_iddepartmentageyears_at_companysalaryleft_company
E01Sales292.552,000Yes
E02Engineering345.081,000No
E03Sales418.067,000No
E04Engineering261.074,000Yes
E05Support5214.058,000No
E06Sales311.549,000Yes
E07Engineering459.096,000No
E08Support270.544,000Yes

Step 1 — Shape

df.shape / df.info() / df.head()

8 rows, 6 columns. One categorical (department), one boolean-like categorical (left_company), four numeric. No missing values in this small sample — unusual for a real dataset, and worth naming rather than assuming.

Step 2 — Summary Statistics

df.describe()

Salary ranges roughly $44,000–$96,000, mean around $65,000. Years at company ranges from 0.5 to 14 — a wide spread worth investigating directly in Step 3.

Step 3 — Univariate Analysis

Histogram of years_at_company; value_counts() on department and left_company
df["years_at_company"].plot(kind="hist")
df["department"].value_counts()
df["left_company"].value_counts()

✓ Finding: years_at_company skews heavily toward short tenures — most employees in this sample have been at the company under three years.

Step 4 — Bivariate Analysis

Scatter (years_at_company vs. salary); box plot (salary by left_company)
df.plot(kind="scatter", x="years_at_company", y="salary")
sns.boxplot(data=df, x="left_company", y="salary")

✓ Finding: a box plot of salary grouped by left_company shows the "Yes" group's salaries sitting visibly lower than the "No" group's — a real, visible bivariate pattern, not yet a conclusion.

Step 5 — Multivariate Analysis

Correlation heatmap and pair plot across age, years_at_company, and salary
sns.heatmap(df[["age","years_at_company","salary"]].corr(), annot=True)
sns.pairplot(df[["age","years_at_company","salary"]])

✓ Finding: age and years_at_company correlate strongly and positively with each other — unsurprising, since both track roughly the same underlying "how established" idea — while salary correlates more weakly with either alone.

Step 6 — Spotting Anomalies

IQR rule on salary; visual confirmation via box plot

Nothing in this small sample clears the 1.5×IQR threshold — a genuinely different, and equally valid, outcome from ds1-9's own used-car dataset, where the Jaguar was flagged immediately. A real EDA process sometimes finds nothing anomalous, and that absence is itself a legitimate, worth-recording finding, not a failed step.

Step 7 — Forming Hypotheses

Turning Steps 3–6's own findings into testable questions
  • "Does lower salary predict a higher likelihood of an employee leaving, once tenure is accounted for?" — directly motivated by Step 4's own box-plot finding.
  • "Is short tenure itself a risk factor for leaving, independent of salary?" — directly motivated by Step 3's own skewed-tenure finding.
  • "Does department matter, or is department's own apparent effect actually explained by department-level salary differences instead?" — a genuine, unresolved confounding-variable question, straight out of ds1-6's own material.

Chapter Attribution

Capstone elementDrawn from
The seven-step structure itselfds1-9
df.shape / df.info() / df.head()ds1-3
df.describe(), the salary/tenure vocabularyds1-6
Histogram, scatter plotds1-7
Box plot, heatmap, pair plotds1-8
The IQR anomaly rule, and the honest "flag, not verdict" framingds1-4 / ds1-6
The confounding-variable question in Step 7ds1-6
The Explore/Model boundary itselfds1-1

Honest Scope Note

What this capstone deliberately doesn't attempt
  • No ML modeling. This capstone forms real hypotheses about what predicts attrition — it never builds, trains, or tests a model that actually predicts it. That's ml1's entire job, and this course's own boundary, stated since ds1-1.
  • No live web scraping or API-based data collection. All three datasets in this course (coffee-shop sales, used cars, this chapter's own employee table) were provided directly. A real, hands-on data-collection project — building a tool that actually gathers data from the web — is dsproj1's own first project, chosen specifically because it doesn't need ML at all.
  • No big-data or distributed processing. Every dataset in this course fits comfortably in memory on an ordinary laptop. Datasets too large for that are a genuinely different, more advanced problem this course doesn't attempt to solve.

Hands-On Exercises

Exercise 1

Explain why Step 6 finding no outliers in this capstone's dataset is described as "a legitimate, worth-recording finding, not a failed step," contrasting it with ds1-9's own used-car dataset where an outlier was found immediately.

📄 View solution
Exercise 2

Explain why the Step 7 hypothesis about department is described as "a genuine, unresolved confounding-variable question," using ds1-6's own confounding-variable material to justify why department's own apparent effect can't simply be trusted at face value.

📄 View solution
Exercise 3

Using this chapter's own scope note, explain why "no ML modeling" and "no live web scraping" are both listed as deliberately out of scope here, and identify which specific future course or project each one is deferred to.

📄 View solution

Chapter 10 Quick Reference — Course Summary

  • The data science workflow: Collect → Clean → Explore → Model → Communicate (ds1-1) — this course covered everything except Model
  • NumPy's ndarray (ds1-2) underlies pandas' Series/DataFrame (ds1-3), which ds1-4 cleans and ds1-5 combines/reshapes
  • ds1-6's statistical vocabulary is what ds1-7's Matplotlib charts and ds1-8's Seaborn statistical plots actually visualize
  • ds1-9 formalized the seven-step EDA methodology; this capstone applied it, unchanged, to a third, genuinely different dataset
  • Next up in the Data Science & ML subject: ml1 (Machine Learning Fundamentals) — picking up exactly where this course's own Explore/Model boundary leaves off