{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Titanic：现代统计模型\n",
    "将Notebook、train.csv、test.csv、course_model.py与validation-folds.csv放在同一目录。\n",
    "依次运行，或修改单个因素并固定验证设计。\n"
   ],
   "id": "titanic-000"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from IPython.display import display\n",
    "df = pd.read_csv(\"train.csv\")\n",
    "test = pd.read_csv(\"test.csv\")\n"
   ],
   "execution_count": null,
   "outputs": [],
   "id": "titanic-001"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 01 · 读取数据与变量\n",
    "df与test已读取；它们仍是原始数据。试着把head(5)改成head(10)。"
   ],
   "id": "titanic-002"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "print(\"train:\", df.shape, \"test:\", test.shape)\n",
    "print(\"Survived counts:\")\n",
    "print(df[\"Survived\"].value_counts().sort_index())\n",
    "display(df.head(5))"
   ],
   "execution_count": null,
   "outputs": [],
   "id": "titanic-003"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 02 · 数据质量与缺失\n",
    "dtype与统计类型不是同一概念。Survived在test中不存在，对应摘要的NaN表示不适用。"
   ],
   "id": "titanic-004"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "audit = pd.DataFrame({\n",
    "    \"dtype\": df.dtypes.astype(str),\n",
    "    \"train_missing\": df.isna().sum(),\n",
    "    \"test_missing\": test.isna().sum()\n",
    "})\n",
    "display(audit)\n",
    "print(\"Duplicate rows:\", df.duplicated().sum())\n",
    "print(\"PassengerId unique:\", df[\"PassengerId\"].is_unique)\n",
    "display(df[[\"Age\", \"Fare\", \"SibSp\", \"Parch\"]].describe().round(2))"
   ],
   "execution_count": null,
   "outputs": [],
   "id": "titanic-005"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 03 · 分组人数与生还率\n",
    "改分组字段后重新运行；size包含该组所有记录，mean利用0/1响应得到组内生还率。"
   ],
   "id": "titanic-006"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "group = \"Sex\"  # 改为 Pclass 或 Embarked\n",
    "summary = df.groupby(group, dropna=False)[\"Survived\"].agg(\n",
    "    n=\"size\", survived=\"sum\", rate=\"mean\"\n",
    ")\n",
    "display(summary.round(4))\n",
    "print(\"Denominator: all passengers in each group\")"
   ],
   "execution_count": null,
   "outputs": [],
   "id": "titanic-007"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 04 · 年龄直方图\n",
    "更改箱宽只改变汇总方式。这里是人数直方图，并非各年龄段生还率。"
   ],
   "id": "titanic-008"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "import matplotlib.pyplot as plt\n",
    "width = 10  # 可改为5或20岁\n",
    "bins = np.arange(0, 81, width)\n",
    "fig, ax = plt.subplots(figsize=(7, 3.6))\n",
    "for y, color, name in [(0, \"#aec2d5\", \"Died\"), (1, \"#00918b\", \"Survived\")]:\n",
    "    age = df.loc[df[\"Survived\"].eq(y), \"Age\"].dropna()\n",
    "    ax.hist(age, bins=bins, alpha=.65, label=name, color=color)\n",
    "ax.set(xlabel=\"Age\", ylabel=\"Count\", title=\"Age distribution by outcome\")\n",
    "ax.legend()\n",
    "fig.tight_layout()\n",
    "print(\"Known Age:\", df[\"Age\"].notna().sum(), \"Missing:\", df[\"Age\"].isna().sum())"
   ],
   "execution_count": null,
   "outputs": [],
   "id": "titanic-009"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 05 · Fare与log1p变换\n",
    "对数压缩高值间距，保留Fare=0。修改箱数，观察右偏与局部结构。"
   ],
   "id": "titanic-010"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "import matplotlib.pyplot as plt\n",
    "fare = df[\"Fare\"].dropna()\n",
    "print(\"Mean:\", round(fare.mean(), 2), \"Median:\", round(fare.median(), 2))\n",
    "fig, axes = plt.subplots(1, 2, figsize=(8, 3.4))\n",
    "axes[0].hist(fare, bins=20, color=\"#426c9b\")\n",
    "axes[1].hist(np.log1p(fare), bins=20, color=\"#00918b\")\n",
    "axes[0].set(xlabel=\"Fare\", ylabel=\"Count\")\n",
    "axes[1].set(xlabel=\"log(1 + Fare)\", ylabel=\"Count\")\n",
    "fig.tight_layout()"
   ],
   "execution_count": null,
   "outputs": [],
   "id": "titanic-011"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 06 · 类别缺失的众数填补\n",
    "尝试constant与fill_value=\"Unknown\"。众数填补与Unknown类别表达了不同假设。"
   ],
   "id": "titanic-012"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "from sklearn.impute import SimpleImputer\n",
    "# 展示全train拟合、应用到test；CV时相同步骤须在训练折内执行\n",
    "fill = SimpleImputer(strategy=\"most_frequent\")\n",
    "train_port = fill.fit_transform(df[[\"Embarked\"]])\n",
    "test_port = fill.transform(test[[\"Embarked\"]])\n",
    "print(\"Learned mode:\", fill.statistics_[0])\n",
    "print(\"Train missing after fill:\", pd.isna(train_port).sum())\n",
    "print(\"Test rows preserved:\", len(test_port))"
   ],
   "execution_count": null,
   "outputs": [],
   "id": "titanic-013"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 07 · 数值填补与待预测缺失\n",
    "填补值只从训练表学习。这里不把0票价当成缺失；CV需要每折重新fit。"
   ],
   "id": "titanic-014"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "from sklearn.impute import SimpleImputer\n",
    "fill = SimpleImputer(strategy=\"median\")\n",
    "fill.fit(df[[\"Fare\"]])\n",
    "filled = fill.transform(test[[\"Fare\"]]).ravel()\n",
    "mask = test[\"Fare\"].isna()\n",
    "result = test.loc[mask, [\"PassengerId\", \"Pclass\", \"Fare\"]].copy()\n",
    "result[\"Fare_filled\"] = filled[mask]\n",
    "display(result)\n",
    "print(\"Train median:\", fill.statistics_[0])"
   ],
   "execution_count": null,
   "outputs": [],
   "id": "titanic-015"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 08 · 分组年龄填补与回退\n",
    "本例用于查看规则。验证分数必须用包含FeatureBuilder的完整Pipeline重新计算。"
   ],
   "id": "titanic-016"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "from course_model import FeatureBuilder\n",
    "rule = FeatureBuilder(age_strategy=\"group\").fit(df.drop(columns=\"Survived\"))\n",
    "display(rule.age_groups_.rename(\"Age_median\").to_frame())\n",
    "filled = rule.transform(df.drop(columns=\"Survived\"))\n",
    "examples = df.loc[df[\"Age\"].isna(), [\"PassengerId\", \"Sex\", \"Pclass\", \"Age\"]].head(8).copy()\n",
    "examples[\"Age_filled\"] = filled.loc[examples.index, \"Age\"]\n",
    "display(examples)\n",
    "print(\"Fallback median:\", rule.age_median_)"
   ],
   "execution_count": null,
   "outputs": [],
   "id": "titanic-017"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 09 · Ridge年龄模型填补\n",
    "人工遮蔽实验不证明真实缺失者具有相同误差。这里检查年龄预测，不是生还预测的CV得分。"
   ],
   "id": "titanic-018"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "from course_model import ridge_age_impute\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "known = df.loc[df[\"Age\"].notna()].copy()\n",
    "learn, hold = train_test_split(known, test_size=.2, random_state=42)\n",
    "# 在已知年龄者中人工遮蔽，用于说明如何检查年龄预测\n",
    "hidden = hold.copy()\n",
    "hidden[\"Age\"] = np.nan\n",
    "predicted = ridge_age_impute(learn, hidden)\n",
    "print(\"Artificial holdout MAE:\", round(mean_absolute_error(hold[\"Age\"], predicted), 2))\n",
    "print(\"Median baseline MAE:\", round(mean_absolute_error(hold[\"Age\"], np.repeat(learn[\"Age\"].median(), len(hold))), 2))"
   ],
   "execution_count": null,
   "outputs": [],
   "id": "titanic-019"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10 · Title、FamilySize与Deck\n",
    "FeatureBuilder中的固定规则可以在下载的course_model.py查看。尝试按FamilyBand汇总结果。"
   ],
   "id": "titanic-020"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "from course_model import FeatureBuilder\n",
    "work = FeatureBuilder().fit_transform(df.drop(columns=\"Survived\"))\n",
    "cols = [\"PassengerId\", \"Name\", \"Title\", \"FamilySize\", \"IsAlone\", \"FamilyBand\", \"Cabin\", \"HasCabin\", \"Deck\"]\n",
    "display(work[cols].head(8))\n",
    "print(\"Title counts:\")\n",
    "print(work[\"Title\"].value_counts())"
   ],
   "execution_count": null,
   "outputs": [],
   "id": "titanic-021"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11 · 训练侧票号频数与票价缩放\n",
    "计数来源是训练表，未见票号为0，分母回退到1。缩放后变量不等于已核实的人均支付。"
   ],
   "id": "titanic-022"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "from course_model import FeatureBuilder\n",
    "rule = FeatureBuilder().fit(df.drop(columns=\"Survived\"))\n",
    "work = rule.transform(test)\n",
    "cols = [\"PassengerId\", \"Ticket\", \"Fare\", \"TicketCountTrain\", \"FarePerRecordedTicket\"]\n",
    "display(work[cols].head(10).round(3))\n",
    "print(\"Unseen test tickets:\", work[\"TicketCountTrain\"].eq(0).sum())"
   ],
   "execution_count": null,
   "outputs": [],
   "id": "titanic-023"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12 · 独热编码与参考类别\n",
    "本例明确把S放在首位作为参考。实际Pipeline由训练数据学习类别，查看categories_确认参考组。"
   ],
   "id": "titanic-024"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "ports = pd.DataFrame({\"Embarked\": [\"S\", \"C\", \"Q\", \"S\"]})\n",
    "encoder = OneHotEncoder(categories=[[\"S\", \"C\", \"Q\"]], drop=\"first\",\n",
    "                        handle_unknown=\"ignore\", sparse_output=False)\n",
    "matrix = encoder.fit_transform(ports)\n",
    "display(pd.DataFrame(matrix, columns=encoder.get_feature_names_out(), index=ports[\"Embarked\"]))"
   ],
   "execution_count": null,
   "outputs": [],
   "id": "titanic-025"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13 · 完整Pipeline与折外概率\n",
    "每条概率来自未使用该行训练的模型。此处类别为0/1，predict_proba的第二列对应生还1。"
   ],
   "id": "titanic-026"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "from course_model import make_model, course_splits\n",
    "from sklearn.model_selection import cross_val_predict\n",
    "X, y = df.drop(columns=\"Survived\"), df[\"Survived\"]\n",
    "cv = course_splits(X)  # 按PassengerId复用已发布的验证折\n",
    "model = make_model(\"engineered\")\n",
    "prob = cross_val_predict(model, X, y, cv=cv, method=\"predict_proba\")[:, 1]\n",
    "result = df[[\"PassengerId\", \"Survived\"]].copy()\n",
    "result[\"OOF_survival_probability\"] = prob\n",
    "display(result.head(10).round(4))\n",
    "print(\"OOF probabilities:\", len(prob))"
   ],
   "execution_count": null,
   "outputs": [],
   "id": "titanic-027"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14 · 同一五折比较特征方案\n",
    "可只保留两个方案减少运行时间。多变量同时改动的结果不是单变量贡献，也不能直接作为最终独立测试表现。"
   ],
   "id": "titanic-028"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "from course_model import make_model, course_splits\n",
    "from sklearn.model_selection import cross_val_score\n",
    "X, y = df.drop(columns=\"Survived\"), df[\"Survived\"]\n",
    "cv = course_splits(X)  # 按PassengerId复用已发布的验证折\n",
    "settings = [(\"basic\", \"median\"), (\"engineered\", \"median\"),\n",
    "            (\"engineered\", \"group\"), (\"advanced\", \"group\")]\n",
    "results = []\n",
    "for mode, age in settings:\n",
    "    scores = cross_val_score(make_model(mode, age), X, y, cv=cv, scoring=\"accuracy\")\n",
    "    results.append({\"features\": mode, \"Age\": age,\n",
    "                    \"mean\": scores.mean(), \"fold_sd\": scores.std(),\n",
    "                    \"folds\": np.round(scores, 4).tolist()})\n",
    "display(pd.DataFrame(results))"
   ],
   "execution_count": null,
   "outputs": [],
   "id": "titanic-029"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15 · 阈值与混淆矩阵\n",
    "这里只演示阈值如何改变错误类型，不将同一批验证结果上的最优阈值当成最终表现。"
   ],
   "id": "titanic-030"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "from course_model import make_model, course_splits\n",
    "from sklearn.model_selection import cross_val_predict\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score, precision_score, recall_score\n",
    "X, y = df.drop(columns=\"Survived\"), df[\"Survived\"]\n",
    "cv = course_splits(X)  # 按PassengerId复用已发布的验证折\n",
    "p = cross_val_predict(make_model(\"engineered\"), X, y, cv=cv, method=\"predict_proba\")[:, 1]\n",
    "threshold = 0.5  # 改成0.7\n",
    "pred = (p >= threshold).astype(int)\n",
    "print(\"Rows=actual 0/1; columns=predicted 0/1\")\n",
    "print(confusion_matrix(y, pred, labels=[0, 1]))\n",
    "print(\"Accuracy:\", round(accuracy_score(y, pred), 4))\n",
    "print(\"Precision:\", round(precision_score(y, pred, zero_division=0), 4))\n",
    "print(\"Recall:\", round(recall_score(y, pred), 4))"
   ],
   "execution_count": null,
   "outputs": [],
   "id": "titanic-031"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 16 · 按Ticket分组验证\n",
    "复用validation-folds.csv中的Ticket分组五折，同票号不跨训练与验证。它检验对未见票号团体的外推，仍不等于完全独立家庭或未来总体。"
   ],
   "id": "titanic-032"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "from course_model import make_model, course_splits\n",
    "from sklearn.model_selection import cross_val_score\n",
    "X, y = df.drop(columns=\"Survived\"), df[\"Survived\"]\n",
    "scores = cross_val_score(make_model(\"engineered\"), X, y,\n",
    "                         cv=course_splits(X, \"ticket\"), scoring=\"accuracy\")\n",
    "print(\"Group folds:\", np.round(scores, 4))\n",
    "print(\"Mean:\", round(scores.mean(), 4), \"Fold SD:\", round(scores.std(), 4))"
   ],
   "execution_count": null,
   "outputs": [],
   "id": "titanic-033"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}