Colab notebooks: use Kaggle KAGGLE_USERNAME/KAGGLE_KEY env vars

Kaggle now hands out an API token (username + token string) instead of a
kaggle.json file. Both Colab notebooks now resolve creds in this order:

1. KAGGLE_USERNAME + KAGGLE_KEY already in os.environ (e.g. Colab Secrets).
2. Drive token file at MyDrive/ccdp/kaggle_token.txt (KEY=value lines).
3. Legacy kaggle.json (fallback for users with the old token).
4. Interactive getpass prompt, with an offer to persist to (2) for next session.

The Kaggle CLI/SDK reads the env vars directly so no kaggle.json is needed.

notebooks/colab_train_identifier.ipynb

@@ -4,29 +4,7 @@
    "cell_type": "markdown",
    "id": "e0a4182e",
    "metadata": {},
-   "source": [
-    "# ccdp — train the VMMRdb identifier on Colab (GDrive-backed, crash-resumable)\n",
-    "\n",
-    "Same training loop as the Kaggle notebook (`ccdp train identifier-continue`), but built\n",
-    "for Google Colab. Checkpoints write straight to Google Drive every epoch, so a runtime\n",
-    "disconnect at epoch 9/12 doesn't lose work — the next launch auto-resumes from `last.pt`.\n",
-    "\n",
-    "### Before you run\n",
-    "1. **Runtime → Change runtime type → GPU** (T4 free tier or A100 if you have Pro).\n",
-    "2. You'll be prompted to mount your Drive in section 2. Approve it.\n",
-    "3. You'll need a `kaggle.json` API token to pull the VMMRdb dataset\n",
-    "   (Kaggle → Account → Create New Token). Either:\n",
-    "   - drop it into `MyDrive/kaggle/kaggle.json` once, or\n",
-    "   - upload it via the file picker each session.\n",
-    "\n",
-    "### Why GDrive?\n",
-    "Colab's `/content` is ephemeral — it dies with the VM. `/content/drive/MyDrive/...` is durable.\n",
-    "We point the run dir at GDrive so every `epoch_NNN.pt` lands on Drive immediately.\n",
-    "\n",
-    "### Time + budget\n",
-    "~12 epochs on a T4 ≈ **6–10 h**. Free Colab sessions cap at ~12 h with idle disconnects;\n",
-    "Colab Pro avoids most of that. Either way, the resume logic below makes a mid-run kill cheap.\n"
-   ]
+   "source": "# ccdp — train the VMMRdb identifier on Colab (GDrive-backed, crash-resumable)\n\nSame training loop as the Kaggle notebook (`ccdp train identifier-continue`), but built\nfor Google Colab. Checkpoints write straight to Google Drive every epoch, so a runtime\ndisconnect at epoch 9/12 doesn't lose work — the next launch auto-resumes from `last.pt`.\n\n### Before you run\n1. **Runtime → Change runtime type → GPU** (T4 free tier or A100 if you have Pro).\n2. You'll be prompted to mount your Drive in section 2. Approve it.\n3. **Kaggle API token** for the VMMRdb dataset. Kaggle → Settings → **API → Create New Token**.\n   The new flow gives you a **username + token** pair (not a JSON file). You'll paste them\n   into a `getpass` prompt in section 4. They're held in env vars `KAGGLE_USERNAME` /\n   `KAGGLE_KEY` for the session only — nothing is written to Drive unless you ask.\n   - Optional: stash them once in `MyDrive/ccdp/kaggle_token.txt` (one line per var:\n     `KAGGLE_USERNAME=…`, `KAGGLE_KEY=…`) and the cell will auto-load them on later runs.\n   - Legacy `kaggle.json` is still supported as a fallback.\n\n### Why GDrive?\nColab's `/content` is ephemeral — it dies with the VM. `/content/drive/MyDrive/...` is durable.\nWe point the run dir at GDrive so every `epoch_NNN.pt` lands on Drive immediately.\n\n### Time + budget\n~12 epochs on a T4 ≈ **6–10 h**. Free Colab sessions cap at ~12 h with idle disconnects;\nColab Pro avoids most of that. Either way, the resume logic below makes a mid-run kill cheap.\n"
   },
   {
    "cell_type": "markdown",
@@ -109,12 +87,7 @@
    "cell_type": "markdown",
    "id": "1b7431c4",
    "metadata": {},
-   "source": [
-    "## 4. Fetch VMMRdb via Kaggle API\n",
-    "\n",
-    "The CC0 mirror is `prabashwara/vmmrdb-dataset` (~291k images, ~9,170 classes). One-time:\n",
-    "upload your `kaggle.json` to Drive (`MyDrive/kaggle/kaggle.json`)."
-   ]
+   "source": "## 4. Fetch VMMRdb via Kaggle API\n\nThe CC0 mirror is `prabashwara/vmmrdb-dataset` (~291k images, ~9,170 classes).\n\n**Auth resolution order** — the cell tries each in turn and stops at the first that works:\n1. `KAGGLE_USERNAME` / `KAGGLE_KEY` already in `os.environ` (e.g. set via Colab Secrets).\n2. `MyDrive/ccdp/kaggle_token.txt` with two `KEY=value` lines.\n3. Legacy `MyDrive/kaggle/kaggle.json`.\n4. Interactive `getpass` prompt (paste username + token from Kaggle → Settings → API).\n"
   },
   {
    "cell_type": "code",
@@ -122,39 +95,7 @@
    "id": "79c6621b",
    "metadata": {},
    "outputs": [],
-   "source": [
-    "import os, shutil, pathlib\n",
-    "KAGGLE_JSON_DRIVE = '/content/drive/MyDrive/kaggle/kaggle.json'\n",
-    "kdir = pathlib.Path.home() / '.kaggle'\n",
-    "kdir.mkdir(exist_ok=True)\n",
-    "if not (kdir / 'kaggle.json').exists():\n",
-    "    assert os.path.exists(KAGGLE_JSON_DRIVE), (\n",
-    "        f'put your kaggle API token at {KAGGLE_JSON_DRIVE} (or upload via Files panel)')\n",
-    "    shutil.copy(KAGGLE_JSON_DRIVE, kdir / 'kaggle.json')\n",
-    "    os.chmod(kdir / 'kaggle.json', 0o600)\n",
-    "!pip -q install kaggle\n",
-    "\n",
-    "# Cache the dataset on Drive too — re-downloading 70GB every session is brutal.\n",
-    "DATA_ROOT_DRIVE = f'{DRIVE_ROOT}/data/raw/vmmrdb-dataset/prabashwara/vmmrdb-dataset'\n",
-    "os.makedirs(DATA_ROOT_DRIVE, exist_ok=True)\n",
-    "# Mirror to the loader's expected path:\n",
-    "os.makedirs('data/raw/vmmrdb-dataset/prabashwara', exist_ok=True)\n",
-    "local_data = pathlib.Path('data/raw/vmmrdb-dataset/prabashwara/vmmrdb-dataset')\n",
-    "if local_data.exists() or local_data.is_symlink():\n",
-    "    local_data.unlink()\n",
-    "local_data.symlink_to(DATA_ROOT_DRIVE)\n",
-    "\n",
-    "# Only download if Drive copy is empty.\n",
-    "import glob\n",
-    "if not glob.glob(f'{DATA_ROOT_DRIVE}/*'):\n",
-    "    !kaggle datasets download -d prabashwara/vmmrdb-dataset -p {DATA_ROOT_DRIVE} --unzip\n",
-    "else:\n",
-    "    print('VMMRdb already cached on Drive — skipping download')\n",
-    "\n",
-    "from ccdp.data import vmmrdb\n",
-    "counts = vmmrdb._class_dir_counts('data/raw/vmmrdb-dataset')\n",
-    "print(f'class folders: {len(counts)}  total images: {sum(counts.values())}')"
-   ]
+   "source": "import os, json, pathlib, glob, getpass\n\nDRIVE_TOKEN_FILE = '/content/drive/MyDrive/ccdp/kaggle_token.txt'\nDRIVE_LEGACY_JSON = '/content/drive/MyDrive/kaggle/kaggle.json'\n\ndef _resolve_kaggle_creds():\n    # 1. already in env (e.g. Colab Secrets injected)\n    if os.environ.get('KAGGLE_USERNAME') and os.environ.get('KAGGLE_KEY'):\n        return 'env'\n    # 2. token file on Drive (KAGGLE_USERNAME=… / KAGGLE_KEY=…)\n    if os.path.exists(DRIVE_TOKEN_FILE):\n        for line in pathlib.Path(DRIVE_TOKEN_FILE).read_text().splitlines():\n            line = line.strip()\n            if not line or line.startswith('#') or '=' not in line:\n                continue\n            k, v = line.split('=', 1)\n            os.environ[k.strip()] = v.strip().strip('\"').strip(\"'\")\n        if os.environ.get('KAGGLE_USERNAME') and os.environ.get('KAGGLE_KEY'):\n            return f'drive token file ({DRIVE_TOKEN_FILE})'\n    # 3. legacy kaggle.json on Drive\n    if os.path.exists(DRIVE_LEGACY_JSON):\n        data = json.loads(pathlib.Path(DRIVE_LEGACY_JSON).read_text())\n        os.environ['KAGGLE_USERNAME'] = data['username']\n        os.environ['KAGGLE_KEY'] = data['key']\n        return f'legacy kaggle.json ({DRIVE_LEGACY_JSON})'\n    # 4. interactive prompt\n    print('No Kaggle creds found. Paste them from Kaggle → Settings → API.')\n    os.environ['KAGGLE_USERNAME'] = input('KAGGLE_USERNAME: ').strip()\n    os.environ['KAGGLE_KEY'] = getpass.getpass('KAGGLE_KEY (token): ').strip()\n    # offer to stash them on Drive for next session\n    save = input(f'Save to {DRIVE_TOKEN_FILE} for next session? [y/N] ').strip().lower()\n    if save == 'y':\n        pathlib.Path(DRIVE_TOKEN_FILE).parent.mkdir(parents=True, exist_ok=True)\n        pathlib.Path(DRIVE_TOKEN_FILE).write_text(\n            f\"KAGGLE_USERNAME={os.environ['KAGGLE_USERNAME']}\\n\"\n            f\"KAGGLE_KEY={os.environ['KAGGLE_KEY']}\\n\"\n        )\n        os.chmod(DRIVE_TOKEN_FILE, 0o600)\n        print(f'saved → {DRIVE_TOKEN_FILE}')\n    return 'interactive'\n\nsource = _resolve_kaggle_creds()\nprint(f'[kaggle] auth via {source} (user={os.environ.get(\"KAGGLE_USERNAME\")})')\n\n# The Kaggle CLI/SDK reads KAGGLE_USERNAME/KAGGLE_KEY directly — no kaggle.json needed.\n!pip -q install kaggle\n\n# Cache the dataset on Drive too — re-downloading 70GB every session is brutal.\nDATA_ROOT_DRIVE = f'{DRIVE_ROOT}/data/raw/vmmrdb-dataset/prabashwara/vmmrdb-dataset'\nos.makedirs(DATA_ROOT_DRIVE, exist_ok=True)\n# Mirror to the loader's expected path:\nos.makedirs('data/raw/vmmrdb-dataset/prabashwara', exist_ok=True)\nlocal_data = pathlib.Path('data/raw/vmmrdb-dataset/prabashwara/vmmrdb-dataset')\nif local_data.exists() or local_data.is_symlink():\n    local_data.unlink()\nlocal_data.symlink_to(DATA_ROOT_DRIVE)\n\n# Only download if Drive copy is empty.\nif not glob.glob(f'{DATA_ROOT_DRIVE}/*'):\n    !kaggle datasets download -d prabashwara/vmmrdb-dataset -p {DATA_ROOT_DRIVE} --unzip\nelse:\n    print('VMMRdb already cached on Drive — skipping download')\n\nfrom ccdp.data import vmmrdb\ncounts = vmmrdb._class_dir_counts('data/raw/vmmrdb-dataset')\nprint(f'class folders: {len(counts)}  total images: {sum(counts.values())}')"
   },
   {
    "cell_type": "markdown",
@@ -293,4 +234,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 5
-}
+}

notebooks/train_damage_seg_hitl_pathA.ipynb

@@ -203,32 +203,7 @@
    "id": "de6ed90c",
    "metadata": {},
    "outputs": [],
-   "source": [
-    "import os, glob\n",
-    "HITL_SLUG = 'YOUR_USERNAME/hitl-vehicle-damage'   # <-- update with actual Kaggle slug\n",
-    "HITL_DRIVE_CACHE = f'{DURABLE_ROOT}/data/raw/hitl'\n",
-    "os.makedirs(HITL_DRIVE_CACHE, exist_ok=True)\n",
-    "\n",
-    "# Kaggle creds\n",
-    "if PLATFORM == 'colab':\n",
-    "    import shutil, pathlib\n",
-    "    src = '/content/drive/MyDrive/kaggle/kaggle.json'\n",
-    "    kdir = pathlib.Path.home()/'.kaggle'; kdir.mkdir(exist_ok=True)\n",
-    "    if os.path.exists(src) and not (kdir/'kaggle.json').exists():\n",
-    "        shutil.copy(src, kdir/'kaggle.json'); os.chmod(kdir/'kaggle.json', 0o600)\n",
-    "\n",
-    "# CarDD — uses our packaged builder\n",
-    "from ccdp.data import cardd_yolo\n",
-    "cardd_yaml = cardd_yolo.build()\n",
-    "print('CarDD yaml:', cardd_yaml)\n",
-    "\n",
-    "# HITL — download if not cached on Drive\n",
-    "if not glob.glob(f'{HITL_DRIVE_CACHE}/*'):\n",
-    "    !pip -q install kaggle\n",
-    "    !kaggle datasets download -d {HITL_SLUG} -p {HITL_DRIVE_CACHE} --unzip\n",
-    "else:\n",
-    "    print('HITL cache present:', HITL_DRIVE_CACHE)"
-   ]
+   "source": "import os, json, pathlib, glob, getpass\n\nHITL_SLUG = 'YOUR_USERNAME/hitl-vehicle-damage'   # <-- update with actual Kaggle slug\nHITL_DRIVE_CACHE = f'{DURABLE_ROOT}/data/raw/hitl'\nos.makedirs(HITL_DRIVE_CACHE, exist_ok=True)\n\n# --- Kaggle creds ------------------------------------------------------\n# New flow: KAGGLE_USERNAME + KAGGLE_KEY env vars (token from Kaggle → Settings → API).\n# Fallbacks: Drive token file -> legacy kaggle.json -> interactive prompt.\ndef _resolve_kaggle_creds():\n    if os.environ.get('KAGGLE_USERNAME') and os.environ.get('KAGGLE_KEY'):\n        return 'env'\n    if PLATFORM == 'kaggle':\n        # Kaggle kernels are auto-authenticated for their own datasets — nothing to do.\n        return 'kaggle-native'\n    # Colab: try the durable token file, then legacy json, then prompt.\n    token_file = '/content/drive/MyDrive/ccdp/kaggle_token.txt'\n    legacy_json = '/content/drive/MyDrive/kaggle/kaggle.json'\n    if os.path.exists(token_file):\n        for line in pathlib.Path(token_file).read_text().splitlines():\n            line = line.strip()\n            if not line or line.startswith('#') or '=' not in line: continue\n            k, v = line.split('=', 1)\n            os.environ[k.strip()] = v.strip().strip('\"').strip(\"'\")\n        if os.environ.get('KAGGLE_USERNAME') and os.environ.get('KAGGLE_KEY'):\n            return f'drive token file ({token_file})'\n    if os.path.exists(legacy_json):\n        data = json.loads(pathlib.Path(legacy_json).read_text())\n        os.environ['KAGGLE_USERNAME'] = data['username']\n        os.environ['KAGGLE_KEY'] = data['key']\n        return f'legacy kaggle.json ({legacy_json})'\n    print('No Kaggle creds found. Paste them from Kaggle → Settings → API.')\n    os.environ['KAGGLE_USERNAME'] = input('KAGGLE_USERNAME: ').strip()\n    os.environ['KAGGLE_KEY'] = getpass.getpass('KAGGLE_KEY (token): ').strip()\n    if input(f'Save to {token_file} for next session? [y/N] ').strip().lower() == 'y':\n        pathlib.Path(token_file).parent.mkdir(parents=True, exist_ok=True)\n        pathlib.Path(token_file).write_text(\n            f\"KAGGLE_USERNAME={os.environ['KAGGLE_USERNAME']}\\n\"\n            f\"KAGGLE_KEY={os.environ['KAGGLE_KEY']}\\n\"\n        )\n        os.chmod(token_file, 0o600)\n        print(f'saved → {token_file}')\n    return 'interactive'\n\nsource = _resolve_kaggle_creds()\nprint(f'[kaggle] auth via {source} (user={os.environ.get(\"KAGGLE_USERNAME\",\"<kaggle-native>\")})')\n\n# CarDD — uses our packaged builder\nfrom ccdp.data import cardd_yolo\ncardd_yaml = cardd_yolo.build()\nprint('CarDD yaml:', cardd_yaml)\n\n# HITL — download if not cached on Drive\nif not glob.glob(f'{HITL_DRIVE_CACHE}/*'):\n    !pip -q install kaggle\n    !kaggle datasets download -d {HITL_SLUG} -p {HITL_DRIVE_CACHE} --unzip\nelse:\n    print('HITL cache present:', HITL_DRIVE_CACHE)"
   },
   {
    "cell_type": "markdown",
@@ -524,4 +499,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 5
-}
+}