SmartUp/browser-session-cookie-persistence-plan.md

# Plan: Browser Session Cookie Persistence

**Generated**: 2026-05-29
**Estimated Complexity**: Medium

## Overview

Remote browser profiles already live under `browser_profiles_dir` (`/app/data/browser-profiles` by default), so Chromium profile files are expected to survive container restarts when `/app/data` is mounted. The remaining login loss case is mainly session-only cookies that Chromium removes when Playwright closes a persistent context normally.

Implement a backend-only cookie persistence layer in `BrowserSessionService`:

- Save all cookies from active persistent browser contexts into a JSON file under the profile directory.
- Restore those cookies immediately after `launch_persistent_context(...)` and before `page.goto(...)`.
- Keep session-only cookies as session cookies when restoring, instead of rewriting them as long-lived cookies by default.
- Exclude ephemeral auth-capture sessions so temporary login extraction profiles keep their current lifecycle.

## Prerequisites

- Current Playwright dependency: `backend/requirements.txt` pins `playwright==1.52.0`.
- Playwright `BrowserContext.add_cookies()` accepts cookies with `name`, `value`, and either `url` or `domain` + `path`; it also supports `expires`, `httpOnly`, `secure`, `sameSite`, and `partitionKey`.
- No frontend changes are required.

## Key Design Decisions

- **Storage location**: `browser-profiles/{profile_key}/session-cookies.json`.
- **Scope**: persistent remote-browser sessions only. Do not persist cookies for `auth-capture-*` profiles.
- **Save trigger**: after user interaction events, with debounce; before `context.close()` as a final save.
- **Restore trigger**: after `launch_persistent_context(...)`, before the first navigation.
- **Session cookie behavior**: if `expires` is missing, `None`, or negative, omit `expires` when restoring. This preserves session-cookie behavior inside the new context while still allowing our JSON backup to survive service restarts.
- **Security**: treat the JSON file as sensitive. Keep it under the already-private profile directory and write it with owner-readable permissions where practical.

## Sprint 1: Cookie Persistence Helpers

**Goal**: Add isolated helper methods without changing session behavior yet.

**Demo/Validation**:
- Unit tests can save, read, normalize, and restore cookie data using fake contexts and a temp profile directory.
- Invalid or empty JSON files do not break browser startup.

### Task 1.1: Add cookie file path helper

- **Location**: `backend/app/services/browser_session_service.py`
- **Description**: Add `_cookies_path(profile_key: str) -> Path`, returning `self._profile_dir(profile_key) / "session-cookies.json"`.
- **Dependencies**: None
- **Acceptance Criteria**:
  - Path is profile-local.
  - Existing `clear_profile()` deletes the cookie JSON automatically because it removes the full profile directory.
- **Validation**:
  - Unit test with a temp `browser_profiles_dir` confirms path location.

### Task 1.2: Add cookie serialization helpers

- **Location**: `backend/app/services/browser_session_service.py`
- **Description**: Add helpers:
  - `_normalize_cookie_for_save(cookie: dict[str, Any]) -> dict[str, Any] | None`
  - `_normalize_cookie_for_restore(cookie: dict[str, Any], now: float) -> dict[str, Any] | None`
- **Dependencies**: Task 1.1
- **Acceptance Criteria**:
  - Preserve supported Playwright fields: `name`, `value`, `domain`, `path`, `expires`, `httpOnly`, `secure`, `sameSite`, `partitionKey`.
  - Drop unsupported or unserializable fields.
  - Skip cookies missing `name` or `value`.
  - For restore, skip expired cookies when `expires > 0 and expires <= now`.
  - For restore, omit `expires` when it is missing, `None`, or negative.
  - Ensure every restored cookie has either `domain` + `path` or `url`.
- **Validation**:
  - Unit tests for persistent cookies, session cookies, expired cookies, and partitioned cookies.

### Task 1.3: Add atomic JSON read/write helpers

- **Location**: `backend/app/services/browser_session_service.py`
- **Description**: Add:
  - `_read_saved_cookies(profile_key: str) -> list[dict[str, Any]]`
  - `_write_saved_cookies(profile_key: str, cookies: list[dict[str, Any]]) -> None`
- **Dependencies**: Tasks 1.1 and 1.2
- **Acceptance Criteria**:
  - JSON schema includes `version`, `profile_key`, `saved_at`, and `cookies`.
  - Write is atomic via `session-cookies.json.tmp` then `replace(...)`.
  - Malformed JSON logs a warning and returns an empty cookie list.
  - Empty cookie list writes a valid file rather than failing.
- **Validation**:
  - Unit tests for normal write/read, corrupted JSON, and atomic replacement.

## Sprint 2: Restore Cookies On Session Create

**Goal**: Restore saved cookies before the first page load.

**Demo/Validation**:
- A fake Playwright context receives `add_cookies(...)` before `page.goto(...)`.
- Startup continues even if restore fails.

### Task 2.1: Add restore method

- **Location**: `backend/app/services/browser_session_service.py`
- **Description**: Add `_restore_cookies(session_or_context, profile_key: str) -> None`, using `context.add_cookies(cookies)` when saved cookies exist.
- **Dependencies**: Sprint 1
- **Acceptance Criteria**:
  - No-op for missing JSON or empty cookie list.
  - Logs count of restored cookies at `info` or `debug` level without logging cookie values.
  - Catches Playwright restore errors and logs them without blocking session creation.
- **Validation**:
  - Unit test fake context records restored cookies.
  - Unit test invalid cookie list does not raise.

### Task 2.2: Wire restore into `create()`

- **Location**: `backend/app/services/browser_session_service.py`
- **Description**: Call restore after `launch_persistent_context(...)` and before `page.goto(...)`.
- **Dependencies**: Task 2.1
- **Acceptance Criteria**:
  - Applies only to normal persistent sessions.
  - Existing health-check path for already-open sessions is unchanged.
  - `page.goto(...)` sees restored cookies on first request.
- **Validation**:
  - Unit test call order with fakes.
  - Manual test: log into a remote page, restart backend, reopen page, verify logged-in state when server-side session is still valid.

### Task 2.3: Do not restore for `create_ephemeral()`

- **Location**: `backend/app/services/browser_session_service.py`
- **Description**: Leave auth-capture sessions isolated.
- **Dependencies**: Task 2.1
- **Acceptance Criteria**:
  - No cookie JSON is read for `auth-capture-*`.
  - Existing auth-capture cleanup behavior remains unchanged.
- **Validation**:
  - Unit test or code assertion via fake profile key.

## Sprint 3: Save Cookies During Activity And Close

**Goal**: Keep the JSON cache fresh while users interact and before Playwright closes the context.

**Demo/Validation**:
- User interactions cause cookie JSON to appear/update.
- Closing a session saves cookies before `context.close()`.

### Task 3.1: Add save method

- **Location**: `backend/app/services/browser_session_service.py`
- **Description**: Add `_save_cookies(session: BrowserSession, *, force: bool = False) -> None`.
- **Dependencies**: Sprint 1
- **Acceptance Criteria**:
  - Calls `await session.context.cookies()`.
  - Normalizes cookies and writes JSON.
  - Skips `auth-capture-*` sessions.
  - Does not log cookie values.
  - Handles closed contexts or Playwright errors without raising during cleanup.
- **Validation**:
  - Unit test fake context cookies are written.
  - Unit test auth-capture profile is skipped.

### Task 3.2: Debounce saves after browser events

- **Location**: `backend/app/services/browser_session_service.py`
- **Description**: After supported `event()` actions complete, call `_save_cookies(session)` with a debounce interval, for example 5 seconds.
- **Dependencies**: Task 3.1
- **Acceptance Criteria**:
  - Reuses existing `BrowserSession.last_saved_state_at`.
  - Saves after meaningful actions including click, type, key, reload, back, forward, resize, and scroll.
  - Does not save on every screenshot request.
  - Does not block event responses for long; if cookie reads are fast, inline is acceptable. If they prove slow, use a background task guarded by session lock.
- **Validation**:
  - Unit test repeated events inside debounce produce one write.
  - Unit test event after debounce writes again.

### Task 3.3: Force save before close and shutdown

- **Location**: `backend/app/services/browser_session_service.py`
- **Description**: In `close()`, call `_save_cookies(session, force=True)` before `context.close()`.
- **Dependencies**: Task 3.1
- **Acceptance Criteria**:
  - Save happens before CDP detach/close when possible.
  - `shutdown()` benefits automatically because it calls `close()` for each session.
  - Close still proceeds even if cookie save fails.
- **Validation**:
  - Unit test fake session records save before context close.

## Sprint 4: Tests And Operational Verification

**Goal**: Prove the feature works without depending entirely on live websites.

**Demo/Validation**:
- Unit tests pass.
- Manual Docker restart test demonstrates retained login for a site whose server-side session remains valid.

### Task 4.1: Add unit tests

- **Location**: `backend/test_browser_session_service.py`
- **Description**: Extend current fake-based tests for cookie persistence helper behavior.
- **Dependencies**: Sprints 1-3
- **Acceptance Criteria**:
  - Tests cover save, restore, malformed JSON, expired cookie skip, session cookie restore, auth-capture skip, close-before-context-close order, and event debounce.
- **Validation**:
  - Run `pytest backend/test_browser_session_service.py`.

### Task 4.2: Add manual verification checklist

- **Location**: `browser-session-cookie-persistence-plan.md` or a future PR description
- **Description**: Document how to verify in Docker.
- **Dependencies**: Implementation complete
- **Acceptance Criteria**:
  - Start Docker deployment with `/app/data` mounted.
  - Open remote browser page and log in.
  - Perform an event after login to trigger cookie save.
  - Confirm `session-cookies.json` exists under the matching profile directory.
  - Restart backend/container.
  - Reopen remote browser page and verify login state.
  - Clear profile and confirm both Chromium profile and cookie JSON are removed.

## Testing Strategy

- **Unit tests**: Primary coverage using fake context/page/session objects. This avoids requiring real browser binaries in normal test runs.
- **Manual integration test**: Required for confidence because real sites differ in cookie, localStorage, and server-side session behavior.
- **Regression checks**:
  - `pytest backend/test_browser_session_service.py`
  - Existing backend tests if time allows: `pytest backend`

## Potential Risks & Gotchas

- Some sites use server-side session expiry or revocation; restoring cookies cannot bypass that.
- Some sites bind sessions to IP, user agent, device fingerprint, or TLS/browser state.
- Some login state is stored in `localStorage`, `sessionStorage`, IndexedDB, or service-worker cache. Persistent context already helps with some of this, but this plan only adds explicit cookie backup.
- `sessionStorage` is tab-lifetime state and is not covered here. If a target site depends on it heavily, a later phase can add origin-scoped storage backup.
- Cookie JSON contains authentication secrets. It must not be committed, logged, or exposed via APIs.
- For CHIPS/partitioned cookies, preserve `partitionKey` when Playwright returns it.
- Do not rewrite all session cookies to long-lived cookies by default; that changes browser semantics and may create security surprises.

## Rollback Plan

- Remove the new helper methods and calls from `BrowserSessionService.create()`, `event()`, and `close()`.
- Delete `session-cookies.json` files from affected profile directories if needed.
- Existing Chromium persistent profile behavior will continue to work as before.

## Open Decisions

- Whether to add a config flag such as `browser_cookie_persistence_enabled: bool = True`. Default can be enabled because this directly addresses the production issue.
- Whether to also save `localStorage` through Playwright `storage_state()` in a later phase. Not required for the first implementation.
- Whether cookie JSON should be encrypted at rest. For the current Docker single-host deployment, profile-directory isolation is probably sufficient; encryption can be added if this becomes multi-tenant or shared-host.