Apprenticeship Learning

Apprenticeship learning learns from demonstrations by trying to perform like an expert on the features that matter. The learner does not need to reproduce every steering correction, every mouse movement, or every clinical habit. It tries to match the expert's outcomes and behavioral statistics: staying safe, reaching goals, conserving resources, avoiding discomfort, or keeping a process within constraints.

This is useful when giving examples is easier than writing rewards. A warehouse operator can demonstrate safe forklift paths. A surgeon can demonstrate tool handling. A customer-support specialist can demonstrate when to answer directly, escalate, or ask for clarification. Apprenticeship learning turns those demonstrations into a target for policy learning or reward inference.

A feature-based formulation represents each trajectory with feature counts or feature expectations. If $\phi(s,a)$ maps a state-action pair to task features, the expert's expected feature vector is:

$$\mu_E=\mathbb{E}_{\tau\sim\pi_E}\left[\sum_t \gamma^t\phi(s_t,a_t)\right]$$

The learner's feature expectation is:

$$\mu_\pi=\mathbb{E}_{\tau\sim\pi}\left[\sum_t \gamma^t\phi(s_t,a_t)\right]$$

Apprenticeship learning tries to find a policy whose feature expectations are close to the expert's. If the features are safety margin, speed, and smoothness, then many different exact trajectories could be acceptable as long as they match those expert-level properties.

The method depends heavily on feature design. If the features omit a crucial concern, the learner can match the expert on the measured dimensions while failing the real task. A driving policy might match average speed and lane position but ignore pedestrian comfort. A medical policy might match treatment frequency but ignore contraindications for a subgroup. A recommender might match click patterns but miss long-term user satisfaction.

There is also an expertise assumption. Demonstrations may be noisy, biased, constrained, or produced under missing information. A nurse may choose a treatment because a hospital policy requires it, not because it is intrinsically best. A human driver may brake late because visibility was poor. The learner needs enough context to avoid mistaking constraints or mistakes for preferences.