Physics-structured exploration · Temporal contrast · Hamiltonian phase space

HTC2

Hamiltonian Temporal Contrastive Curiosity for Reinforcement Learning Exploration

Anonymous authors · ICLR 2027 submission

TODO χ arXiv TODO TODO PDF Paper TODO TODO </> Code TODO
Overview

Exploration through temporally informative physical futures.

HTC2 formulates intrinsic exploration as temporal future discovery in a learned Hamiltonian phase space. It trains a horizon-conditioned contrastive critic on same-episode future pairs, then modulates the resulting temporal curiosity signal with bounded multi-step Hamiltonian variation.

Multi-step novelty

Rewards futures that remain poorly organized by the current temporal representation.

Structured dynamics

Uses Hamiltonian phase variables to encode physically meaningful changes over time.

Control benchmarks

Evaluated on DeepMind Control Suite tasks with SAC and PPO against common intrinsic rewards.

Method

HTC2 at a glance.

The method combines temporal contrastive learning, learned Hamiltonian phase-space structure, and a bounded energy-variation modulation to guide policy exploration beyond local one-step novelty.

Overview diagram of Hamiltonian Temporal Contrastive Curiosity
Overview of the HTC2 exploration objective and its phase-space curiosity signal.
01

Horizon-conditioned critic

Contrasts same-episode future pairs over multiple horizons to detect temporally informative outcomes.

02

Hamiltonian phase space

Represents dynamics with phase variables so curiosity can respond to structured physical transitions.

03

Bounded modulation

Uses multi-step Hamiltonian variation as structure-aware modulation rather than as a standalone objective.

Results

Learning curves and rollout summaries.

HTC2 is evaluated across physical-control tasks and compared with prediction-error, state-novelty, representation-displacement, and uncertainty-driven exploration baselines.

PPO learning curves on DeepMind Control Suite tasks

PPO learning curves

Task-level comparison under on-policy optimization.

SAC learning curves on DeepMind Control Suite tasks

SAC learning curves

Task-level comparison under off-policy optimization.

Rollout summary over six control tasks

Six-task rollout summary

Aggregated rollout behavior across evaluated environments.

Distributional return comparison across control tasks

Return distributions

Distributional view of performance across methods and tasks.

Coverage

State-space coverage diagnostics.

Coverage visualizations help characterize how the intrinsic signal changes exploration behavior beyond final return.

Hamiltonian-aware state coverage on three tasks

Hamiltonian-aware coverage

Coverage patterns under structured phase-space exploration.

Ablation and Robustness

Components, dynamics shifts, and retention.

Ablation and out-of-distribution tests examine whether temporal contrast, physical consistency, energy modulation, warm start, normalization, and dynamics-shift retention contribute to the exploration behavior.

HTC2 ablation learning curves

Ablation curves

Component-level comparison of the HTC2 objective.

Parametric retention across all baselines

Baseline retention

Robustness comparison against intrinsic-reward baselines.

Diagnostics

Phase-space probes.

Diagnostic probes visualize how learned phase variables, Hamiltonian energy, and action-conditioned geometry organize across tasks and control modes.

Alternative action-conditioned phase geometry

Action-conditioned geometry

Phase-space organization under action-conditioned transitions.

Hamiltonian energy phase geometry

Energy geometry

Energy-organized structure in learned phase latents.

Updated separate PCA visualizations for q and p

Separate q/p PCA

Projection view for position-like and momentum-like phase variables.

Multi-task q-p coupling diagnostic

Multi-task q/p coupling

Coupling pattern across multiple control tasks.

Citation

Anonymous BibTeX placeholder.

The final citation will be updated after the paper link is ready. The placeholder preserves anonymous review.

@inproceedings{anonymous2027htc2,
  title     = {Hamiltonian Temporal Contrastive Curiosity for Reinforcement Learning Exploration},
  author    = {Anonymous Authors},
  booktitle = {Anonymous ICLR 2027 Submission},
  year      = {2027}
}