強化學習與邊緣 AI
VSPG Updates Discrete-Action Actors Without Backpropagation, Outperforming the DNN Baseline on Both SustainGym Benchmarks
VSPG stores each action as a unit hypervector, reformulating the softmax policy gradient as advantage-weighted vector superposition and normalization, without requiring automatic differentiation for the actor or optimizer state. It achieves higher returns and greater resilience to bit errors in two building-control climates, but performance depends heavily on the fixed encoder.
Vector-Symbolic Policy Gradient (VSPG) attempts to reformulate discrete-action policies using hyperdimensional computing. The system uses a fixed encoder to map states to high-dimensional unit vectors, while each action corresponds to a unit hypervector. Their inner product, multiplied by a temperature parameter, produces the softmax logits. The study proves that, under the standard policy-gradient surrogate, the actor update can be written exactly as `C ← row-normalize(C + ηΛᵀS)`: advantage-weighted state vectors are superposed onto the selected action, while corresponding components are subtracted from competing actions.
This reformulation eliminates the actor’s need for autograd or optimizer state from methods such as Adam or SGD. Each action vector can also be interpreted as a fixed-size compressed kernel memory that superposes advantage evidence from past states within a single representation. Inference requires only one inner product per action, and memory usage does not grow with the number of training samples. For binary hypervectors, the paper also derives a theoretical bound: given a fixed similarity margin, the probability that random bit flips change the greedy action decreases exponentially with vector dimensionality.
The experiments cover classic control, MiniGrid, and multi-agent SustainGym. After 500 training episodes on the building-control task, VSPG with FHRR or RFF encoding achieved average per-step returns of -7.28 and -7.11 in hot-dry and warm-humid climates, respectively, outperforming the DNN baseline’s -13.41 and -12.12; higher values are better. The authors also quantized the actor to 1, 2, 4, and 8 bits and then randomly flipped stored bits. Overall, VSPG degraded more gradually than both neural and original linear actors. The public code includes complete training scripts, hyperparameter-search configurations, and bit-error tests.
A key limitation is that “backpropagation-free” applies only to the VSPG actor; SustainGym’s centralized critic still estimates GAE during training. The encoder is also far from an inconsequential preprocessing step: Basis-VSPG’s returns fell to -40.36 and -79.65 in the two climates, while RFF also failed on the more difficult DoorKey task in some runs. Future work must compare actual energy consumption, memory footprint, and fault-tolerance gains on hardware, while determining whether the method can scale to high-dimensional or continuous action spaces.