Reward function compression facilitates goal-dependent reinforcement learning.
Nature communications · 2026
Abstract originale
Humans can uniquely assign value to novel, abstract outcomes to support reinforcement learning. However, this flexibility is cognitively costly and reduces learning efficiency. We propose that goal-dependent learning initially relies on capacity-limited working memory. With consistent experience, learners create a compressed reward function - a simplified rule - that transfers to long-term memory for automatic evaluation upon receiving feedback. This automaticity frees working memory resources, thereby boosting learning efficiency. Across six experiments, we demonstrate that learning is impaired by the size of the goal space but improves when this space allows for compression. Additionally, faster reward processing correlates with better learning. Although the algorithmic details remain to be established, computational modeling revealed that individual differences in compression efficiency, proxied by reward processing speed, led to higher choice accuracy. Together, our behavioral results and computational models suggest that efficient goal-directed learning relies on compressing complex goals into stable reward functions.
Parole chiave
Leggi il testo integrale originale
Testo estratto dalla pubblicazione: sono esclusi immagini, media, supplementi e parti soggette a diritti distinti.
Reward function compression facilitates goal-dependent reinforcement learning
Humans can uniquely assign value to novel, abstract outcomes to support reinforcement learning. However, this flexibility is cognitively costly and reduces learning efficiency. We propose that goal-dependent learning initially relies on capacity-limited working memory. With consistent experience, learners create a compressed reward function — a simplified rule — that transfers to long-term memory for automatic evaluation upon receiving feedback. This automaticity frees working memory resources, thereby boosting learning efficiency. Across six experiments, we demonstrate that learning is impaired by the size of the goal space but improves when this space allows for compression. Additionally, faster reward processing correlates with better learning. Although the algorithmic details remain to be established, computational modeling revealed that individual differences in compression efficiency, proxied by reward processing speed, led to higher choice accuracy. Together, our behavioral results and computational models suggest that efficient goal-directed learning relies on compressing complex goals into stable reward functions.
People can learn to achieve arbitrary goals, but this flexibility requires cognitive effort. Here, the authors show that the brain resolves this by progressively compressing complex reward information into simplified rules, freeing up working memory to facilitate reinforcement learning.
Introduction
Reinforcement learning is a powerful process by which agents learn to make decisions based on experienced rewards and punishments, which has seen broad applications across neuroscience, psychology, and machine learning1. The study of animal reinforcement learning (RL) has predominantly focused on two outcome categories: primary rewards, which satisfy basic physiological needs (e.g., food, water), and secondary rewards, which derive their value through learned associations with primary rewards (tokens, money, numeric points2). This focus has advanced our understanding of RL behavior and its neural basis in dopaminergic signaling3. However, primary and secondary rewards do not adequately capture the complexity of human reward processing4, especially in contexts involving abstract goals and deliberate planning5. For example, people often find activities rewarding in the absence, if not at the expense, of external incentives6. Even when the benefits of engaging in an activity are clear, people can attribute value to its successful completion a priori, without needing to directly associate new outcomes with established rewards. Indeed, humans can flexibly imbue novel outcomes with value and use them to guide learning through a process that recruits subcortical reward circuits7,8. Together, these findings suggest that humans use a third class of rewards: flexible, goal-dependent outcomes that derive value from current objectives rather than learned associations9.
Goal-dependent reinforcement learning implies the construction of a goal-dependent reward function, i.e., a mapping between an outcome space and scalar rewards. While individuals show remarkable flexibility in creating ad hoc reward functions according to current goals, the underlying process is often inefficient7. This is illustrated, for example, by the discrepancy between the high frequency at which people set New Year’s resolutions and the low rate at which they complete them10. However, the precise cognitive mechanisms by which humans align their value system to current goals—and which aspects make such alignment hard to maintain – remain uncharted. Identifying the key players involved in goal-dependent learning and their respective weaknesses could inform behavioral strategies that help people achieve their objectives in both personal and educational settings.
Setting up custom reward functions likely involves the interplay of multiple cognitive systems7. The existence of interactions between RL and executive functions is now well-established11,12. In particular, WM can act as a source of information for RL, affording faster or more efficient learning13,14. WM has also been proposed to support the verbalization of critical task components15,16 and the representation of states, actions, and rewards over which RL operates17. However, much less is known about how exchanges between RL and WM support goal-dependent reward functions, and how they might be limited. Building on existing work7, we designed a task where participants had to learn which responses led to desired outcomes, but using abstract images labeled as goal/nongoal as feedback instead of standard numeric points. We replicated our finding that people can indeed imbue abstract, novel stimuli with reinforcing properties, but that learning from goal-dependent outcomes resulted in worse learning performance (7,8; Experiment 3 in Supplementary note 1), highlighting the trade-offs between flexible goal-dependent value attribution and learning efficiency. This impairment persisted (albeit substantially smaller) when goal images were kept consistent for the entire duration of the task (8, Experiment 4 in Supplementary note 1). However, evidence for the precise reason behind such performance improvements remained inconclusive.
Here, we propose a dynamic model of goal-dependent reward processing to capture the observed trade-offs between flexibility and learning efficiency. As a real-life example, consider learning chess: a novice might initially hold specific winning positions in mind (queen on d5), then compress these into simple tactical rules (control the center squares), and eventually apply such patterns automatically without conscious effort, freeing mental resources for strategic planning. In our conceptual model, a working memory (WM) component enables us to instantly attribute goal-contingent rewarding properties even to abstract, novel goals. This allows for almost complete freedom in defining what counts as a reward (including usually unrewarding outcomes18), which can then be used as a signal for learning (Fig. 1a). However, such freedom comes at a cost, as it requires active maintenance of a goal19, using up resources otherwise utilized for the learning process itself20. Information stored repeatedly in WM transfers to long-term storage that resists cognitive load21–24. Consequently, we propose that pursuing similar goals consistently alleviates the burden on WM through the creation of a compressed reward function from high-dimensional stimuli, which contains the minimum amount of information required to recognize successful outcomes as such while trimming redundant details away (e.g., all goal images are purple). Such a lightweight reward function can eventually be transferred to longer-term memory storage, from where it may easily be retrieved without impinging on other executive functions required to support learning through RL or WM (11,12,15; Fig. 1b).
While we remain agnostic with regard to its algorithmic implementation in the brain and formal instantiations of reward function compression itself, our model makes specific predictions, which we test in this study. If WM supports goal evaluation, then learning should worsen as we increase the number of goals to build a reward function across the learning problem (i.e., WM load). In Experiment 1, we manipulate the number of unique goals participants need to learn from, and observe that learning parametrically decays. The model also predicts that performance will benefit from repeated encounters with the same set of goals when it either naturally fits within WM capacity or can be compressed to do so. Only then can the transfer to long-term memory occur, freeing up executive resources to support learning. In Experiment 2, we keep the number of goals constant across blocks, but vary whether the goal space can be compressed along a subset of its dimensions. We find that learning performance significantly improves when people are able to build compressed reward functions, compared to when they are not. In Experiments 3–6, we eliminate alternative mechanisms by which reward functions may be set following instruction, such as cost-free top-down value attribution or cached estimates of reward that are acquired iteratively. While no single experiment eliminates all competing hypotheses at once, the study as a whole tells a coherent story. Taken together and paired with computational modeling, our behavioral findings support reward function compression as a possible mechanism for efficient goal-dependent reinforcement learning.
Results
Experimental design
All experiments share the same fundamental structure, which we adapted from existing tasks7,13 where participants had to learn, by trial and error, the correct association between stimuli and specific keys on their computer keyboard (Fig. 2). Both Experiment 1 and Experiment 2 consisted of six blocks, divided into interleaved Points and Goals blocks (two and four of each, respectively) in a pseudorandomized order. Six stimuli were presented in each block and repeated 12 times in an interleaved fashion, for a total of 72 trials. Compared to the original RLWM task13, only having blocks with six stimuli in each allowed us to emphasize the RL component relative to value updates directly informed by WM. In the Points blocks, which reflected the structure of a standard RL task and served as a baseline, participants received numeric points as feedback for their responses to the stimuli. Each trial started with a fixation screen (0.5 s), followed by the presentation of +1 and +0 messages indicating desirable and undesirable outcomes, respectively (1.8 s). Participants then saw a stimulus image and were prompted to press one of three keys on their keyboard (1.5 s). Upon responding, they received deterministic feedback showing +1 or +0 messages based on the accuracy of their response (1.5 s) or a warning message if they failed to respond on time. Note that, although feedback was deterministic, participants were in a learning regime for the majority of the task, such that reward prediction errors did not saturate until the very end, if ever (Fig. 3c, g). Participants were instructed to collect desirable outcomes by pressing a separate key within the allotted time while ignoring undesirable outcomes (as in a Go/No-go task). Below, we refer to the latter stage of each trial as reward collection. The Goals blocks followed a similar structure but used abstract fractal-like images instead of numeric points. In each trial, before the stimulus was shown, a pair of fractal images was displayed with the labels Goal and Nongoal to indicate the desired outcome. Fractal images were created using the MandelBrowser and Artmatic Pro applications. Participants needed to encode and recall which images represented goals to learn the correct stimulus-action mappings, then collect goal outcomes on each trial. Correct action selection for each stimulus image deterministically led to the presentation of the current trial’s goal image.
Experiment 1: Goal-dependent value attribution is impacted by load
Our conceptual model (Fig. 1) proposes that participants will, over time, try to infer a simplified reward function that can be applied to the presented outcome, as opposed to remembering each goal image separately and later comparing it to the experienced outcome. The easier the reward function compression process, the faster goal-dependent reward functions can be ascribed to long-term memory, freeing up cognitive resources for the learning process itself. If only one goal is being pursued across a set of trials, it is easy to compress reward information from the complex fractal goal image to one or a few defining attributes (e.g., color, pattern, etc.) that distinguish it from the nongoal image. When multiple goals are present across trials, such an operation increases the load on WM and thus becomes harder. Therefore, we predicted that modulating the number of goal images in each block would result in performance gains or losses. Returning to our chess example, this would be akin to working toward a single target position across many games vs. juggling a repertoire of several distinct target positions. The latter is harder because resources needed for learning must be spared for active goal maintenance.
In each of the Goals blocks of Experiment 1, outcome images belonged to a set of one, two, three, or four pairs, such that each fractal image pair appeared 12, six, four, or three times per stimulus (Fig. 3a). Block order was pseudorandomized. We additionally ensured that a Points block appeared within the first three and last three blocks. We expected learning performance to be best in the block with a single goal, and worst in the block with four different goals to pursue, with intermediate learning efficiency in the conditions in between. Importantly, conditions did not differ in the amount of information participants had to memorize on a trial-by-trial basis, since only a single goal/non-goal pair was relevant in each trial. Thus, the difference in load was only effective across trials, in keeping with the assumption that participants would consider information from the entire goal space and try to build an overall goal-dependent reward function. Sixty-seven participants completed Experiment 1, 13 of whom did not match our inclusion criteria (see “Methods”). We analyzed data from the remaining 54 participants (74% female, ages 18–25, M = 20.13, SD = 1.48).
Learning performance was higher for Points, compared to Goals trials – even when compared to the easiest, single-goal block (t(53) = 3.74, P < 0.001, Cohen’s d = 0.41, 95% CI = [0.03, 0.1]), highlighting the cost of learning from goal-dependent outcomes even in a simple setting and confirming our previous findings8. To compare performance across blocks with one, two, three, or four pairs of goal/nongoal outcomes, we ran a linear mixed-effects model predicting average learning performance from the number of outcome image pairs in each Goals block, with random intercepts for participant number. Performance decreased with the number of different outcomes in a block (β = 0.073 ± 0.02, P < 0.001, 95% CI = [0.04, 0.11]; Fig. 3b, c). By contrast, reward collection d’ was not statistically different in Goals (M = 3.86 ± 0.08) and Points trials (M = 3.78 ± 0.09; t(53) = -0.81, P = 0.424, d = 0.12, 95% CI = [−0.27, 0.11], BF10 = 0.20 i.e., moderate evidence for the null hypothesis) or across Goals blocks (β = −0.01 ± 0.06, P = 0.820, 95% CI = [−0.12, 0.10]; Supplementary Fig. 1), showing that the load manipulation did not impact outcome identification and goal maintenance within trials.
In sum, modulating the number of goal/nongoal image pairs from one to four resulted in parametric impairments in learning. This sort of phenomenon, with relatively constrained ceiling-floor effects and proportionally increasing performance in between, is a hallmark of WM25, suggesting a similar executive function might be at play in the reward function compression process. However, it is also possible that the observed learning impairments were caused by interference, occurring more frequently in blocks with more unique goal images. To disentangle the two hypotheses, we devised Experiment 2.
Experiment 2: reward function compression facilitates goal-dependent learning
Our model predicts that the mere repetition of goals is not always sufficient for performance improvements: the goals must be structured in a way that allows for meaningful compression. If participants were simply benefiting from seeing the same goal images repeatedly (as might happen in basic interference accounts), then any repeated goal set should show improvement. However, if our reward function compression hypothesis is correct, performance should improve only when the goal space can be reduced to a simple, memorable rule. Experiment 2 tests this prediction by holding both the number of unique goal images and their frequency of presentation constant while manipulating whether the underlying structure of the goal space permits reward information compression through dimensionality reduction. To achieve this, we replaced fractal images with simpler geometric figures that varied along three different binary axes within each block (Fig. 3e). This resulted in eight figures per block, each having one of two shapes (e.g., square or triangle), colors (e.g., purple or yellow), and fills (full or empty). In Compressible blocks, a single feature was sufficient to separate goal from nongoal images, while other features were irrelevant (e.g., all goal images were full; classification type I in Shepard et al.26). In Incompressible blocks, all three features had to be taken into account to determine whether an outcome was a goal or a nongoal image (e.g., goal images were a full orange diamond, a full green circle, an empty green diamond, or an empty orange circle; classification type VI in Shepard et al.26). The same rules governed blocks of the same type for a given participant, but rule assignment was pseudorandomized across individuals. Going back to the chess analogy, the two Goal block types are like learning the same number of target positions, but differ in whether all winning positions can be generalized via common features (e.g., controlling the center) or depend on the conjunction of several features, leaving the player to memorize each target board in full. Seventy-one participants completed Experiment 2. After excluding 20 who failed to meet our inclusion criteria, we analyzed data from the remaining 51 participants (80% female, ages 18–32, M = 20.48, SD = 2.27).
As predicted, learning accuracy was significantly higher in Compressible than Incompressible blocks (t(50) = 3.31, P = 0.002, d = 0.54, 95% CI = [0.03, 0.13]; Fig. 3f–g). Since the number of occurrences of each goal outcome was identical across the two block types, these results cannot be attributed to differences in goal outcome repetitions (as could have been the case for Experiment 1). Differences in reward collection d’ based on the type of Goals block were significant (t(50) = 8.62, P < 0.001, d = 1.40, 95% CI = [0.64, 1.03]; Supplementary Fig. 1). However, they were not significantly correlated with corresponding differences in learning performance (ρ(49) = 0.17, P = 0.226). Moreover, a trial-level logistic mixed-effects model predicting choice accuracy confirmed a significant effect of Goals condition (Compressible vs. Incompressible; β = 0.51 ± 0.06, P < 0.001, 95% CI = [0.40, 0.63]), even after explicitly controlling for false positive (β = −0.40 ± 0.22, P = 0.071, 95% CI = [−0.84, 0.03]) and false negative reward collection errors (β = −1.58 ± 0.17, P < 0.001, 95% CI = [−1.91, −1.26]) on the previous encounter of that stimulus (in addition to stimulus iteration, block number, and their interaction with goal condition). These results suggest that failures in goal identification are unlikely to fully explain the main effect of the Goals condition. Despite new learning, reward collection RTs decreased from the first set of Goals blocks (M = 422.18 ± 13.92) to the second (where goal/nongoal images were reused; M = 316.06 ± 15.90, t(50) = 8.0, P < 0.001, d = 0.99, 95% CI = [79.47, 132.77]). This RT decrease was sharper for the Compressible, compared to the Incompressible condition: a linear mixed-effects model predicting reward collection RTs showed a significant interaction between Goals condition type and experiment phase (early vs. late; β = −13.58 ± 6.14, P = 0.027, 95% CI = [−25.62, −1.53]), controlling for stimulus iteration, block number, and their interaction with the experimental condition. Similar findings applied to learning performance increases across phases (early learning accuracy M = 0.53 ± 0.02, late learning accuracy M = 0.66 ± 0.02; t(50) = −9.45, P < 0.001, d = 1.00, 95% CI = [−0.16, −0.10]; Goals condition type by experiment phase interaction β = 0.03 ± 0.01, P = 0.004, 95% CI = [0.01, 0.04]). Together, these results suggest that learning benefits could not be fully attributed to more habitual choice behavior (Supplementary Fig. 2).
Overall, we find that, although they were never instructed to try to learn mappings between goal outcome features and reward, participants were able to leverage structure in the condition with compressible goal outcomes, which ultimately benefited their learning.
Reward processing correlates with performance
During the task, participants were instructed to collect positive outcomes (either points or goal images) by pressing a specific key while ignoring negative outcomes, a phase we refer to as reward collection (Fig. 2). In this section, we explore reward collection reaction times (RTs; Supplementary Fig. 1) as a behavioral marker of reward function compression. As participants switch from controlled to more automatic reward processing, their reward collection should become faster. The quicker an individual can recognize and react to an outcome as desirable, the less cognitive effort is presumably expended on this evaluative step. This, in turn, could free up cognitive capacity for the main task of learning new stimulus-response associations. Therefore, we expected individual differences in reward collection RTs across conditions to correlate with corresponding differences in learning performance. By contrast, faster reward collection RTs due to motor habituation, general arousal, or automatic valuation would not systematically differ across the various conditions.
In each experiment, average block-wise individual differences between Points and Goals learning accuracy were strongly correlated with corresponding differences in median log reward collection RTs (Supplementary Fig. 4). Similarly, performance differences between pairs of Goals conditions within the same experiment correlated with corresponding reward collection RT differences (Fig. 3d, h and Supplementary Fig. 5). We sought to ensure that this finding was not driven by differences in engagement or difficulty across conditions. To this end, we reasoned that, if a third, non-specific factor jointly drove block-level changes in choice accuracy and reward collection RTs, we should also expect it to drive choice RTs, and thus a similar correlation between performance and RT. Contrary to this prediction, we found weak or no correlation between choice RT and performance differences across conditions of Experiments 1–2 (Supplementary Figs. 6 and 7). This shows that block-level reward collection RT is uniquely predictive of accuracy. This finding is also counter to the competing hypothesis that goal maintenance impacts choice through a dual-tasking effect, whereby the cognitive load of maintaining the goal to interpret goal-dependent feedback would interfere with and slow down the choice process.
This reliable—albeit correlational—link between reward collection RTs and learning success is difficult to reconcile with simpler alternative models. For instance, an immediate top-down value attribution process would likely be uniformly efficient, leading to consistent reward collection speeds. Similarly, bottom-up explanations that lack an abstraction mechanism would struggle to account for these results—especially in Experiment 2, where goal outcomes were equally frequent but more or less diverse. By contrast, the observed pattern of results strengthens the case for an active reward compression process supporting goal-dependent learning, whereby WM information can be transferred to longer-term storage and be applied without interfering with other executive functions. To test the internal validity of our model, we turn to computational models, leveraging reward collection RTs as a signature of reward function compression.
Computational models leveraging signatures of reward function compression accurately recover human behavior
To gain a closer understanding of the latent processes guiding learning and decision-making in our task, we fit a set of candidate models that shared a basic RL architecture27. Details about computational modeling methods are provided in the “Methods” section. In brief, the value of each stimulus-action pair was updated on each trial through the delta rule28. Actions were selected via a noisy softmax policy29. This basic RL component had a learning rate α, a learning rate multiplier for negative outcomes ζ, a fixed inverse temperature β = 25, and a free noise parameter ϵ. We note that this RL component is meant to capture the reward-dependent learning process, with no claim that it maps onto specific underlying neural mechanisms1,30; indeed, we expect that this RL component additionally reflects contributions of working memory to learning31. All models also included additional mechanisms, such as forgetting and stickiness, which we found to improve fit. In particular, forgetting proxies RL-WM mixtures that could not be disentangled due to the fixed set-size design of the experiment. An additional learning process tracked stimulus-action associations in an outcome-insensitive, habit-like manner30. This component had a separate learning rate τ for outcome-insensitive value estimates, and a parameter κ that defined the mixture of outcome-sensitive and outcome-insensitive values the RL learner would take into account.
This model, in its base form, best captured the overall learning dynamics, but could not support differences between Points and Goals blocks or within-Goal block sub-conditions. To remedy this, we expanded the base model with separate free parameters for each condition of interest, one mechanism at a time. This resulted in a new family of models where Point trials and each condition of the goal blocks had a specific learning rate, noise parameter, or stickiness bias, etc. Among these, the winning model was the one with separate learning rates α for each type of block (Supplementary Fig. 8). In line with previous studies7, this result suggests that the value updating process (and not, for instance, the choice selection aspect) was the most negatively impacted by goal-dependent reward attribution.
Treating reward collection RTs as proxies for reward processing efficiency (potentially gained through long-term memory transfer), we also built a family of models where one of the parameters was directly influenced by the block-wise median (log) reward collection RT for the corresponding condition (i.e., Points, or a specific Goals sub-condition; see “Methods” for details) to a degree defined by the parameter ω. This model design rests on the assumption that faster RTs indicate more efficient processing and that, if reward-processing gains free up shared resources, they should relate to specific benefits in learning. In contrast, if learning improvements were unrelated to reward processing, linking the two systems would not capture behavior more accurately than a model with parameters independent of reward collection RTs. By applying this method to the learning rate, we were able to capture participants’ behavior (Experiment 1: Fig.3b, c; Experiment 2: Fig. 3f–g) at least as well, and more parsimoniously than the model with free parameters for each experimental condition (Experiment 1: ΔAIC = 0.91, Wilcoxon’s W = 564.0, P = 0.124, r = 0.21; Experiment 2: ΔAIC = 5.95, W = 225.0, P < 0.001, r = 0.57; Supplementary Fig. 8). This finding provides formal support for our hypothesis that, when outcome evaluation (proxied by reward collection RT) is efficient, more cognitive resources can be allocated to update stimulus-action associations from feedback. Moreover, the model with learning rates conditioned on reward collection RTs provided a better fit and posterior predictive checks to participants’ data than a model where the choice noise parameter ϵ was altered by reward collection RTs (Experiment 1: ΔAIC = 8.24, W = 84.0, P < 0.001, r = 0.77; Experiment 2: ΔAIC = 8.53, W = 136.0, P < 0.001, r = 0.69; Supplementary Figs. 9 and 13), showing that the bottleneck in goal-dependent learning occurs at the value-updating stage. We note that these results cannot simply be attributed to a shared variance in the learning process captured by reward collection RT. To illustrate this point, we also fit a model where the learning rate is affected by average choice RT and show that it does not explain behavior as accurately (Experiment 1: ΔAIC = 5.32, W = 302, P < 0.001, r = 0.51; Experiment 2: ΔAIC = 7.10, W = 238, P < 0.001, r = 0.56; Supplementary Figs. 9 and 14).
Additional experiments
We conducted four additional experiments (Supplementary notes 1, 2 and Supplementary Fig. 3) that provide further support to our model by establishing basic phenomena or eliminating some key alternative explanations. Experiments 3 and 4 are re-analyses of previous studies8. In Experiment 3, goal images changed on each trial, which made reward function compression impossible and forced participants to attribute value in a one-shot manner. This experiment provides a close replication of McDougle et al.7 and shows that goal-dependent learning is more effortful than learning from standard reinforcers. Here, we also find that simple lapses in goal outcome encoding or recognition do not explain impairments in learning performance. This is consistent with the results presented above and suggests that while goal maintenance in itself is efficient, it is a costly process that can utilize resources otherwise allocated to learning. In Experiment 4, goal images were kept consistent across blocks, facilitating reward function compression. We find that, while goal-dependent learning is initially impaired, it eventually reaches similar efficiency levels to learning from standard rewards. We attribute such dynamic changes to a transfer of information from WM to longer-term memory storage. Confirming this hypothesis, we find that individual preferences for goal, over nongoal outcome images—initially neutral—become stronger over time. Although Experiment 2 already showed that repeating goals is not sufficient to achieve performance savings, we provide a more controlled test for this hypothesis in Experiment 5, where we directly manipulate the number of goal outcome repetitions within a block. We find no performance gains for more frequent goal outcomes, which we attribute to the fact that, when the goal space is beyond WM capacity, there is no opportunity for reward function compression, meaning WM is continually burdened. In Experiment 6, we directly address the hypothesis that goal-dependent value attribution is purely done in a one-shot manner, as opposed to being sensitive to the previous history of goal setting. To this end, we manipulate whether goal/nongoal labels remain consistent across trials, controlling for the number of unique goals and how often each is presented, and find that learning is specifically impaired by inconsistent goal/nongoal image labeling across trials. This result suggests that goal-dependent learning takes advantage of repeated goal instances, which we attribute to reward function compression and transfer to longer-term memory storage, and which can make resources available for other online computations.
Discussion
From mundane activities to long-term ambitions, humans are remarkably capable of setting and pursuing a vast range of goals9,32. This flexibility, however, often comes at the cost of efficiency. In this study, we introduced a computational framework that formalizes such trade-offs. Our model posits that goal-dependent learning is initially supported by a fast, but costly, working memory (WM) system that maintains goal representations to evaluate outcomes and inform the reinforcement learning (RL) process7. While this allows for one-shot value attribution, it inevitably takes away from the limited capacity of WM, at the cost of either less robust or less precise task-relevant representations, which ultimately impair performance20. With consistent experience, however, learners can create a compressed reward function – a simplified representation containing only the essential features needed to identify a goal – which can then be transferred to long-term memory, enabling the reallocation of WM resources and boosting learning efficiency.
Through a series of experiments and computational modeling, we found support for this framework. Experiment 1 demonstrated that goal-dependent learning is parametrically impaired by the number of possible goals: as this number increased within the expected range of WM capacity, learning accuracy decreased. This finding aligns with classic research showing the capacity-limited nature of WM and its impact on concurrent tasks19,25, and shows that participants seem to maintain the goal space in WM, despite not needing to do so, since goals were instructed on each trial. This active maintenance is a prerequisite for leveraging executive functions to build a goal-dependent reward function. By keeping the number of goal outcomes constant but varying the complexity of the rule defining a goal, we showed in Experiment 2 that learning significantly improves when the goal space is compressible. This suggests that performance improvements are not merely a function of repeated exposure to the same goals, but rather depend on the ability to extract a simple, generalizable reward function – especially when the goals themselves are too complex to be fully stored in WM. Several mechanisms might support reward function compression, including attention-based feature selection33,34. While this specific study allowed for associations between success and nameable rules, similar processes might apply to differently learned reward functions, which may or may not require compression. In addition, the observed correlation between faster reward collection and better learning performance showed that more efficient reward processing interferes less with the resource-demanding process of task pursuit, leading to more efficient learning.
Our findings also rule out competing explanations. In their initial study, McDougle et al.7 proposed that the difference in learning rate between Points and Goals could be attributed to occasional lapses occurring in Goals trials, where participants failed to encode or recognize the goal image and subsequently update the probability of repeating the chosen response in future iterations. Excluding this possibility, Experiments 3–4 showed excellent goal outcome recognition in spite of performance differences. In principle, dynamic performance improvements over the task horizon could be explained by cached estimates for the value of an initially neutral outcome (the goal image) that becomes more predictive of reward over time (reminiscent of temporal difference learning; see35,36 for similar principles). However, Experiments 5 and 6 showed that manipulating the number of times a goal image is experienced does not, by itself, improve learning—in line with our proposal that reward function compression is necessary for performance gains. Nonetheless, we note that repetition likely fosters long-term reward functions with small enough goal spaces. Another possibility is that goal-dependent reward attribution is done in a completely cost-free, top-down manner that does not take the entire goal spaces into account. By illustrating that performance drops when goal/nongoal labels for the same outcome image are inconsistent with each other, Experiment 6 excludes this possibility. Finally, computational modeling in Experiments 1-2 showed that the performance bottleneck is not at the decision-making step, but rather at the outcome evaluation and integration stages.
Our findings extend a growing body of literature highlighting the interplay between executive functions and RL11–13,31. WM in particular is required to support several aspects of RL, including binding relevant elements of the task at hand, retrieving recent episodes, and even replacing RL entirely11,12,15,17,31. Here, we focus on the less explored aspect of reward function construction and utilization. We build on prior studies showing that learning from novel, abstract goals is possible but effortful7,8,37, and propose a specific mechanism – reward function compression—to explain how this initial cost can be overcome.
Our suggested process of reward function compression and transfer to long-term memory is consistent with long-standing theories, which established that consistent practice allows cognitive processes to shift from controlled, effortful search to automatic, effortless retrieval from memory21,23,38. For this to occur, the information maintained in WM must be consistent22. A similar dynamic occurs in action-effect learning, where outcomes produced by responses gradually become incorporated into action representations within a few trials39, but only under restricted conditions40. In our paradigm, it is only when the goal space is stable and simple enough to be held in WM (as in our single goal pair condition of Experiment 1 or the Compressible condition of Experiment 2) that a consistent mapping from outcomes to value can be learned and automatized. However, a formal model of reward functions’ long-term memory transfer remains to be established.
In real life, a major challenge of pursuing arbitrary goals is that they exist in a high-dimensional state space where most features are uninformative41. Our work suggests that a key mechanism for efficient learning in such environments is the ability to compress information about rewarding outcomes, ignoring irrelevant features to facilitate generalization and valuation. This resonates with philosophical concepts like value capture, where individuals gravitate towards simple, quantifiable metrics, sometimes at the expense of the richer, more complex values they are meant to represent42. Our work suggests compressed reward functions are sought because they help reduce cognitive demands and ensure learning efficiency – albeit towards a different behavior than initially intended.
Our work also opens several further questions. The goals in our experiments were simple and highly simplified. Future research should investigate whether the principles of reward function compression apply in more naturalistic settings, especially with self-imposed goals. Furthermore, the process by which people create compressed reward functions from raw example outcomes—which we did not model in detail—remains to be elucidated. Follow-up work in this direction might benefit from existing literature on function learning. For instance, reward functions may be created based on learned rules, judgments of similarity, or a mixture of the two43–46. It would also be interesting to test whether lossy reward function compression leads to systematic errors in value generalization to newly experienced outcomes. More generally, future work should propose and test a model of how reward function compression occurs dynamically on a trial-by-trial basis, and how it is implemented neurally. Moreover, the present study focused on goal-dependent values as a third kind of reinforcer, but goals might also impact primary or secondary rewards18,47,48. Systematic differences in reward function compression between deterministic tasks (such as the ones presented here) and probabilistic goal-dependent RL tasks7 also remain to be tested. Another avenue for future research would thus be to test the directionality of the association between the performance differences predicted by the reward function compression hypothesis and reward collection RTs, since we can only claim correlation, not causation. For example, faster reward collection RTs could be due to learning improvements, leading participants to expect desirable outcomes more reliably. Finally, our sample was relatively uniform, consisting of university undergraduates. While our exclusion rates were typical for online studies of similar difficulty18,49,50, we cannot make conclusions about the applicability of our results to less attentive participants.
In conclusion, our work provides a formal model and empirical evidence for how humans manage the trade-off between flexibility and efficiency in goal-directed learning. We demonstrate that the ability to learn from novel goal-conditioned outcomes is constrained by WM capacity and facilitated by the creation of compressed reward functions that can be transferred to long-term memory. By offering new insights into the cognitive science of motivation, our study provides a potential avenue for developing behavioral and educational strategies to help people achieve their ambitions.
Methods
All data and analysis code are publicly available at https://doi.org/10.5281/zenodo.2206928951.
Data collection
All participants were recruited through the University of California, Berkeley’s pool of participants (RPP) and received course credit for their participation. Upon providing informed consent, participants completed the experiment online from their own devices. The University of California, Berkeley Institutional Review Board approved the experimental procedure for all experiments (protocol #2024-04-17357). Sex and gender were not considered in the study design. Participants’ sex was determined based on self-report. The experiment was coded using jsPsych (version 6)52 and custom software.
To reduce spurious effects due to inattentive participants, exclusion criteria were devised based on visually identified elbow points53. Participants were excluded if they: 1) failed to respond on more than 25 trials in total, 2) missed more than 10 trials consecutively, 3) selected the same response more than 15 times in a row, 4) failed to accurately collect rewards in Points trials, as evidenced by either a d’ below or equal to 1, or more than 20 reward collection errors, 5) did not understand the instructions for reward collection in Goals trials, as evidenced by a reward collection d’ below or equal to 0, or more than 50 reward collection errors in the Goals condition, 6) failed to collect rewards entirely in one or more blocks, 7) had an average response time of less than 300 ms in either the Points or the Goals condition (or both). Participants under one or more such categories were excluded from further analysis. In each new experiment, we aimed to gather data from about 50 participants after applying exclusion criteria, guided by prior literature using similar reinforcement learning and working memory paradigms7,13.
Data analysis
We used the pingouin package (version 0.5.5)54 for Wilcoxon signed-rank tests, correlations, Cohen’s d and r effect sizes, and Bayesian Factors, the pymer4 package (0.8.2)55 for mixed-effects models, and the scipy package (version 1.16)56 for other statistical tests. We used pandas (version 2.3.1)57 and numpy (version 2.3.2)58 packages for data preprocessing. We used double-sided t-tests to compare average learning performance across pairs of outcome conditions for simplicity, consistency across tests, robustness with sufficiently large sample sizes, and the availability of statistical software. However, results remain unchanged when the non-parametric Wilcoxon signed-rank test is applied to the same comparisons. When comparing average scores across different experiments with unequal variances, we applied Welch’s correction. For null t-test results, we derived Bayes factors via the pingouin package54, and interpreted them following Lee and Wagenmakers59. All reported correlations are computed as Spearman’s ρ. Results for mixed-effects models (MEMs) were estimated via the pymer4 package60 and included separate intercepts for each participant. All continuous variables entered in MEMs were first standardized. When block numbers were added to the MEM, they refer to the block number within the corresponding Points or Goals condition. Whenever stimulus iterations were added to the MEM, they were entered as raw values, but we obtained similar results when first taking their log. Categorical variables were first turned into continuous variables, with the following coding scheme. When comparing between Points and Goals, block conditions were coded as 0.5 and −0.5, respectively. When comparing different conditions across Goals blocks, we used 0, −0.5, −1, and −1.5 for one, two, three, and four-pair blocks in Experiment 1 and 0.5 and −0.5 for the Compressible and Incompressible blocks in Experiment 2. When comparing different phases within Experiment 2, we used −0.5 for the first set of blocks and 0.5 for the second. Throughout the manuscript, the ± symbol refers to the S.E.M.
Computational modeling
All models shared the base mechanisms of a reinforcement learning (RL) agent that tracks Q values for each stimulus and response, yielding an Ns × Na table where Ns = 6 is the number of stimuli, and Na = 3 is the number of possible responses in each block. Values are initialized at Q0 = 1/Na at the beginning of each block, i.e., with uniform probability. Upon receiving feedback r on trial t, the value of the chosen response (ac) to the presented stimulus (s) is updated via the delta rule28: 1\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$${\delta }_{s,{a}_{c},t}={r}_{s,{a}_{c},t}-{Q}_{t}(s,{a}_{c})$$\end{document}δs,ac,t=rs,ac,t−Qt(s,ac)2\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$${Q}_{t+1}(s,{a}_{c})={Q}_{t}(s,{a}_{c})+\alpha \cdot {\delta }_{s,{a}_{c},t}$$\end{document}Qt+1(s,ac)=Qt(s,ac)+α⋅δs,ac,t where α is the learning rate. A multiplier ζ was applied to the learning rate when outcomes were negative61. Q values were converted to a policy via the softmax function29 with inverse temperature β and the noise parameter ϵ to capture random errors in the participants’ choices: 3\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$${P}_{t}({a}_{c}| s)=(1-\epsilon )\dfrac{\exp (\beta \cdot {Q}_{t}(s,{a}_{c}))}{{\sum }_{i}\exp (\beta \cdot {Q}_{t}(s,{a}_{i}))}+\epsilon \dfrac{1}{{N}_{a}}$$\end{document}Pt(ac∣s)=(1−ϵ)exp(β⋅Qt(s,ac))∑iexp(β⋅Qt(s,ai))+ϵ1Na Because it trades off with other parameters (e.g., the learning rate—potentially undermining parameter recoverability), β = 25 was a fixed parameter in all models, as is common practice with similar models and tasks14,62. Note that decision noise is captured by the free parameter ϵ mentioned above. On each trial, a forgetting rate ϕ brought Q values closer to 1/Na7: 4\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$${Q}_{t}={Q}_{t}+\phi ({Q}_{0}-{Q}_{t})$$\end{document}Qt=Qt+ϕ(Q0−Qt)
After a decision was made and Q values were updated, a stickiness value η ∈ [−1, 1] was added to Qt(s, ac), where ac is the recently chosen action63. This yielded a combined value table W, which entered the softmax (Eq. (3)) on the next trial and made the model biased towards (or against) the previous response in a stimulus- and outcome-insensitive fashion. Unchosen actions were left unmodified. Hence: 5\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$${W}_{t+1,{a}_{i}}=\left\{\begin{array}{l}{Q}_{t,{a}_{i}}+\eta \,{{\mbox{if}}}\,{a}_{i}={a}_{c}\quad \\ {Q}_{t,{a}_{i}}\,{{\mbox{if}}}\,{a}_{i}\,\ne\, {a}_{c}\quad \hfill\end{array}\right.$$\end{document}Wt+1,ai=Qt,ai+ηifai=acQt,aiifai≠ac Stickiness thus captures biases in participants’ choice selection, which drove them to either replicate or avoid the most recently chosen action, irrespective of the outcome or the stimulus being presented.
Models also kept track of chosen stimulus-response pairs in a stimulus-dependent, but outcome-independent fashion30. This was implemented through a separate table of values, called C, for each stimulus and response. After being initialized at C = 0, the value of the chosen stimulus-action pair was updated via the delta rule, but ignoring the outcome: 6\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$${C}_{t+1}(s,{a}_{c})={C}_{t}(s,{a}_{c})+\tau \cdot (1-{C}_{t}(s,{a}_{c}))$$\end{document}Ct+1(s,ac)=Ct(s,ac)+τ⋅(1−Ct(s,ac)) where τ is the learning rate. C values were mixed with Q values as determined by the value of parameter κ before entering the softmax (Eq. (3)): 7\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$${M}_{t}=(1-\kappa ){Q}_{t}+\kappa {C}_{t}$$\end{document}Mt=(1−κ)Qt+κCt In models with stickiness, Qt in Eq. (7) was replaced with W.
All the parameters listed in the previous sections could optionally vary according to the block’s condition (Points or Goals) and within-block sub-conditions (specific manipulations within Goals blocks). A separate family of models had only one instance of each free parameter, but this value was then scaled by the free parameter ω and multiplied by the median log reward collection RT (or choice RT, for alternative models) for the corresponding block and condition. This was intended to model how reward compression efficiency impacted the learning process. We use the median log RTs to mitigate the impact of any shared variance between reward collection/choice RT and trial-by-trial noise.
The fmincon function from MATLAB 2022a (via createOptimProblem and the Python MATLAB engine) was used to obtain the best-fitting parameters for all computational models. The function was used to optimize the log-likelihood of the data under the specified model architecture64. All free parameters were bounded between 0 and 1, except for η (stickiness parameter), which could take any value between −1 and 1 (since persistence biases are often found, but our deterministic task design could encourage switch behavior) and ω, which was constrained between −10 and 10.
Models were evaluated in terms of both predictive and generative validity65. We used the Akaike information criterion (AIC) to assess each model’s relative ability to predict the observed data. For differences in model AICs, we report the Wilcoxon signed-rank test statistic. To assess each model’s ability to reproduce the effects of interest, we simulate its behavior with the best-fit 20 times per participant, amounting to roughly 1000 total iterations per model.
We verified that models were identifiable in all key comparisons (Supplementary Fig. 10), and that the winning model’s best-fitting parameters (Supplementary Fig. 11) were recoverable (Supplementary Fig. 12).
Reporting summary
Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.
Supplementary information
References
1.Eckstein MK Wilbrecht L Collins AG What do reinforcement learning models measure? interpreting model parameters in cognition and neuroscience Curr. Opin. Behav. Sci. 2021 41 128 137 34984213 10.1016/j.cobeha.2021.06.004 Eckstein, M. K., Wilbrecht, L. & Collins, A. G. What do reinforcement learning models measure? interpreting model parameters in cognition and neuroscience. Curr. Opin. Behav. Sci.41, 128–137 (2021).10.1016/j.cobeha.2021.06.004 PMC8722372 34984213
2.Skinner BF Selection by consequences Behav. Brain Sci. 1984 7 477 481 10.1017/S0140525X0002673X Skinner, B. F. Selection by consequences. Behav. Brain Sci.7, 477–481 (1984).
3.Niv Y Reinforcement learning in the brain J. Math. Psychol. 2009 53 139 154 10.1016/j.jmp.2008.12.005 Niv, Y. Reinforcement learning in the brain. J. Math. Psychol.53, 139–154 (2009).
4.Daniel R Pollmann S A universal role of the ventral striatum in reward-based learning: evidence from human studies Neurobiol. Learn. Mem. 2014 114 90 100 24825620 10.1016/j.nlm.2014.05.002 Daniel, R. & Pollmann, S. A universal role of the ventral striatum in reward-based learning: evidence from human studies. Neurobiol. Learn. Mem.114, 90–100 (2014).10.1016/j.nlm.2014.05.002 PMC4143465 24825620
5.Diuk C Tsai K Wallis J Botvinick M Niv Y Hierarchical learning induces two simultaneous, but separable, prediction errors in human basal ganglia J. Neurosci. 2013 33 5797 5805 23536092 10.1523/JNEUROSCI.5445-12.2013 Diuk, C., Tsai, K., Wallis, J., Botvinick, M. & Niv, Y. Hierarchical learning induces two simultaneous, but separable, prediction errors in human basal ganglia. J. Neurosci.33, 5797–5805 (2013).10.1523/JNEUROSCI.5445-12.2013 PMC3865543 23536092
6.Blain B Sharot T Intrinsic reward: potential cognitive and neural mechanisms Curr. Opin. Behav. Sci. 2021 39 113 118 10.1016/j.cobeha.2021.03.008 Blain, B. & Sharot, T. Intrinsic reward: potential cognitive and neural mechanisms. Curr. Opin. Behav. Sci.39, 113–118 (2021).
7.McDougle SD Ballard IC Baribault B Bishop SJ Collins AG Executive function assigns value to novel goal-congruent outcomes Cereb. Cortex 2022 32 231 247 10.1093/cercor/bhab205 PMC8634563 34231854 McDougle, S. D., Ballard, I. C., Baribault, B., Bishop, S. J. & Collins, A. G. Executive function assigns value to novel goal-congruent outcomes. Cereb. Cortex32, 231–247 (2022).10.1093/cercor/bhab205 PMC8634563 34231854
8.Molinaro, G. & Collins, A. G. Human hacks and bugs in the recruitment of reward systems for goal achievement. In Proceedings of the Annual Meeting of the Cognitive Science Society, vol. 45 (2023).
9.Molinaro G Collins AG A goal-centric outlook on learning Trends Cogn. Sci. 2023 27 1150 1164 37696690 10.1016/j.tics.2023.08.011 Molinaro, G. & Collins, A. G. A goal-centric outlook on learning. Trends Cogn. Sci.27, 1150–1164 (2023).10.1016/j.tics.2023.08.011 37696690
10.Woolley K Giurge LM Fishbach A Adherence to personal resolutions across time, culture, and goal domains Psychol. Sci. 2025 36 607 621 40663621 10.1177/09567976251350960 Woolley, K., Giurge, L. M. & Fishbach, A. Adherence to personal resolutions across time, culture, and goal domains. Psychol. Sci.36, 607–621 (2025).10.1177/09567976251350960 PMC13428958 40663621
11.Rmus M McDougle SD Collins AG The role of executive function in shaping reinforcement learning Curr. Opin. Behav. Sci. 2021 38 66 73 35194556 10.1016/j.cobeha.2020.10.003 PMC8859995 Rmus, M., McDougle, S. D. & Collins, A. G. The role of executive function in shaping reinforcement learning. Curr. Opin. Behav. Sci.38, 66–73 (2021).10.1016/j.cobeha.2020.10.003 PMC8859995 35194556
12.Yoo AH Collins AG How working memory and reinforcement learning are intertwined: a cognitive, neural, and computational perspective J. Cogn. Neurosci. 2022 34 551 568 34942642 10.1162/jocn_a_01808 Yoo, A. H. & Collins, A. G. How working memory and reinforcement learning are intertwined: a cognitive, neural, and computational perspective. J. Cogn. Neurosci.34, 551–568 (2022).10.1162/jocn_a_01808 34942642
13.Collins AG Frank MJ How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis Eur. J. Neurosci. 2012 35 1024 1035 22487033 10.1111/j.1460-9568.2011.07980.x Collins, A. G. & Frank, M. J. How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis. Eur. J. Neurosci.35, 1024–1035 (2012).10.1111/j.1460-9568.2011.07980.x PMC3390186 22487033
14.Collins AG Frank MJ Within-and across-trial dynamics of human EEG reveal cooperative interplay between reinforcement learning and working memory Proc. Natl. Acad. Sci. 2018 115 2502 2507 29463751 10.1073/pnas.1720963115 Collins, A. G. & Frank, M. J. Within-and across-trial dynamics of human EEG reveal cooperative interplay between reinforcement learning and working memory. Proc. Natl. Acad. Sci.115, 2502–2507 (2018).10.1073/pnas.1720963115 PMC5877949 29463751
15.Yoo AH Keglovits H Collins AG Lowered inter-stimulus discriminability hurts incremental contributions to learning Cogn. Affect. Behav. Neurosci. 2023 23 1346 1364 10.3758/s13415-023-01104-5 37656373 Yoo, A. H., Keglovits, H. & Collins, A. G. Lowered inter-stimulus discriminability hurts incremental contributions to learning. Cogn. Affect. Behav. Neurosci.23, 1346–1364 (2023).10.3758/s13415-023-01104-5 PMC10545593 37656373
16.Rmus M Zou A Collins AG Choice type impacts human reinforcement learning J. Cogn. Neurosci. 2023 35 314 330 10.1162/jocn_a_01947 36473098 Rmus, M., Zou, A. & Collins, A. G. Choice type impacts human reinforcement learning. J. Cogn. Neurosci.35, 314–330 (2023).10.1162/jocn_a_01947 36473098
17.Ham H McDougle SD Collins AG Dual effects of dual-tasking on instrumental learning Cognition 2025 264 106228 40651360 10.1016/j.cognition.2025.106228 Ham, H., McDougle, S. D. & Collins, A. G. Dual effects of dual-tasking on instrumental learning. Cognition264, 106228 (2025).10.1016/j.cognition.2025.106228 40651360
18.Pan, T.-F., Molinaro, G. & Collins, A. G. E. When 0 is good: instrumental learning with counterintuitive goals decreases working memory engagement. In Proceedings of the Annual Meeting of the Cognitive Science Society, vol. 47 (2025).
19.Baddeley A Working memory Comptes Rendus de. l’Acad.émie des. Sci.-Ser. III-Sci. de. la Vie 1998 321 167 173 10.1016/s0764-4469(97)89817-4 9759336 Baddeley, A. Working memory. Comptes Rendus de. l’Acad.émie des. Sci.-Ser. III-Sci. de. la Vie321, 167–173 (1998).10.1016/s0764-4469(97)89817-4 9759336
20.Oberauer K Working memory and attention–a conceptual analysis and review J. Cognit. 2019 2 36 10.5334/joc.58 PMC6688548 31517246 Oberauer, K. Working memory and attention–a conceptual analysis and review. J. Cognit.2, 36 (2019).10.5334/joc.58 PMC6688548 31517246
21.Carlisle NB Arita JT Pardo D Woodman GF Attentional templates in visual working memory J. Neurosci. 2011 31 9315 9322 21697381 10.1523/JNEUROSCI.1097-11.2011 Carlisle, N. B., Arita, J. T., Pardo, D. & Woodman, G. F. Attentional templates in visual working memory. J. Neurosci.31, 9315–9322 (2011).10.1523/JNEUROSCI.1097-11.2011 PMC3147306 21697381
22.Logan GD Toward an instance theory of automatization Psychol. Rev. 1988 95 492 10.1037/0033-295X.95.4.492 Logan, G. D. Toward an instance theory of automatization. Psychol. Rev.95, 492 (1988).
23.Shiffrin RM Schneider W Controlled and automatic human information processing: Ii. perceptual learning, automatic attending and a general theory Psychol. Rev. 1977 84 127 10.1037/0033-295X.84.2.127 Shiffrin, R. M. & Schneider, W. Controlled and automatic human information processing: Ii. perceptual learning, automatic attending and a general theory. Psychol. Rev.84, 127 (1977).
24.Woodman GF Chun MM The role of working memory and long-term memory in visual search Vis. Cognit. 2006 14 808 830 10.1080/13506280500197397 Woodman, G. F. & Chun, M. M. The role of working memory and long-term memory in visual search. Vis. Cognit.14, 808–830 (2006).
25.Luck SJ Vogel EK The capacity of visual working memory for features and conjunctions Nature 1997 390 279 281 10.1038/36846 9384378 Luck, S. J. & Vogel, E. K. The capacity of visual working memory for features and conjunctions. Nature390, 279–281 (1997).10.1038/36846 9384378
26.Shepard RN Hovland CI Jenkins HM Learning and memorization of classifications Psychol. Monogr.: Gen. Appl. 1961 75 1 10.1037/h0093825 Shepard, R. N., Hovland, C. I. & Jenkins, H. M. Learning and memorization of classifications. Psychol. Monogr.: Gen. Appl.75, 1 (1961).
27.Sutton, R. S. & Barto, A. G. Reinforcement learning: an introduction (MIT Press, 2018).
28.Rescorla, R. A. & Wagner, A. R. A theory of pavlovian conditioning: Variations in the effectiveness of reinforcement and non-reinforcement. Current Research andTheory 64–99 (1972).
29.Luce RD On the possible psychophysical laws Psychol. Rev. 1959 66 81 13645853 10.1037/h0043178 Luce, R. D. On the possible psychophysical laws. Psychol. Rev.66, 81 (1959).10.1037/h0043178 13645853
30.Collins AG A habit and working memory model as an alternative account of human reward-based learning. Nat. Hum. Behav. 2026 10 357 369 10.1038/s41562-025-02340-0 41249816 PMC12932107 Collins, A. G. A habit and working memory model as an alternative account of human reward-based learning. Nat. Hum. Behav.10, 357–369 (2026).10.1038/s41562-025-02340-0 PMC12932107 41249816
31.Collins AG The tortoise and the hare: Interactions between reinforcement learning and working memory J. Cogn. Neurosci. 2018 30 1422 1432 29346018 10.1162/jocn_a_01238 Collins, A. G. The tortoise and the hare: Interactions between reinforcement learning and working memory. J. Cogn. Neurosci.30, 1422–1432 (2018).10.1162/jocn_a_01238 29346018
32.Colas, C., Chu, J., Molinaro, G. & Hawkins, R. What should i do now? goal-centric outlooks on learning, exploration, and communication. In Proceedings of the Annual Meeting of the Cognitive Science Society, vol. 46 (2024).
33.Niv Y Reinforcement learning in multidimensional environments relies on attention mechanisms J. Neurosci. 2015 35 8145 8157 26019331 10.1523/JNEUROSCI.2978-14.2015 Niv, Y. et al. Reinforcement learning in multidimensional environments relies on attention mechanisms. J. Neurosci.35, 8145–8157 (2015).10.1523/JNEUROSCI.2978-14.2015 PMC4444538 26019331
34.Leong YC Radulescu A Daniel R DeWoskin V Niv Y Dynamic interaction between reinforcement learning and attention in multidimensional environments Neuron 2017 93 451 463 28103483 10.1016/j.neuron.2016.12.040 Leong, Y. C., Radulescu, A., Daniel, R., DeWoskin, V. & Niv, Y. Dynamic interaction between reinforcement learning and attention in multidimensional environments. Neuron93, 451–463 (2017).10.1016/j.neuron.2016.12.040 PMC5287409 28103483
35.Gershman SJ Markman AB Otto AR Retrospective revaluation in sequential decision making: a tale of two systems J. Exp. Psychol.: Gen. 2014 143 182 23230992 10.1037/a0030844 Gershman, S. J., Markman, A. B. & Otto, A. R. Retrospective revaluation in sequential decision making: a tale of two systems. J. Exp. Psychol.: Gen.143, 182 (2014).10.1037/a0030844 23230992
36.Sutton RS Dyna, an integrated architecture for learning, planning, and reacting ACM Sigart Bull. 1991 2 160 163 10.1145/122344.122377 Sutton, R. S. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bull.2, 160–163 (1991).
37.Braem S Chai M Held LK Xu S One cannot simply ’be flexible’: regulating control parameters requires learning Curr. Opin. Behav. Sci. 2024 55 101347 10.1016/j.cobeha.2023.101347 Braem, S., Chai, M., Held, L. K. & Xu, S. One cannot simply ’be flexible’: regulating control parameters requires learning. Curr. Opin. Behav. Sci.55, 101347 (2024).
38.Woodman GF Carlisle NB Reinhart RM Where do we store the memory representations that guide attention? J. Vis. 2013 13 1 1 23444390 10.1167/13.3.1 Woodman, G. F., Carlisle, N. B. & Reinhart, R. M. Where do we store the memory representations that guide attention? J. Vis.13, 1–1 (2013).10.1167/13.3.1 PMC3590103 23444390
39.Wolfensteller U Ruge H On the timescale of stimulus-based action–effect learning Q. J. Exp. Psychol. 2011 64 1273 1289 10.1080/17470218.2010.546417 21416458 Wolfensteller, U. & Ruge, H. On the timescale of stimulus-based action–effect learning. Q. J. Exp. Psychol.64, 1273–1289 (2011).10.1080/17470218.2010.546417 21416458
40.Janczyk M Eichfelder L Liesefeld HR Franz VH Learning and transfer of response–effect relations. Q. J. Exp. Psychol. 2024 79 1445 1462 10.1177/17470218241284259 PMC13168614 39256971 Janczyk, M., Eichfelder, L., Liesefeld, H. R. & Franz, V. H. Learning and transfer of response–effect relations. Q. J. Exp. Psychol.79, 1445–1462 (2024).10.1177/17470218241284259 PMC13168614 39256971
41.Shenhav A The affective gradient hypothesis: an affect-centered account of motivated behavior. Trends Cogn. Sci. 2024 28 1089 1104 39322489 10.1016/j.tics.2024.08.003 Shenhav, A. The affective gradient hypothesis: an affect-centered account of motivated behavior. Trends Cogn. Sci.28, 1089–1104 (2024).10.1016/j.tics.2024.08.003 PMC11620945 39322489
42.Nguyen CT Value capture J. Ethics Soc. Philos. 2024 27 469 10.26556/jesp.v27i3.3048 Nguyen, C. T. Value capture. J. Ethics Soc. Philos.27, 469 (2024).
43.Carroll JD Functional learning: the learning of continuous functional mappings relating stimulus and response continua ETS Res. Bull. Ser. 1963 1963 i 144 Carroll, J. D. Functional learning: the learning of continuous functional mappings relating stimulus and response continua. ETS Res. Bull. Ser.1963, i–144 (1963).
44.DeLosh EL Busemeyer JR McDaniel MA Extrapolation: the sine qua non for abstraction in function learning J. Exp. Psychol.: Learn., Mem., Cognit. 1997 23 968 10.1037//0278-7393.23.4.968 9231439 DeLosh, E. L., Busemeyer, J. R. & McDaniel, M. A. Extrapolation: the sine qua non for abstraction in function learning. J. Exp. Psychol.: Learn., Mem., Cognit.23, 968 (1997).10.1037//0278-7393.23.4.968 9231439
45.Lucas CG Griffiths TL Williams JJ Kalish ML A rational model of function learning Psychon. Bull. Rev. 2015 22 1193 1215 10.3758/s13423-015-0808-5 25732094 Lucas, C. G., Griffiths, T. L., Williams, J. J. & Kalish, M. L. A rational model of function learning. Psychon. Bull. Rev.22, 1193–1215 (2015).10.3758/s13423-015-0808-5 25732094
46.Mcdaniel MA Busemeyer JR The conceptual basis of function learning and extrapolation: comparison of rule-based and associative-based models Psychon. Bull. Rev. 2005 12 24 42 10.3758/BF03196347 15948282 Mcdaniel, M. A. & Busemeyer, J. R. The conceptual basis of function learning and extrapolation: comparison of rule-based and associative-based models. Psychon. Bull. Rev.12, 24–42 (2005).10.3758/bf03196347 15948282
47.Juechems K Summerfield C Where does value come from? Trends Cogn. Sci. 2019 23 836 850 31494042 10.1016/j.tics.2019.07.012 Juechems, K. & Summerfield, C. Where does value come from? Trends Cogn. Sci.23, 836–850 (2019).10.1016/j.tics.2019.07.012 31494042
48.Keramati M Gutkin B Homeostatic reinforcement learning for integrating reward collection and physiological stability Elife 2014 3 e04811 25457346 10.7554/eLife.04811 Keramati, M. & Gutkin, B. Homeostatic reinforcement learning for integrating reward collection and physiological stability. Elife3, e04811 (2014).10.7554/eLife.04811 PMC4270100 25457346
49.Li J-J Collins AG An algorithmic account for how humans efficiently learn, transfer, and compose hierarchically structured decision policies Cognition 2025 254 105967 39368350 10.1016/j.cognition.2024.105967 PMC12052257 Li, J.-J. & Collins, A. G. An algorithmic account for how humans efficiently learn, transfer, and compose hierarchically structured decision policies. Cognition254, 105967 (2025).10.1016/j.cognition.2024.105967 PMC12052257 39368350
50.Vaidya AR Jones HM Castillo J Badre D Neural representation of abstract task structure during generalization ELife 2021 10 e63226 33729156 10.7554/eLife.63226 Vaidya, A. R., Jones, H. M., Castillo, J. & Badre, D. Neural representation of abstract task structure during generalization. ELife10, e63226 (2021).10.7554/eLife.63226 PMC8016482 33729156
51.Molinaro, G. Code and data for “Reward function compression facilitates goal-dependent reinforcement learning”, 10.5281/zenodo.22069290 (2026).PMC13737099 42982847
52.De Leeuw JR jspsych: a JavaScript library for creating behavioral experiments in a web browser Behav. Res. Methods 2015 47 1 12 24683129 10.3758/s13428-014-0458-y De Leeuw, J. R. jspsych: a JavaScript library for creating behavioral experiments in a web browser. Behav. Res. Methods47, 1–12 (2015).10.3758/s13428-014-0458-y 24683129
53.Xia L Modeling changes in probabilistic reinforcement learning during adolescence PLoS Comput. Biol. 2021 17 e1008524 34197447 10.1371/journal.pcbi.1008524 Xia, L. et al. Modeling changes in probabilistic reinforcement learning during adolescence. PLoS Comput. Biol.17, e1008524 (2021).10.1371/journal.pcbi.1008524 PMC8279421 34197447
54.Vallat R Pingouin: statistics in python J. Open Source Softw. 2018 3 1026 10.21105/joss.01026 Vallat, R. Pingouin: statistics in python. J. Open Source Softw.3, 1026 (2018).
55.Jolly E Pymer4: connecting R and Python for linear mixed modeling J. Open Source Softw. 2018 3 862 10.21105/joss.00862 Jolly, E. Pymer4: connecting R and Python for linear mixed modeling. J. Open Source Softw.3, 862 10.21105/joss.00862 (2018).
56.Virtanen P Scipy 1.0: fundamental algorithms for scientific computing in Python Nat. Methods 2020 17 261 272 10.1038/s41592-019-0686-2 32015543 Virtanen, P. et al. Scipy 1.0: fundamental algorithms for scientific computing in Python. Nat. Methods17, 261–272 (2020).10.1038/s41592-019-0686-2 PMC7056644 32015543
57.The pandas development team. pandas-dev/pandas: Pandas v2.2.0. Zenodo 10.5281/zenodo.10537285 (2024).
58.Harris CR Array programming with NumPy Nature 2020 585 357 362 10.1038/s41586-020-2649-2 32939066 Harris, C. R. et al. Array programming with NumPy. Nature585, 357–362 10.1038/s41586-020-2649-2 (2020).PMC7759461 32939066
59.Lee, M. D. & Wagenmakers, E.-J. Bayesian cognitive modeling: a practical course (Cambridge University Press, 2014).
60.Jolly E Pymer4: Connecting R and Python for linear mixed modeling J. Open Source Softw. 2018 3 862 10.21105/joss.00862 Jolly, E. Pymer4: Connecting R and Python for linear mixed modeling. J. Open Source Softw.3, 862 (2018).
61.Lefebvre G Lebreton M Meyniel F Bourgeois-Gironde S Palminteri S Behavioural and neural characterization of optimistic reinforcement learning Nat. Hum. Behav. 2017 1 0067 10.1038/s41562-017-0067 Lefebvre, G., Lebreton, M., Meyniel, F., Bourgeois-Gironde, S. & Palminteri, S. Behavioural and neural characterization of optimistic reinforcement learning. Nat. Hum. Behav.1, 0067 (2017).
62.Westbrook, A. et al. Striatal dopamine can enhance learning, both fast and slow, and also make it cheaper. bioRxiv10.1101/2024.02.14.580392 (2024).
63.Sugawara M Katahira K Dissociation between asymmetric value updating and perseverance in human reinforcement learning Sci. Rep. 2021 11 3574 10.1038/s41598-020-80593-7 33574424 Sugawara, M. & Katahira, K. Dissociation between asymmetric value updating and perseverance in human reinforcement learning. Sci. Rep.11, 3574 (2021).10.1038/s41598-020-80593-7 PMC7878894 33574424
64.Wilson RC Collins AG Ten simple rules for the computational modeling of behavioral data Elife 2019 8 e49547 31769410 10.7554/eLife.49547 Wilson, R. C. & Collins, A. G. Ten simple rules for the computational modeling of behavioral data. Elife8, e49547 (2019).10.7554/eLife.49547 PMC6879303 31769410
65.Palminteri S Wyart V Koechlin E The importance of falsification in computational cognitive modeling Trends Cogn. Sci. 2017 21 425 433 28476348 10.1016/j.tics.2017.03.011 Palminteri, S., Wyart, V. & Koechlin, E. The importance of falsification in computational cognitive modeling. Trends Cogn. Sci.21, 425–433 (2017).10.1016/j.tics.2017.03.011 28476348
66.Brodeur MB Dionne-Dostie E Montreuil T Lepage M The bank of standardized stimuli (boss), a new set of 480 normative photos of objects to be used as visual stimuli in cognitive research PLoS ONE 2010 5 e10773 20532245 10.1371/journal.pone.0010773 Brodeur, M. B., Dionne-Dostie, E., Montreuil, T. & Lepage, M. The bank of standardized stimuli (boss), a new set of 480 normative photos of objects to be used as visual stimuli in cognitive research. PLoS ONE5, e10773 (2010).10.1371/journal.pone.0010773 PMC2879426 20532245
Come citare questo studio
Molinaro G, Collins AGE. (2026). Reward function compression facilitates goal-dependent reinforcement learning.. Nature communications. DOI: 10.1038/s41467-026-78295-1
Consulta la pubblicazione originale ↗