Language-based affect assessments capture experiment-induced changes beyond rating scales.
Behavior research methods · 2026
Abstract originale
Abstract Language-based assessments (LBAs) have shown strong convergent and criterion validity cross-sectionally and longitudinally. However, no study has tested whether LBAs of affect can capture experiment-induced changes (i.e., causal validity ). In this randomized mixed-design experiment, participants ( N = 892) completed a mood induction procedure comprising different conditions ( church, shopping mall, and park ) using physical locations ( N = 153) and online video clips ( N = 739). Participants reported their pre- and post-induction affect through open-ended responses and ratings (i.e., the Positive and Negative Affect Schedule; PANAS). Using large language models and machine learning, we compared how accurately PANAS and language responses classified the experimental conditions. The LBA derived from the language responses exhibited significantly higher accuracy compared to the PANAS scores in classifying the conditions in both the training set (cross-validated AUC = .74 vs. .63), the online holdout set (AUC = .76 vs. .70), and the offline holdout set (AUC = .67 vs. .53). Apart from showing higher predictive accuracy, we assessed LBAs of specific affective dimensions (valence, arousal, and basic emotions). These analyses revealed that positive valence was most strongly expressed in the park condition, while anger was particularly elevated in the mall condition. We also visualized language differences across conditions, showing more calm and peaceful expressions in the park condition and more tired and bored expressions in the mall condition. These findings support the causal validity of language-based affect assessments and their ability to provide insights into affective states. We discuss the potential of using LBA in experimental psychological research.
Parole chiave
Leggi il testo integrale originale
Testo estratto dalla pubblicazione: sono esclusi immagini, media, supplementi e parti soggette a diritti distinti.
Language-based affect assessments capture experiment-induced changes beyond rating scales
Language-based assessments (LBAs) have shown strong convergent and criterion validity cross-sectionally and longitudinally. However, no study has tested whether LBAs of affect can capture experiment-induced changes (i.e., causal validity). In this randomized mixed-design experiment, participants (N = 892) completed a mood induction procedure comprising different conditions (church, shopping mall, and park) using physical locations (N = 153) and online video clips (N = 739). Participants reported their pre- and post-induction affect through open-ended responses and ratings (i.e., the Positive and Negative Affect Schedule; PANAS). Using large language models and machine learning, we compared how accurately PANAS and language responses classified the experimental conditions. The LBA derived from the language responses exhibited significantly higher accuracy compared to the PANAS scores in classifying the conditions in both the training set (cross-validated AUC = .74 vs. .63), the online holdout set (AUC = .76 vs. .70), and the offline holdout set (AUC = .67 vs. .53). Apart from showing higher predictive accuracy, we assessed LBAs of specific affective dimensions (valence, arousal, and basic emotions). These analyses revealed that positive valence was most strongly expressed in the park condition, while anger was particularly elevated in the mall condition. We also visualized language differences across conditions, showing more calm and peaceful expressions in the park condition and more tired and bored expressions in the mall condition. These findings support the causal validity of language-based affect assessments and their ability to provide insights into affective states. We discuss the potential of using LBA in experimental psychological research.
Supplementary Information
The online version contains supplementary material available at https://doi.org/10.3758/s13428-026-03200-z.
Introduction
When people are asked about their emotional state in a natural setting (e.g., How are you feeling?), they typically respond with descriptive language (e.g., I'm feeling pretty good) rather than expressing themselves in numerical values or closed-ended categories (e.g., 7
= Completely Agree; Kjell et al., 2019). Language is a natural and powerful tool for conveying one’s state of mind (Pennebaker et al., 2003). How individuals use words to express themselves and their emotional states provides a valuable source of information for understanding them (Boyd & Schwartz, 2021; Pennebaker, 2011). However, psychological constructs such as affect are often measured using predefined, closed-ended numerical rating scales (Flake et al., 2017). The conventional rating-scale method of assessing subjective states has advanced psychological science in the past decades. They are easy to use, fast, and mostly interpretable. However, they have relatively limited dimensionality (measuring only a few predefined items), range (fixed scale range, e.g., 1–7), and resolution (inability to capture nuances between predefined items and fixed ranges) of the response information as compared to free expressions (Kjell et al., 2024), which often fail to capture transient, moment-to-moment fluctuations in everyday emotional experiences (Sun et al., 2019).
Recent advances in artificial intelligence have improved the ability to measure and quantify linguistic behavior, offering the possibility of numerically capturing emotional states and traits derived from open-ended language in reproducible language models (Althoff et al., 2016; Min et al., 2021; Srivastava et al., 2022). These assessments, referred to as language-based assessments (LBAs), measure mental states or traits by transforming language into numeric representations (word embeddings) used to create assessment scores. In this work, we focus specifically on probed LBAs, which rely on open-ended responses to targeted questions (e.g., Kjell et al., 2019), rather than on unstructured language from sources such as social media (e.g., Schwartz et al., 2013).
Probed LBAs have demonstrated validity by showing strong convergence with corresponding rating scales, i.e., convergent validity (e.g., Gu et al., 2024; Kjell et al., 2022; Nilsson et al., 2024b), and by cross-sectionally and longitudinally predicting theoretically related outcomes more accurately than corresponding ratings, i.e., criterion validity (Gu et al., 2024; Kjell et al., 2021a, b, c; Nilsson et al., 2024a). In addition to convergent and criterion validity, causal validity—the ability to capture change resulting from a change in the to-be-measured construct (Borsboom et al., 2004)—is essential for measurement validation (as further described below). To our knowledge, no research has directly compared the ability of LBAs and rating scales to detect experimentally induced changes in affect.
To address this gap, the present study evaluates the performance of probed LBAs and rating scales in a randomized experimental setting. We assess the ability of these methods to classify participants’ randomly assigned mood induction procedure conditions (church, shopping mall, or park) based on affect descriptions and ratings collected before and after the induction. The randomly assigned condition serves as a ground truth criterion to test whether each method can detect systematic affective changes elicited by the induction. We also show how LBAs can characterize the nature of these changes, using theoretically grounded pretrained models to assess dimensions such as valence, arousal, and basic emotions, along with visualizations of statistically significant words and topics that differentiate conditions.
Conceptualizing affect and emotions
Affect and emotions are complex, related but distinct psychological phenomena theorized through multiple frameworks. Dimensional models describe affect in terms of continuous dimensions such as the Circumplex Model of Affect, which arranges affective states in a two-dimensional space defined by valence (pleasant-unpleasant) and arousal (activation-deactivation; Russell, 1980a, b). In contrast, discrete emotion theories emphasize basic, evolutionarily rooted emotions such as anger, fear, or joy, understood as qualitatively distinct categories (Ekman, 1992). Constructivist models argue that emotional experience is assembled from core affect, conceptual knowledge, and social context (Barrett, 2017). These perspectives underscore the complexity of understanding and measuring affective change. Hence, a comprehensive assessment of affect should account for both the underlying dimensions and the discrete labels individuals use to describe their states. Recognizing these complexities, the present study is conceptually grounded in psychological constructionist models (Barrett, 2017). This framework suggests that language is not merely a post hoc report of a state but an active component in constructing emotional meaning (Hoemann et al., 2025). Affective states are inherently situated, requiring individuals to flexibly adapt their expressions to changing environmental demands and dynamic social contexts (Aldao, 2013; Greenaway et al., 2018). Consequently, we adopt theoretically informed, pretrained LBA models designed to quantify affect from language in terms of dimensional valence and arousal, as well as discrete emotions, while acknowledging that such operationalizations necessarily simplify the rich, socially situated nature of emotional language. Transforming open-ended emotional language into numeric embeddings offers a method for quantifying affect while maintaining its inherent complexity, integrating dimensional, discrete, and constructivist affect perspectives (Hoemann et al., 2024).
Language-based assessments
LBAs can be seen as a broad collection of methods that measure psychological constructs by transforming natural language into numerical representations (e.g., counts or word embeddings), language summaries (e.g., topics), or other language features. The term LBA dates back to 2015, when topics of Facebook status updates were used to assess self-reported personality traits (Park et al., 2015). In the current study, we focus mainly on large language model (LLM)-based methods using probed responses as the language source. Probed LBAs rely on open-ended responses to targeted questions (e.g., “How do you feel right now?”), which allows researchers to capture specific momentary states. Compared to unstructured LBAs based on passively collected or naturally occurring language (e.g., from social media or medical records), probed LBAs allow for more standardized input and experimental control and typically align the language source in time and content with the construct of interest (Gu et al., 2024; Kjell et al., 2024). Recent developments in large language models (LLMs) have driven significant progress in natural language processing (Bommasani, 2021; Devlin et al., 2018; Min et al., 2021; Vaswani et al., 2017). Research suggests that LLM-based LBAs have favorable psychometric properties (Kjell et al., 2024). For example, probed LBAs of depression, life satisfaction, and harmony in life have converged with corresponding rating scales at r =.81,.80, and.85 (Gu et al., 2024; Kjell et al., 2022). Such high levels of convergence approach the theoretical upper limit, since internal reliability sets an upper limit on convergence (Muchinsky, 1996; Spearman, 1961). However, validating LBA against rating scales is an important but limited benchmark since scales do not represent the objective truth of these constructs (Cronbach & Meehl, 1955; Eijsbroek et al., 2025a, b; Krosnick, 1999). Consequently, these studies may overlook effects that rating scales cannot capture.
To validate LBA beyond rating scales, studies have compared it with actual behaviors and other external criteria. For example, researchers have validated unstructured LBAs against external criteria involving demographics and behaviors ranging from daily sleep hours to Facebook behaviors (Eichstaedt et al., 2018; Schwartz et al., 2014; Son et al., 2023). For behaviors, probed LBAs of anxiety and depression have tended to predict sick leave and health care visits more strongly than the corresponding ratings (Gu et al., 2024); probed LBA of ecological momentary assessment-based well-being has predicted alcohol consumption nearly four times more accurately than the corresponding ratings (Nilsson et al., 2024a); and probed LBA of harmony in life has predicted theoretically relevant cooperative behavior from a give-some dilemma game up to four times more accurately than the corresponding ratings (Kjell et al., 2021a, b, c). Thus, LBA not only reaches the theoretical upper limit of accuracy in converging with corresponding ratings, but often surpasses them in criterion validity—especially when predicting behaviors and outcomes.
Causal validity
Causal validity is the ability of a measure to be causally influenced by the targeted construct (Borsboom et al., 2004). Critically, establishing causal validity does not require, and is often hindered by, comparison to a criterion measure. Instead of relying on correlations between measures, which can be misleading if the criterion itself is noisy, the focus is on the causal change in measurement (e.g., experimental manipulation). Showing how a construct affects test-item responses can help distinguish valid measures and their sensitivity. For example, a valid affect measure should be affected by a valid mood induction procedure. In turn, one should observe differences in affect measurement scores between those randomly assigned to different mood induction conditions and, consequently, be able to distinguish the different conditions. While several LBAs have shown convergent validity by predicting the corresponding rating scales (Gu et al., 2024; Kjell et al., 2022; Sametoğlu et al., 2024) as well as criterion validity by predicting theoretically related outcomes both cross-sectionally (Kjell et al., 2021a, b, c; Nilsson et al., 2024b; Schwartz et al., 2014) and longitudinally (Eichstaedt et al., 2018; Son et al., 2023), these forms of validity rely on correlational evidence. This is problematic because correlation alone does not demonstrate measurement validity; for instance, age is highly correlated with height, but age is not a measure of height. Convergent and criterion validity cannot, on their own, establish that an assessment truly measures affect. To address this, we ask whether LBAs, compared to ratings, can capture changes in affect following an experimental manipulation, thereby demonstrating causal validity.
To test the causal validity of LBAs and ratings, this study used a mood induction procedure (Merz et al., 2013) by randomly assigning participants to different locations (church, mall, park). Mood induction procedures are widely used to elicit emotional states (e.g., happiness, sadness, anxiety) and examine their effects on cognitive, emotional, and behavioral processes (Kučera & Haviger, 2012). By intentionally altering mood in a controlled way, mood induction procedures provide a strong test of whether different measures can detect and differentiate these changes, helping to establish a causal link between mood and measured outcomes (Marcusson-Clavertz et al., 2019). This study focuses on short-term effects, with meta-analyses showing mood induction procedures typically produce about a one-standard-deviation change in immediate mood compared to controls (Joseph et al., 2020; Westermann et al., 1996).
The dimensionality of measures
A measure's dimensionality or a model’s dimensionality may affect its ability to capture change when testing causal validity. Dimensionality refers to the number of components or predictors in a measure or model—such as factors in factor analysis, items in a rating scale, or predictors in a regression model. In psychology, when exploring and validating different models and measures of personality (e.g., the Big Five-factor model; Costa & McCrae, 1992), item-level measures have been shown to explain more and generate more accurate predictions than factor-level models (Nielsen & Kajonius, 2024; Seeboth & Mõttus, 2018; Stewart et al., 2022; Vainre et al., 2025), which implies that measures with a higher dimensionality can hold higher predictive power.
In this study, we compare different levels of dimensionality across measures. Using the Positive and Negative Affect Schedule (PANAS) rating scale (Watson et al., 1988), we assess predictive accuracy using either composite scores (lower dimensionality) or individual items (higher dimensionality). For LBAs, we examine both high-dimensional word embeddings (typically 768 or more dimensions) and theoretically grounded, lower-dimensional outputs that capture core affective dimensions such as valence, arousal, and basic emotions. By systematically comparing these levels of dimensionality, we aim to identify which representations best capture participants’ affective states across experimental conditions while also offering theoretical insight.
Language visualizations
Beyond measuring outcomes, LBA can offer deeper insights into how people express mental states by visualizing significant language differences across different traits, states, or behaviors. For example, analyses of social media language have revealed differences in personality and health risks (Schwartz et al., 2013; Eichstaedt et al., 2015). Similarly, affect descriptions can show meaningful distinctions—such as words reflecting socializing and exercise (e.g., friends, running) being linked to higher well-being, while words indicating imbalance (e.g., too, much) relate to lower well-being (Nilsson et al., 2022). Even when rating scales correlate highly, language can reveal distinct profiles, such as harmony in life emphasizing balance and peaceful, versus satisfaction with life highlighting fulfilled and happy (Kjell et al., 2019). These visualizations can help to better understand individuals’ affective states across different experimental conditions and changes over time.
Present study
We evaluate the ability of probed LBAs versus PANAS to capture affective change. Participants’ language responses were used to directly classify experimental conditions (i.e., data-driven classifications) as part of our pre-registered analyses. We hypothesized that LBAs would outperform rating scales in classifying participants’ conditions: specifically, that a model combining affect description embeddings and PANAS scores would outperform PANAS alone (H1), and that affect description embeddings alone would also outperform PANAS (H2). To better understand the nature of affective differences across conditions and explain what drives the data-driven classifications, we conducted further analyses using theoretically grounded assessments of affective dimensions (valence, arousal, and basic emotions) and language-based visualizations through differential language analysis.
Method
Participants
Participants were recruited from Malmö city (Sweden) for the offline sample and from various countries on Prolific for the online sample (Table 1). Participants were restricted to English fluency and using PCs with Google Chrome browsers. Participants who scored positively for symptoms of major depression were screened out in the offline sample (N = 106, 69.3%), and they were automatically screened out at Prolific in the online sample recruitment before they entered our survey; hence, no data were collected on them. The offline sample comprised 157 participants recruited from July 2022 to May 2023. We excluded four participants with incomplete answers. From the remaining sample (N = 153), 127 were collected during 2022 before the pre-registration, and 26 were collected during 2023 after the pre-registration1. We collected the online sample between April and May 2024. We asked participants who did not watch the entire video or fast-forward it to resubmit (N = 39). We collected all online participants after they submitted the pre-registration. Table 1 provides study sample information and participant demographics. All participants in the offline sample were from Sweden. The countries of residence in the online sample included nine countries: Australia, Canada, Ireland, New Zealand, South Africa, Sweden, the UK, and the US. Participants who scored positively for symptoms of major depression were screened out in the offline sample (N = 106, 69.3%), and they were automatically screened out at Prolific in the online sample recruitment before they entered our survey and hence no data were collected on them.
Table 1Study sample information and participant demographics
Sample Offline Online
N total 153 739
N across conditions Church/Mall/Park 45/54/54 254/248/237
Mean age in years (SD; range) 31.6 (12.09; 18–75) 42.8 (13.77; 18–89)
Gender Female/Male/Other 77/72/4 424/308/7
Survey and original response language Swedish English
Median completion time in mins (SD) 17.30 (8.04) 15.61 (8.88)
Religion Non-religious/Christian/Non-Christian religious/Preferred not to say 105/29/6/13 390/264/72/13
Instruments2
Screening The Patient Health Questionnaire PHQ-2 (Löwe et al., 2005) was used to screen out participants. It comprises two items (i.e., “Over the last 2 weeks, how often have you been bothered by any of the following problems? 1. Little interest or pleasure in doing things. 2. Feeling down, depressed, or hopeless”) answered with rating scales ranging from “1 = not at all” to “4 = nearly every day”. Participants with a score of 3 or above were screened out.
The language-based affect question
To assess participants’ current affective state, we used an open-ended question adapted from Kjell et al. (2019):“How do you feel right now? Describe the feelings that you experience right now. Please answer the question by writing 5 descriptive phrases that describe your feelings right now. Write 1 to 10 words for each phrase.”
The open-ended question was translated into Swedish for the offline sample; all the original questions are available in the pre-registration material.
The affect rating scale
Positive and Negative Affect Schedule (PANAS; Watson et al., 1988) is a widely used rating scale designed to measure affect in diverse contexts (Merz et al., 2013); it measures affect using 20 items that relate to various emotional states. Participants were asked to indicate how much they feel each emotional state using a rating scale ranging from “0 = not at all” to “4 = very much”. Half of the items relate to the Positive Affect Scale (e.g., inspired), and the other half capture the Negative Affect Scale (e.g., nervous). Research supports the psychometric utility of the PANAS across different populations and languages, employing various time course instructions to tap into trait-level (e.g., last year, past month) and state-level (e.g., past day, today) affect (Cotigă, 2012; Merz et al., 2013; Terracciano et al., 2003; von Humboldt et al., 2017; Watson & Clark, 1999; Watson et al., 1988). We used the Swedish version of the scale from Bergman & Daukantatie (2009) for the offline sample. Both the online and offline samples showed favorable psychometric properties (see Supplementary Material Table S1).
Intervention
The mood induction procedure involved asking participants to “take in” and be affected by three different environments/locations in Malmö: inside a commercial shopping mall (Triangeln), inside a church (St Johannes Church), or a park (Magistratsparken offline and Pildammsparken online). The instructions for the offline sample were:“First, we want you to sit comfortably.
Take a few deep breaths to land here and now.
See where you are and “take in” the surroundings.
Listen to the sounds around you. Examine whether you can smell any scents.
See what’s around you.
Take the time you need and try to stay in what you are experiencing for a few minutes. When you feel ready, move on by pressing "Next".
In the video instructions, participants were instructed not to fast-forward and to read the instructions carefully:Now, we would like you to watch a video showing a place – we would like you to imagine that you are there. See if you can let your mood be influenced by what you see in the video.
Here's how to make the most of this "journey":
Begin by positioning yourself comfortably in your seat. Make any adjustments needed to ensure you can relax fully without distractions.
Take a few deep breaths to land here and now. Allow yourself to become fully present in this moment.
When the video starts, see where you are and take in the surroundings in the video. Notice the colors, the shapes, and the movements.
Pay attention to the sounds in the video.
When imagining yourself being in the place shown in the video: How does it feel? What emotions does it evoke?
When you are ready, press "Next" to start the video.
All videos were 201 seconds long, and the experiment ensured that the video was automatically downloaded in mp4 format (480p) before the participant could start the study.
Procedure
This study obtained ethical approval from the Swedish Ethical Review Authority (Dnr 2021–06693-01). Potential participants were approached to partake in a study regarding how the physical environment affects one's emotions. To be eligible, participants had to be at least 18 years old. Upon providing consent, participants were screened for depression using the PHQ-2 questionnaire. Only individuals who scored two or below, indicating non-depression, were allowed to proceed with the study.
The procedure (see Fig. 1) began with participants answering demographic questions and other personality measures not used in this study. We randomized the order of presentation for these measures. Next, participants completed the PANAS, the open affect question, and provided a voice recording of their current affective state. The order of these was also randomized.
Following these initial assessments, participants were randomly assigned to one of three conditions: visiting a nearby church, shopping mall, or park. Upon arrival at their assigned location, participants recorded a standardized question before receiving the mood induction procedure. After completing the mood induction procedure at each location, participants answered the same affective state questions and rating scales as before the intervention, in randomized order. The process concluded with participants recording a standardized question outside each location.
Finally, participants were debriefed regarding the aims of the study. Data collection after the pre-registration phase followed the same procedure as before, except the park condition was excluded due to winter weather conditions.
Conditions
The condition locations were chosen based on 1) convenience; all three places were within a short physical distance for the offline conditions, 2) distinctiveness; all conditions were distinct from each other, and 3) impact; the different locations should theoretically impact and elicit differences in affect. For example, in association with the mall, spending money on things and oneself is linked to lower levels of happiness compared to prosocial spending (Dunn et al., 2008, 2014; Kumar et al., 2020). For the park, connecting and interacting with nature is associated with higher levels of well-being and lower levels of stress (Bratman et al., 2012). In relation to the church environment, everyday spiritual experiences have been shown to increase positive affect directly (Whitehead & Bergeman, 2012).
Pre-registration
The data collection and analyses were pre-registered. In the main analyses, we made one extension by including a more recent LLM (the results from the preregistered model are presented in Supplemental Material 1) and excluded one condition (Market) to harmonize the online and offline samples (analyses including the Market condition are presented in the Supplemental Material 1). The offline sample was pre-registered as part of a prior Master’s thesis study3.
Analytical method
The analysis compared the ability of different measures to capture changes in affect. We tested this by examining whether LBAs or rating scales could more accurately classify which experimental condition participants were exposed to during the mood induction procedure. We used one dataset for training (online; 586 participants) and two different holdout sets for evaluation (online and offline; 153 participants each). For text analyses, training sizes have been shown to reach high convergent accuracy after 100 participants, with slight to diminishing increases in accuracy after 300 participants and upwards (Gu et al., 2024). These numbers informed the sample size. We divided the online training data using reproducible randomness, stratified for gender, age, and study condition. We first evaluated different combinations of PANAS and LBA of affect to classify conditions using multinomial logistic regression. We systematically evaluated the role of dimensionality (e.g., is PANAS capturing more relevant information as a composite score or as individual items?). We then evaluated theoretically grounded assessments of valence, arousal, and basic emotions and, finally, visualized the language related to the different conditions and compared the language related to the online and offline data.
Condition-related word removal
According to the pre-registration, direct references to the experimental conditions (i.e., “Mall,” “Church,” and “Park”) were excluded from the post-induction responses used in the classification analyses (although they were retained in the visualizations for interpretative purposes). To further mitigate potential contextual leakage, we implemented additional, more stringent filtering procedures, as detailed in the Supplemental Material 1. These included three approaches: i) automated identification of context-sensitive terms using ChatGPT-4o; ii) augmentation of the automated list through manual review to ensure comprehensive exclusion of context-revealing words; and iii) application of the above filtering steps followed by restriction of the text to only affect-related terms drawn from the Linguistic Inquiry and Word Count (LIWC), including the positive emotion and negative emotion dictionaries (Boyd et al., 2022).
Manipulation check
We evaluated whether the experiment worked as intended by examining whether there was any change in emotion between pre- and post-induction assessment of affect. We tested whether there was a significant difference in rated affect (from PANAS) by conducting a 2 × 3 mixed ANOVA (pre- post intervention scores + Condition + pre- and post scores × Condition). We also performed a manipulation check of change in affect based on the predicted valence (pre- and post-intervention) of the language responses.
Evaluating affect sensitivity of PANAS and LBAs
Evaluating PANAS
To assess its ability to capture changes in affect, we systematically evaluated PANAS’s ability to classify experimental conditions using three methods. The first method combined PANAS scores, combining pre- and post-induction affect scores in four ways. This included using i) post-induction PANAS, ii) the difference between pre- and post-induction PANAS, iii) the concatenation of pre- and post-induction PANAS, and iv) the concatenation of pre-, post-, and the difference between pre- and post-induction PANAS. Previously, the concatenation (i.e., combination/addition of embedding vectors across the dimensional space, such that two vectors of, for example, 768 dimensions turn into one vector of 1536 dimensions) of pre- and post-measures and their difference scores has shown the highest accuracy in predicting mood changes (Ganesan et al., 2022). The second method examined the breadth of emotions, meaning we compared only positive emotion items, only negative emotion items, and both positive and negative emotion items. The third method examined PANAS dimensionality: we tested concatenating PANAS into composite positive and negative emotion scores and using these as predictors, but we also tested individual PANAS items as predictors, resulting in many more predictors and dimensions than using only the composite scores. Combining these three methods produced 28 models, which we evaluated on cross-validated training data and online and offline holdout data (See supplementary materials S4 for an overview of models). All models classified the study conditions by ridge multinomial logistic regression (see section Predictive models).
Evaluating language-based assessments
We systematically evaluated the language data using two methods. The first method matched the PANAS approach, combining the pre- and post-induction measures in four ways. This included using i) post-induction language, ii) the difference between post- and pre-intervention language, iii) the concatenation of pre- and post-induction language, and iv) the concatenation of pre- and post-induction language and their difference (difference embeddings were achieved through vector subtraction, where the numerical representation of the pre-induction response was subtracted from the post-induction response). Second, we compared two LLMs, including the pre-registered RoBERTa Large (Liu et al., 2019) and the newer Mixed Bread Large (Li & Li, 2023). RoBERTa Large has previously demonstrated strong performance among LLMs in social science tasks (Choi et al., 2023; Ganesan et al., 2021; Ziems et al., 2023). However, the newer Mixed Bread Large has shown promising performance (as of March 2024, the model from Li & Li (2023)) ranked among the top of the Massive Text Embedding Benchmark (MTEB) list shown in Enevoldsen et al. (2025). Combining these two methods resulted in eight LBA models. We performed ridge multinomial logistic regressions to classify the study conditions as described above. The predictor variables were the numerical representations of the language (word embeddings).
Natural language processing
Word embeddings
We transformed the language responses into word embeddings, vectors that numerically represent the language's latent meaning. Each language input is represented as a vector in a high-dimensional space across several layers, with the vector’s position reflecting its meaning relative to other words in the corpus (Boyd & Schwartz, 2021). Each layer represents a slightly different contextual meaning. We transform these into word embeddings using large pre-trained language models: RoBERTa Large and Mixed Bread Large (layer 23) were used.
Language translation
For the offline holdout data, which was in Swedish, we used the DeepL API and the deeplr R-package (Zumbach & Bauer, 2021) to translate the affect phrases of the offline sample from Swedish to English. A psychology PhD student, independent of this study and blind to the study objectives and procedures, checked and corrected the translations. The student’s native language was Swedish.
Predictive models
Statistical modeling
Word embeddings can be trained to make LBA models that, in turn, can predict an outcome, such as experiment conditions or semantic characteristics (e.g., the valence of a text; Kjell et al., 2023). To achieve this, the dimensions of the word embeddings were used as the predictors in ridge regression models to classify the experiment conditions. Ridge regression penalizes the coefficients to avoid overfitting (Hoerl & Kennard, 1970). We used a penalty range from 10⁻⁶ to 10⁶, increasing by a factor of 10 at each step (i.e., 10⁻⁶, 10⁻5, …, 10⁶). To evaluate models with different penalties in the training set, we employed nested tenfold cross-validation, where the data was repeatedly divided into ten sets, including nine training sets and one holdout test set. We then randomly split each of the nine training sets into a development set (75% of the data) and a test set (the remaining 25%) to evaluate the developed models. We used the best-performing penalty in the final model (for further explanation, see Kjell et al., 2023). We used multinomial logistic ridge regression to test the classification accuracy of the 28 PANAS-based and eight LBA models, described in the “Evaluating PANAS” and the “Evaluating LBAs” sections above. We also concatenated the predictors of the best-performing PANAS-based and LBA models and tested if this increased the accuracy.
Model performance
We evaluated model performance on the online and offline holdout sets by computing AUC values (area under the ROC curve). This metric measures the overall accuracy of a logistic regression based on the sensitivity (i.e., true-positive rate) and specificity (i.e., true-negative rate). A model with 100% sensitivity and 100% specificity has an AUC of 1. A model with no classification ability has an AUC of.50. For the multinomial models, we used multinomial AUC as evaluation, which is the average one-vs.-all AUC of each pairwise comparison. To test whether one model performed significantly more accurately at categorizing the conditions than another, we calculated the difference between resampled AUC scores from model 1 and model 2, then tested whether this difference was significantly different from zero.
Theoretically grounded language-based assessments
To assess theoretically grounded dimensions of affect and emotions, we created and used pre-trained LBA models to assess valence (i.e., pleasant/unpleasant), arousal (i.e., calm/exciting), and basic emotions (i.e., anger, disgust, fear, joy, sadness, and surprise) of the participant's responses. We evaluated the theoretically grounded LBAs using multinomial ridge regression, as explained above in the predictive models section. All pre-trained LBA models are based on Mixed Bread layer 23 embeddings (Li & Li, 2023).
Language assessed for affect dimensions
The valence and arousal LBA models were pre-trained using a dataset including words with valence and arousal annotations (i.e., ratings on a scale from 1 to 9; Warriner et al., 2013). Valence was rated based on the stimulus pleasantness, and arousal based on stimulus intensity. Each annotator rated the words solely on one of the two dimensions. There were 1827 annotators, and all words had at least 15 ratings (87% had 18 to 30 annotations (ratings) per word). The models are trained on the average ratings of each word. The pre-trained valence model exhibited a tenfold cross-validated Pearson’s r =.83, and the pre-trained arousal model exhibited a tenfold cross-validated Pearson’s r =.62.
Language assessed for basic emotions
The basic emotion LBA models were pre-trained using a dataset in which a target word was presented with two random words and a synonym; first, the annotator had to select the correct synonym, and then annotate the target word for eight discrete emotions: anger, fear, anticipation, trust, surprise, sadness, joy, and disgust (Mohammad & Turney, 2013). Annotators used a four-point scale (“not”, “weakly”, “moderately”, “strongly”). “not” and “weakly” were interpreted as non-emotive regarding the target emotion, while “moderately” and “strongly” were categorized as emotive to the target emotion. A total of 38,726 assignments were made by 2216 annotators. In the analysis, we embedded the target words with their synonyms, predicting the different categories. The resulting LBA models demonstrated strong discriminative performance, with area under the ROC curve (AUC) values for classifying each target emotion as a multinomial model (AUC =.83).
Language visualizations
After developing the assessment models, we visualized the language significantly associated with the different conditions. We also created visualizations to understand differences in language responses between the online and offline samples. We used three techniques for making the visualizations.
Word comparison visualizations
First, word plots based on Supervised Dimension Projection were used for word comparison visualizations (Kjell et al., 2023). In short, supervised dimension projection plots compare responses from two groups (e.g., church versus remaining conditions), showing words significantly related to one group based on their position in the word embedding space. Since RoBERTa Large’s skewed distributions make it inappropriate for geometric plots (Reif et al., 2019), we used the Mixed Bread embeddings for these figures. The computations involve forming an aggregated direction embedding (i.e., a comparative line on a linear scale through the embedding space). Each unique word is then projected onto this line by calculating the dot product between positioned embeddings of all individual words and the aggregated direction embeddings (i.e., supervised dimension projections). We used a permutation method to compute p values for each dot product for each static word embedding (here, we made 500 permutations). We applied false-discovery rate corrections for multiple comparisons (for more information, see Kjell et al., 2023).
To analyze language related to one condition, we grouped that condition as the first group and the remaining two conditions as the second group. After repeating this for all conditions, we visualized only the language positively associated with each condition. We also conducted additional linguistic inquiry and word count (LIWC) comparisons, available in the Supplemental Material (Table S12).
Topic modeling
The second visualization technique was topic modeling. Topics are clusters of words that often occur in similar contexts. We used the algorithm latent Dirichlet allocation (LDA; Blei et al., 2003) to create the topics, a probabilistic model that uncovers latent topics within text collections. We used both the online and offline data to create the topics. We allowed for both 1-grams and 2-grams (1-grams are single words and 2-grams are two words occurring together, e.g., “not_happy”). We excluded all words that appeared less than three times, created 25 topics following the rule of thumb to have 50 documents per topic (we had 1153 responses; Eichstaedt et al., 2021), and iterated the LDA algorithm 1000 times. For these first two techniques, we excluded stop words defined by the snowball lexicon (Benoit et al., 2018), as these words do not carry much meaningful information without any context, and LDA handles very common words poorly (Eichstaedt et al., 2021). We then compared the topics in each condition similarly to how we did in the supervised projection plot, where we grouped one condition as the first group and the remaining two conditions as the second group.
Response probability
The third and final technique involved showing the responses with the highest probability of being in each condition, as identified in our multinomial logistic regressions on the online holdout. In the multinomial logistic regressions, we assigned each response out-of-sample probabilities for each of the three conditions. We simply selected those responses with the highest probabilities of being in each condition. We transformed the probabilities into log odds, which range from – 4.6 to 4.6 when the probability range is 1 to 99% but can theoretically reach infinity. A log-odds of 0 means the response has a 50% probability of being from a certain condition. We also used this for the logistic regression models to estimate the probability that responses came from the online versus offline sample.
Statistical tools
The analyses were computed in R (v4.2.2; R Core Team, 2022) using RStudio (v2023.3.0.386; Posit team, 2023). The following R-packages were used: car (Fox & Weisberg, 2019), caret (Kuhn et al., 2020), corrplot (Wei & Simko, 2021), furrr (Elsey, 2021) ggcorrplot (Kassambara, 2022), ggpubr (Kassambara, 2023a, b), GPArotation (Bernaards & Jennerich, 2005), magrittr (Bache et al., 2022), psych (Revelle, 2023), purrr (Mailund, 2022), readxl (Wickham & Bryan, 2019; Wickham et al., 2019), rstatix (Kassambara, 2023a, b), text (v1.4, Kjell et al., 2021a, b, c), tidyverse (Wickham & Bryan, 2019; Wickham et al., 2019), topics (Ackermann et al., 2024; Eijsbroek et al., 2025a, b).
Results
Randomization and manipulation checks
To examine the experiment's internal validity, we conducted a randomization and manipulation check. The randomization check assessed whether pre-intervention affect measures alone could classify the experimental conditions. As expected, the models performed poorly using both PANAS ratings and language-assessed valence (AUCs <.57), indicating negligible baseline differences between groups (see Supplemental Material Table S2). Further, the manipulation check confirmed that the experimental conditions affected participants’ affect, as shown by significant time × condition interaction effects for PANAS ratings (F(2,739) = 29.84, p <.001, generalized eta2 =.01) and language-assessed valence (F(2,739) = 30.97, p <.001, generalized eta2 =.02). A full correlation table between post-intervention measures can be found in Supplemental Material Table S3.
Data-driven classification models capture changes caused by the mood induction
The LBA of affect significantly outperformed PANAS in capturing changes in affect caused by the experimental manipulation (Table 2; Fig. 2). We systematically evaluated 28 PANAS-based models and 8 LBA models using multinomial ridge regression (see Supplemental Material Tables S4–S5). The best-performing LBA model used high-dimensional language embeddings from the Mixed Bread Large LLM, combining representations from participants’ pre-, post-, and difference responses. This model achieved an AUC of.74 [CI =.69 to.80] on the online holdout and.67 [CI =.60 to.73] on the offline holdout. In comparison, the best-performing PANAS model, which concatenated all 20 pre-, 20 post-induction items, and the difference score between the two items (60 dimensions), reached an AUC of.72 [CI =.65 to.78] on the online holdout but failed to generalize to the offline sample (AUC =.39, [CI =.33 to.46]). These findings indicate that LBA of affect shows higher causal validity than affect ratings. Furthermore, robustness checks, i.e., removing contextual words from the data, only marginally decreased the classification accuracy of the LBA (down to.65, [CI =.58 to.71]; see Tables S6–7 and Appendix A).
Table 2Evaluating the best-performing models combining language and PANAS for classifying conditions
Dimensions AUC
CV Holdout
Online Offline
Best-performing models based on performance in the online holdout set:
1. Best PANAS model1 60 .65*** .72*** .39**
2. Best embedding model2 3072 .73***+++ .74*** .67***++
3. Embeddings + PANAS 3112 .75***+++ .75*** .67***+++
Best-performing models based on performance in the offline holdout set:
1. Best PANAS model3 1 .53* .58** .53ns
2. Best embedding model4 3072 .68***+++ .72***+++ .69***+++
3. Embedding + PANAS 3073 .67***+++ .72***+++ .69***+++
CV = cross-validated, AUC = area under the ROC curve. PANAS = Positive and Negative Affect Schedule. Ntraining = 586; Nonline and offline holdout = 153
1 = Individual positive and negative affect ratings from pre- and post-induction, and difference score
2 = Pre-, post-, and difference embeddings from Mixed Bread
3 = Composite of negative affect ratings from post-induction
4 = Pre-, post-, and difference embeddings from RoBERTa
*= p <.05, **
= p <.01, *** = p <.001 indicating whether AUC significantly differs from.50. ns = not significant; + indicates whether models in row 1 significantly differ from models' performance in other rows using bootstrapping sampling corrected for multiple comparisons using false discovery rate. +++
= p <.001, ++
= p <.01
When models were evaluated on the offline holdout dataset, the best PANAS model (a single post-induction negative affect composite score) achieved a significant AUC of.53 (p >.05, [CI =.47 to 60]). In contrast, the equivalent LBA model achieved an AUC of.69 (CI =.62 to.75)—highlighting its robustness across settings. Further, combining PANAS ratings with affect description embeddings did not significantly improve model performance, suggesting that the LBA models captured almost all of the variance that ratings did.
Theoretically grounded models capture changes in affect caused by the experimental conditions
Pre-trained LBAs of valence provided a theoretically grounded measurement that significantly outperformed PANAS ratings in distinguishing experimental conditions based on post-induction affect descriptions (Table 3; Fig. 2). The bar plots show that language-assessed valence produced substantially larger between-condition differences (F = 34.65, generalized eta2 =.086) compared to PANAS ratings (F = 9.37, generalized eta2 =.025). Effect size estimates (Cohen’s d for LBA comparisons =.34 [CI =.24 to.61],.75 [CI =.55 to.96], and.43 [CI =.16 to.53] versus Cohen’s d for PANAS comparisons.27 [CI =.10 to.45],.39 [CI =.22 to.57] and.11 [CI =.06 to.29]) further demonstrated stronger contrasts among conditions for language-assessed valence, particularly in differentiating the park and mall conditions (Cohen’s d =.75 versus.39). The density plots also illustrate clearer separation of distributions by condition in the LBAs of valence compared to PANAS (Fig. 2b).
Table 3Evaluation of pre-trained LBAs and PANAS on classifying conditions based on post-induction affect descriptions
Assessment Classifying conditions (church, mall, park) AUC
Online Offline
(N = 153) (N = 153)
PANAS1 .59* .52ns
Language assessed for affect dimensions
Valence .68***+ .63***++
Arousal .61* .58ns
Language assessed for basic emotions
Anger .66***+ .64*++
Anticipate .50ns .53ns
Disgust .59* .58ns
Fear .55ns .62*++
Joy .61* .62**+
Sadness .56ns .58**
Surprise .55ns .54ns
Trust .65*** .63**+
The pre-trained large language model is Mixed Bread. Best valence and arousal model: Valence and arousal trained from Warriner et al. (2013; for other valence and arousal models see Table S8). More model assessments are in Supplemental Material Table S7. The basic emotions model is trained based on Mohammad & Turney (2013). No removal of any words. ns = not significant, *
= p <.05, **
= p <.01, *** = p <.001 indicating whether AUC significantly differs from.50. +++
=p <.001, ++
=p <.01, +
=p <.05 compared to the PANAS model
1 = Post-intervention PANAS total score
Table 3 provides further detail on these theoretically informed assessments. In addition to valence, the discrete emotion anger showed significantly higher AUC compared to PANAS for both the online and offline holdout sets (p <.05, AUC.66 online to.64 offline). LBAs of arousal and the discrete emotions of disgust, joy, and trust also showed significant discrimination of experimental conditions in both the training and online holdout sets (p <.05; AUCs =.59 to.65). While their AUCs were consistently on par with or above those of PANAS (which had AUC =.59), these differences were not statistically significant (p >.05). In the offline sample, the emotions fear, joy, sadness and trust showed significant AUCs (p <.05; AUCs =.58 to.63) while the PANAS composite score did not reach significance. More visualizations of the comparisons together with confusion matrices are available in the Supplemental Material (Figs. S1–S3, Table S13).
Overall, these results demonstrate that theoretically grounded LBAs possess at least as strong causal validity compared to traditional rating scales. These findings highlight the potential value of pretrained, theory-informed LBA models for capturing subtle, contextually induced changes in affective states that may be less accessible to closed-ended rating scales.
Language responses reveal condition-specific affective profiles
Participants’ post-induction affect descriptions showed distinct, condition-specific affective patterns, demonstrating that open-ended responses can capture nuanced emotional shifts aligned with the experimental manipulation. Figure 3 visualizes these significant, differentiated words and topics using three complementary techniques, which we interpreted through the Circumplex Model of Affect (Russell, 1980a, b). In panel A, words significantly associated with each condition in the online data (compared to the other conditions) are displayed. The mall condition was characterized by both low- and high-arousal negative affect (e.g., bored, indifferent, annoyed, irritated), physical needs (e.g., hungry, thirsty), and high-arousal positive affect (e.g., excited, alert). In contrast, the park and church conditions predominantly featured low-arousal positive affect words (e.g., serene, calm, peaceful). The park condition included more low-arousal positive descriptors overall, while the church condition also included some negative affect terms (e.g., stress, nervous).
Panel B shows the topics significantly associated with each condition. The topics showed a similar pattern to the significant words, with negative affect (e.g., tired, slightly bored, and little stressed topics) in the mall conditions, as well as high-arousal positive affect (e.g., excited, interested, and alert topics). Again, all topics significantly related to the park condition involved low-arousal positive affect (e.g., calm, relaxed, and peaceful topics). For the church condition, one topic involved positive affect (calm, focused, alert), and one was a church topic. Overall, for both panels A and B, more language was associated with the mall condition, followed by the park, and lastly, the church.
Lastly, panel C shows the responses that the best-performing multinomial logistic regressions identified as having the highest probabilities for being in each condition. They generally supported the previous results, with mall responses including low- and high-arousal negative affect (e.g., Bored, Tired, Exasperated, Annoyed, and Disappointed) and high-arousal positive affect (e.g., Enthusiastic, Interested, Determined, Happy, and Content). The park responses involved mostly low-arousal positive affect, with all three responses starting with “Calm”. The church responses showed a tendency toward broad reflection (e.g., “Thinking of those no longer with us” and “A bit more detached from worldly problems”). Overall, the mall language involved low- and high-arousal negative emotions and high-arousal positive emotions, and these signals were stronger than the signals significantly related to the park and church conditions (i.e., the mall language generated more significant words and topics). Both the park and church language involved low-arousal positive emotions, with a bit more of this signal in the park condition. The church condition showed tendencies to invoke deeper reflections about life.
The difference between online and offline language
Given the relatively poor generalization from the online-trained models to the offline holdout—observed for both LBA and rating-based models—we investigated how the language differed between the two samples. As shown in Fig. 4, the language patterns between online and offline participants were clearly distinguishable, with a classification model accurately separating them (AUC =.79, p <.001). Offline data featured words and topics related to immediate, visceral, or physical states such as “stressed”, “irritated”, and “hungry”, which often reflect real-world pressures or challenges. In contrast, online data predominantly featured words and topics associated with reflective, affective states, including “peaceful”, “feeling”, and “beauty”. This divergence suggests that visiting the offline locations generated language tied to situational stressors and physical experiences, while the online contexts induced more abstract, idealized, or affective states. The distinction was further supported by the specific responses identified as having the highest probability of being in each setting, where one of the offline responses described specific stress-inducing scenarios (“Stressed out have to take the train”), whereas online responses focused on inner calm and readiness, aligning with the tendency of digital platforms to foster reflective and curated expressions (“I feel content”). These patterns underscore the importance of context in interpreting linguistic data from different sources and its implications for understanding affective and cognitive states. They also highlight the limitations of the generalizability of mood induction procedures across online and offline settings.
Discussion
This study evaluated the causal validity of language-based assessments of affect compared with a traditional rating scale, using a randomized experimental mood induction procedure. In line with our pre-registered hypotheses, data-driven classifications using LBAs more accurately identified participants’ experimental conditions than the rating scale. Beyond superior predictive performance, theoretically grounded pre-trained LBAs of affect dimensions and emotions, along with data-driven language visualizations, provided interpretable insights into how descriptions of individuals’ affective experiences varied across conditions and platforms (online vs. offline).
While prior research has demonstrated strong convergence between LBAs and rating scales, as well as robust criterion validity in predicting external behaviors and psychological outcomes (Eichstaedt et al., 2018; Gu et al., 2024; Kjell et al., 2019; Nilsson et al., 2024a; Schwartz et al., 2014; Son et al., 2023), this study is—to our knowledge—the first to provide supporting evidence for the causal validity of LBAs.
Data-driven language-based classifications
The higher classification accuracy of LBAs supports their ability to sensitively capture condition-induced change in affect beyond the corresponding rating. Our systematic evaluation showed that even when combined with PANAS, the best LBA models experienced either negligible improvement or only minor gains in AUC (e.g., an increase of.01), indicating that the linguistic embeddings already encapsulated nearly all the predictive signal from the rating data. However, instead of producing a single interpretable scale score (e.g., level of positive affect), these data-driven classification models generate probabilistic scores reflecting the likelihood that a person participated in a specific experimental condition (e.g., park vs. mall). While inherently context-dependent and not directly comparable to continuous affect scores, the classification metric (AUC) provides an interpretable effect size for how well language-based affect descriptions distinguish between conditions. To move beyond raw accuracy and illuminate the specific emotional content underlying these differences, we applied theoretically grounded, pre-trained LBA models of affect dimensions and basic emotions as well as data-driven language-based visualizations.
Theoretically grounded language-based assessments
Whereas the data-driven classification models operate without predefined emotional frameworks, the theoretically grounded, pre-trained LBAs were explicitly informed by psychological models of affect and emotion. By quantifying emotional content along established dimensions—such as valence and arousal from dimensional theories (Russell, 1980a, b), and discrete categories like joy, anger, and trust from basic emotion theories (Ekman, 1992)—these models directly reflect theoretical constructs from the science of affect and emotion. For example, participants in the park condition expressed significantly higher positive valence than those in the mall condition—aligning with prior research showing that natural environments reliably evoke more positive emotions than urban settings (e.g., Bratman et al., 2012; Kuo, 2015). Notably, language-assessed valence and the discrete emotion anger revealed larger differences between experimental conditions than PANAS ratings.
Data-driven language-based visualizations
The data-driven language-based visualizations also provided insights that go beyond raw classification accuracy. Unlike the theoretically grounded pretrained models, these visualizations are not constrained by predefined emotion categories but instead reveal significant, condition-specific patterns emerging from participants’ affect descriptions. The results are largely consistent with the patterns observed in the theoretically grounded valence analyses: For example, language in the park and church conditions reflected higher levels of low-arousal positive emotions, lower levels of high-arousal positive emotions, and less negative emotion, suggesting a calming influence of these settings as evident from the significant words and topics identified. Moreover, our observation that significantly “more language” (a higher volume of unique words and topics) was associated with the mall, we argue, highlights the complex, mixed affective states that can be triggered by a modern consumer environment. The mall environment simultaneously evoked high-arousal negative affect (irritation), low-arousal negative affect (boredom), and visceral physiological needs (hunger/thirst). Open-ended language captures this multidirectional emotional friction, showing that real-world affect is rarely univalent or isolated from immediate surroundings.
The data-driven visualizations helped explain differences in generalizability between the online and offline samples. The offline participants used more contextually grounded language, linking emotions to specific real-world states such as “Stressed to take the train”, while the online sample was overall more reflective in its emotional language, such as “I am feeling blessed”.
Similarly, the finding that the church condition invoked distinct “deeper reflections about life” (e.g., thoughts of missing loved ones or feeling detached from worldly problems) offers more conceptual clarity. This demonstrates that language does not merely report an internal state; it actively shapes and categorizes it. In this case, the church did not just induce a passive, low-arousal positive state (such as simple relaxation), but actively directed participants toward an existential, cognitive frame of reference. These insights underscore the utility of data-driven language analyses not only for measuring affective change but also for explaining and contextualizing those changes in interpretable ways.
Causal validity: Affect, context, and dimensionality
Taken together, these findings support the causal validity of LBAs of affect, demonstrating their capacity to detect affective changes resulting from a randomized experimental manipulation. While one could argue that the superior classification performance of LBAs might be driven by contextual cues (e.g., “Hungry from looking at a food court”) rather than genuine affective states (e.g., “Enthusiastic, Interested, Determined, Happy, Content”), several lines of evidence counter this concern. First, stricter filtering procedures that removed contextual references led to only marginal reductions in classification accuracy (see Supplemental Material Table 7), suggesting that affective information—not just situational content—drives model performance. Second, the theoretically grounded LBAs rely not on context-specific words, but on affective dimensions (valence and arousal) and probabilities of discrete emotions derived from established language models. Third, data-driven language visualizations revealed psychologically meaningful differences in emotional tone across conditions—such as calmness in park settings and agitation in malls—that extended beyond surface-level context.
Importantly, we argue that emotional experiences are rarely detached from context. People naturally embed affect in real-world narratives (e.g., “At one with nature”), linking feelings to environments, events, and social settings (Barrett, 2017; Kals et al., 1999; Kjell, 2011). From this perspective, contextual language is not semantic noise but a source of ecological validity, offering insight into how affect is genuinely experienced and expressed. A notable finding was the reduced performance between the online and offline holdout, particularly for the PANAS model. The discrepancy likely reflects the effect of surrounding real-world context, highlighted by the different language patterns: more direct, physical emotional language in the offline sample versus more inner, reflective emotional language in the online sample. On a closed-ended rating scale, these real-world factors may introduce noise (e.g., travel stress), compressing response variance and obscuring the subtle signal of the experimental induction. LBAs, by contrast, remained more robust, arguably because the open-ended format allowed participants to express this emotional context. However, although the results give some preliminary hints, we do not know to what degree emotions are embedded in the local context, which future research could examine.
Participants describe their emotional states freely, which may allow the method to capture nuances missed by fixed-item scales. The open-ended responses are then transformed into high-dimensional embeddings that capture affect across thousands of word embedding dimensions. This expressive and computational flexibility may allow LBA to more accurately reflect the complexity and personal relevance of real-world emotional experiences (Boyd & Schwartz, 2021; Kjell et al., 2024). Notably, increasing dimensionality improved rating accuracy (using PANAS as 20 single items), indicating that higher dimensionality generally increased accuracy. However, the 1068 dimensions of LBAs used in this study can rarely be translated into fixed-item scales, which often use only a few items for a single psychological construct.
Limitations and future direction
The present study has some limitations that should be considered when interpreting the findings. First, the data are linked to two separate pre-registrations. The first pertains to a Master’s thesis and the offline sample, which was not followed in this paper (the pre-registration and thesis are available at https://osf.io/97cuj). The second pre-registration was specific to this study and guided the online data collection. We made two deviations: First, because computer science has advanced rapidly, we adjusted the models included to capture the latest developments. The pre-registered models are still part of the Supplemental Material in Table S5, further providing support for our hypotheses. Second, one condition (a video of a market) was excluded from the online sample to harmonize the offline and online data. The Supplemental Material (Table S9) includes additional analyses using all four conditions, further supporting the superiority of the accuracy of LBAs over rating scales. Further pairwise comparisons for LBAs and PANAS can also be found in Supplemental Material (Tables S10–11).
Second, the current study aims for high ecological validity by examining mood inductions through experiences of common urban environments, where the environments for the conditions were chosen partly based on the physical closeness to each other. Rather than explicitly targeting positive or negative emotions, as the PANAS is designed to capture, the present experiment focuses on more naturalistic, context-driven mood shifts. To more directly assess the sensitivity of PANAS and comparable rating scales, future studies should aim to induce specific emotional states—such as positive affect—using different targeted environments, and evaluate how well models perform relative to traditional scales under those conditions.
Third, while the pre-trained language-based models of valence, arousal, and basic emotions in this study offer theoretically grounded assessments of affect, their construct validity warrants continued evaluation. Although our findings show alignment with experimental manipulations and prior theory, further validation against behavioral outcomes, physiological measures, and longitudinal data would strengthen confidence in their broader applicability. We encourage future studies to benchmark these models against multimodal indicators of affect to clarify their psychometric properties. Importantly, all models and code used in this study are openly available through the text package (Kjell et al., 2023) and the language-based assessment model (L-BAM) library (Nilsson et al., 2026), enabling transparent application and replication by other researchers.
Fourth, the reliance on English-language embeddings and the translation of the Swedish offline sample represent a potential ‘cultural bottleneck’. Emotional language is deeply embedded in cultural and contextual frameworks (Hoemann, 2023); therefore, the affective language captured by our models may not be universally representative of all linguistic groups (Blasi et al., 2022). While independent researchers verified our translation process, the English-centric nature of many state-of-the-art LLMs remains a ‘bottleneck’ for the field. Future research should use multilingual LLMs and culturally broader samples to test whether these affective linguistic signatures are invariant across cultural contexts.
We acknowledge that the current results suggest LBAs are currently better suited for research and group-level analysis rather than individual clinical diagnostics. However, this performance should be viewed as a promising baseline for future models that may incorporate even larger training datasets and more nuanced and diverse contexts. One promising area is ecological momentary assessment, where LBAs can capture subtle, in-the-moment affective shifts from open-ended responses in naturalistic settings, helping bridge the gap between traditional laboratory measures and everyday emotional regulation within its natural habitat (Kaplan et al., 2022). Furthermore, recent technological innovations provide secure mobile frameworks to collect spoken or written responses longitudinally, yielding richer verbal data and stronger psychometric signals than traditional text-entry surveys (Kaplan et al., 2025). Another is intervention research—particularly in the domain of well-being and happiness. Although interventions such as exercise, meditation, or time in nature are widely promoted, replication failures are common (Folk & Dunn, 2023, 2024), raising questions about their effectiveness. One possible reason is that traditional rating scales may overlook the diverse ways people experience and express positive affect. Because LBAs can detect nuanced and contextually grounded emotional expressions, they could enhance the evaluation of happiness interventions by revealing how and why affective states change—or fail to change—across individuals and contexts. Future studies could combine LBAs with rating scales to offer a more comprehensive and ecologically valid picture of affective change.
Conclusion
This study provides evidence for the causal validity of language-based assessments in detecting changes in affect and emotion resulting from experimental manipulation—outperforming the traditional affect rating scale PANAS. By leveraging open-ended language, LBAs can capture condition-specific differences in affective expression within the present experimental context, offering richer descriptive information than the rating scale used here. Unlike fixed-item questionnaires, they capture both subtle emotional shifts and the language individuals use to make sense of their experiences. The cross-platform comparison of LBA enables understanding of the biases and reasons for changes in performance across data collection contexts, otherwise masked with rating comparisons. These findings strengthen the scientific foundation of LBAs as valid tools for assessing affective change and highlight their potential for broader application in experimental research where understanding the richness of emotional experience is essential.
Supplementary Information
Below is the link to the electronic supplementary material.
References
Ackermann, L., Gu, Z., & Kjell, O. (2024). “An R-package for visualizing text in topics [Computer software]. Zenodo. 10.5281/zenodo.11165378
Aldao A The future of emotion regulation research: Capturing context Perspectives on Psychological Science 2013 8 2 155 172 10.1177/1745691612459518 26172497 Aldao, A. (2013). The future of emotion regulation research: Capturing context. Perspectives on Psychological Science,8(2), 155–172. 10.1177/1745691612459518 26172497
Althoff T Clark K Leskovec J Large-scale analysis of counseling conversations: An application of natural language processing to mental health Transactions of the Association for Computational Linguistics 2016 4 463 476 28344978 10.1162/tacl_a_00111 Althoff, T., Clark, K., & Leskovec, J. (2016). Large-scale analysis of counseling conversations: An application of natural language processing to mental health. Transactions of the Association for Computational Linguistics,4, 463–476.PMC5361062 28344978
Bache, S. M., Wickham, H., & Henry, M. L. (2022). magrittr: A forward-pipe operator for R (Version 2.0.3) [R package]. CRAN. https://cran.r-project.org/package=magrittr
Barrett LF How emotions are made: The secret life of the brain 2017 Houghton Mifflin HarcourtBarrett, L. F. (2017). How emotions are made: The secret life of the brain. Houghton Mifflin Harcourt.
Benoit K Watanabe K Wang H Nulty P Obeng A Müller S Matsuo A Quanteda: An R package for the quantitative analysis of textual data Journal of Open Source Software 2018 3 30 774 774 10.21105/joss.00774 Benoit, K., Watanabe, K., Wang, H., Nulty, P., Obeng, A., Müller, S., & Matsuo, A. (2018). Quanteda: An R package for the quantitative analysis of textual data. Journal of Open Source Software,3(30), 774–774.
Bergman LR Daukantaité D Stability of typical patterns of subjective well-being in middle-aged Swedish women Journal of Happiness Studies 2009 10 3 293 311 10.1007/s10902-007-9081-2 Bergman, L. R., & Daukantaité, D. (2009). Stability of typical patterns of subjective well-being in middle-aged Swedish women. Journal of Happiness Studies,10(3), 293–311. 10.1007/s10902-007-9081-2
Bernaards C Jennrich RI Gradient projection algorithms and software for arbitrary rotation criteria in factor analysis Educational and Psychological Measurement 2005 65 5 676 696 10.1177/0013164404272507 Bernaards, C., & Jennrich, R. I. (2005). Gradient projection algorithms and software for arbitrary rotation criteria in factor analysis. Educational and Psychological Measurement,65(5), 676–696. 10.1177/0013164404272507
Blasi DE Henrich J Adamou E Kemmerer D Majid A Over-reliance on English hinders cognitive science Trends in Cognitive Sciences 2022 10.1016/j.tics.2022.09.015 36253221 Blasi, D. E., Henrich, J., Adamou, E., Kemmerer, D., & Majid, A. (2022). Over-reliance on English hinders cognitive science. Trends in Cognitive Sciences. 10.1016/j.tics.2022.09.015 36253221
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.
Bommasani, R. (2021). On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258
Borsboom D Mellenbergh GJ Van Heerden J The concept of validity Psychological Review 2004 111 4 1061 15482073 10.1037/0033-295X.111.4.1061 Borsboom, D., Mellenbergh, G. J., & Van Heerden, J. (2004). The concept of validity. Psychological Review,111(4), 1061.10.1037/0033-295X.111.4.1061 15482073
Boyd RL Schwartz HA Natural language analysis and the psychology of verbal behavior: The past, present, and future states of the field Journal of Language and Social Psychology 2021 40 1 21 41 10.1177/0261927x20967028 34413563 PMC8373026 Boyd, R. L., & Schwartz, H. A. (2021). Natural language analysis and the psychology of verbal behavior: The past, present, and future states of the field. Journal of Language and Social Psychology,40(1), 21–41. 10.1177/0261927x20967028 PMC8373026 34413563
Boyd RL Ashokkumar A Seraj S Pennebaker JW The development and psychometric properties of LIWC-22 Austin, TX: University of Texas at Austin 2022 10 1 47 Boyd, R. L., Ashokkumar, A., Seraj, S., & Pennebaker, J. W. (2022). The development and psychometric properties of LIWC-22. Austin, TX: University of Texas at Austin,10, 1–47.
Bratman GN Hamilton JP Daily GC The impacts of nature experience on human cognitive function and mental health Annals of the New York Academy of Sciences 2012 1249 118 36 22320203 10.1111/j.1749-6632.2011.06400.x Bratman, G. N., Hamilton, J. P., & Daily, G. C. (2012). The impacts of nature experience on human cognitive function and mental health. Annals of the New York Academy of Sciences,1249, 118–36.10.1111/j.1749-6632.2011.06400.x 22320203
Choi M Pei J Kumar S Shu C Jurgens D Bouamor Houda Pino Juan Bali Kalika Do LLMs understand social knowledge? Evaluating the sociability of large language models with socKET benchmark Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing 2023 Association for Computational Linguistics11370 11403 10.18653/v1/2023.emnlp-main.699 Choi, M., Pei, J., Kumar, S., Shu, C., & Jurgens, D. (2023). Do LLMs understand social knowledge? Evaluating the sociability of large language models with socKET benchmark. In Houda Bouamor, Juan Pino, & Kalika Bali (Eds.), Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing (pp. 11370–11403). Association for Computational Linguistics.
Costa PT Jr.McCrae RR Revised NEO personality inventory (NEO-PI-R) and NEO five-factor inventory (NEO-FFI) professional manual 1992 Psychological Assessment ResourcesCosta, P. T., Jr., & McCrae, R. R. (1992). Revised NEO personality inventory (NEO-PI-R) and NEO five-factor inventory (NEO-FFI) professional manual. Psychological Assessment Resources.
Cotigă MI Development and validation of a Romanian version of the expanded version of Positive and Negative Affect Schedule (PANAS-X) Procedia - Social and Behavioral Sciences 2012 33 248 252 10.1016/j.sbspro.2012.01.121 Cotigă, M. I. (2012). Development and validation of a Romanian version of the expanded version of Positive and Negative Affect Schedule (PANAS-X). Procedia - Social and Behavioral Sciences,33, 248–252. 10.1016/j.sbspro.2012.01.121
Cronbach LJ Meehl PE Construct validity in psychological tests Psychological Bulletin 1955 52 4 281 13245896 10.1037/h0040957 Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin,52(4), 281.10.1037/h0040957 13245896
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding [Preprint]. arXiv. 10.48550/arXiv.1810.04805
Dunn EW Aknin LB Norton MI Spending money on others promotes happiness Science 2008 319 5870 1687 1688 18356530 10.1126/science.1150952 Dunn, E. W., Aknin, L. B., & Norton, M. I. (2008). Spending money on others promotes happiness. Science,319(5870), 1687–1688.10.1126/science.1150952 18356530
Dunn EW Aknin LB Norton MI Prosocial spending and happiness: Using money to benefit others pays off Current Directions in Psychological Science 2014 23 1 41 47 10.1177/0963721413512503 Dunn, E. W., Aknin, L. B., & Norton, M. I. (2014). Prosocial spending and happiness: Using money to benefit others pays off. Current Directions in Psychological Science,23(1), 41–47.
Eichstaedt JC Schwartz HA Kern ML Park GS Labarthe DR Merchant RM Jha S Agrawal M Dziurzynski L Sap M Weeg C Larson ER Ungar LH Seligman MEP Psychological language on Twitter predicts county-level heart disease mortality Psychological Science 2015 26 2 159 169 10.1177/0956797614557867 25605707 Eichstaedt, J. C., Schwartz, H. A., Kern, M. L., Park, G. S., Labarthe, D. R., Merchant, R. M., ..., & Seligman, M. E. P. (2015). Psychological language on Twitter predicts county-level heart disease mortality. Psychological Science,26(2), 159–169. 10.1177/0956797614557867 PMC4433545 25605707
Eichstaedt JC Smith R Merchant RM Ungar LH Crutchley P Preoţiuc-Pietro D Asch DA Schwartz HA Facebook language predicts depression in medical records Proceedings of the National Academy of Sciences of the United States of America 2018 115 44 11203 11208 10.1073/pnas.1802331115 30322910 Eichstaedt, J. C., Smith, R., Merchant, R. M., Ungar, L. H., Crutchley, P., Preoţiuc-Pietro, D., ..., & Schwartz, H. A. (2018). Facebook language predicts depression in medical records. Proceedings of the National Academy of Sciences of the United States of America,115(44), 11203–11208. 10.1073/pnas.1802331115 PMC6217418 30322910
Eichstaedt, J. C., Kern, M. L., Yaden, D. B., Schwartz, H. A., Giorgi, S., Park, G., Hagan, C. A., Tobolsky, V. A., Smith, L. K., Buffone, A., & Iwry, J. (2021). Closed-and open-vocabulary approaches to text analysis: A review, quantitative comparison, and recommendations. Psychological Methods, 26(4), 398.10.1037/met0000349 34726465
Eijsbroek, V. C., Ackermann, L., Nilsson, A., Gu, Z., Wiebel, C., Ganesan, A. V., Kjell, K., Schwartz, H. A., & Kjell, O. N. E. (2025a). Multiple methods for visualizing human language: A tutorial for social and behavioural scientists [Preprint]. PsyArXiv. 10.31234/osf.io/nxfvr_v1
Eijsbroek VC Kjell K Schwartz HA Boehnke JR Fried EI Klein DN Gustafsson P Augenstein I Bossuyt PM Kjell ON Kjell ON The LEADING guideline: Reporting standards for expert panel, best-estimate diagnosis, and longitudinal expert all data (LEAD) methods Comprehensive Psychiatry 2025 10.1016/j.comppsych.2025.152603 40479924 Eijsbroek, V. C., Kjell, K., Schwartz, H. A., Boehnke, J. R., Fried, E. I., Klein, D. N., ..., & Kjell, O. N. (2025b). The LEADING guideline: Reporting standards for expert panel, best-estimate diagnosis, and longitudinal expert all data (LEAD) methods. Comprehensive Psychiatry. 10.1016/j.comppsych.2025.152603 PMC12584586 40479924
Ekman P An argument for basic emotions Cognition & Emotion 1992 6 3–4 169 200 10.1080/02699939208411068 Ekman, P. (1992). An argument for basic emotions. Cognition & Emotion,6(3–4), 169–200. 10.1080/02699939208411068
Elsey, J. W. B. (2021). Powerful sequential designs using Bayesian estimation: A power analysis tutorial using brms, the tidyverse, and furrr [Preprint]. PsyArXiv. 10.31234/osf.io/kt4pz
Enevoldsen, K., Chung, I., Kerboua, I., Kardos, M., Mathur, A., Stap, D., Gala, J., Siblini, W., Krzemiński, D., Winata, G. I., Sturua, S., Utpala, S., Ciancone, M., Schaeffer, M., Sequeira, G., Misra, D., Dhakal, S., Rystrøm, J., Solomatin, R., ..., & Muennighoff, N. (2025). MMTEB: Massive multilingual text embedding benchmark [Preprint]. arXiv. 10.48550/arXiv.2502.13595
Flake JK Pek J Hehman E Construct validation in social and personality research: Current practice and recommendations Social Psychological and Personality Science 2017 8 4 370 378 10.1177/1948550617693063 Flake, J. K., Pek, J., & Hehman, E. (2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science,8(4), 370–378.
Folk D Dunn E A systematic review of the strength of evidence for the most commonly recommended happiness strategies in mainstream media Nature Human Behaviour 2023 7 10 1697 1707 37474838 10.1038/s41562-023-01651-4 Folk, D., & Dunn, E. (2023). A systematic review of the strength of evidence for the most commonly recommended happiness strategies in mainstream media. Nature Human Behaviour,7(10), 1697–1707.10.1038/s41562-023-01651-4 37474838
Folk D Dunn E How can people become happier? A systematic review of preregistered experiments Annual Review of Psychology 2024 75 1 467 493 37566759 10.1146/annurev-psych-022423-030818 Folk, D., & Dunn, E. (2024). How can people become happier? A systematic review of preregistered experiments. Annual Review of Psychology,75(1), 467–493.10.1146/annurev-psych-022423-030818 37566759
Fox J Weisberg S An R companion to applied regression 2019 3rdSageFox, J., & Weisberg, S. (2019). An R companion to applied regression (3rd ed.). Sage.
Ganesan, A. V., Matero, M., Ravula, A. R., Vu, H., & Schwartz, H. A. (2021). Empirical evaluation of pre-trained transformers for human-level NLP: The role of sample size and dimensionality. In Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: Human language technologies (pp. 4515–4532). Association for Computational Linguistics. 10.18653/v1/2021.naacl-main.357 PMC8294338 34296226
Ganesan, A. V., Varadarajan, V., Mittal, J., Subrahmanya, S., Matero, M., Soni, N., Guntuku, S. C., Eichstaedt, J., & Schwartz, H. A. (2022). WWBP-SQT-lite: Multi-level models and difference embeddings for moments of change identification in mental health forums. In Proceedings of the eighth workshop on computational linguistics and clinical psychology (pp. 251–258). Association for Computational Linguistics. 10.18653/v1/2022.clpsych-1.25
Gosling SD Rentfrow PJ Swann WB Jr.A very brief measure of the big five personality domains Journal of Research in Personality 2003 37 504 528 10.1016/S0092-6566(03)00046-1 Gosling, S. D., Rentfrow, P. J., & Swann, W. B., Jr. (2003). A very brief measure of the big five personality domains. Journal of Research in Personality,37, 504–528.
Greenaway KH Kalokerinos EK Williams LA Context is everything (in emotion research) Social and Personality Psychology Compass 2018 12 6 e12393 10.1111/spc3.12393 Greenaway, K. H., Kalokerinos, E. K., & Williams, L. A. (2018). Context is everything (in emotion research). Social and Personality Psychology Compass,12(6), e12393. 10.1111/spc3.12393
Gu Z Kjell K Schwartz HA Kjell O Natural language response formats for assessing depression and worry with large language models: A sequential evaluation with model pre-registration Assessment 2024 33 6 927 953 10.1177/10731911251364022 PMC13379617 40974258 Gu, Z., Kjell, K., Schwartz, H. A., & Kjell, O. (2024). Natural language response formats for assessing depression and worry with large language models: A sequential evaluation with model pre-registration. Assessment,33(6), 927–953.10.1177/10731911251364022 PMC13379617 40974258
Hoemann K Beyond linguistic relativity, emotion concepts illustrate how meaning is contextually and individually variable Topics in Cognitive Science 2023 15 4 668 675 10.1111/tops.12659 37145872 Hoemann, K. (2023). Beyond linguistic relativity, emotion concepts illustrate how meaning is contextually and individually variable. Topics in Cognitive Science,15(4), 668–675. 10.1111/tops.12659 37145872
Hoemann K Warfel E Mills C Allen L Kuppens P Wormwood JB Using freely generated labels instead of rating scales to assess emotion in everyday life Assessment 2024 10.1177/10731911241283623 39344959 Hoemann, K., Warfel, E., Mills, C., Allen, L., Kuppens, P., & Wormwood, J. B. (2024). Using freely generated labels instead of rating scales to assess emotion in everyday life. Assessment. 10.1177/10731911241283623 39344959
Hoemann K Lee Y Dussault È Devylder S Ungar LH Geeraerts D Mesquita B The construction of emotional meaning in language Communications Psychology 2025 3 1 99 40624347 10.1038/s44271-025-00255-0 Hoemann, K., Lee, Y., Dussault, È., Devylder, S., Ungar, L. H., Geeraerts, D., & Mesquita, B. (2025). The construction of emotional meaning in language. Communications Psychology,3(1), 99.10.1038/s44271-025-00255-0 PMC12234366 40624347
Hoerl, A. E., & Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1), 55–67.
Joseph DL Chan MY Heintzelman SJ Tay L Diener E Scotney VS The manipulation of affect: A meta-analysis of affect induction procedures Psychological Bulletin 2020 146 4 355 31971408 10.1037/bul0000224 Joseph, D. L., Chan, M. Y., Heintzelman, S. J., Tay, L., Diener, E., & Scotney, V. S. (2020). The manipulation of affect: A meta-analysis of affect induction procedures. Psychological Bulletin,146(4), 355.10.1037/bul0000224 31971408
Kals E Schumacher D Montada L Emotional affinity toward nature as a motivational basis to protect nature Environment and Behavior 1999 31 178 202 10.1177/00139169921972056 Kals, E., Schumacher, D., & Montada, L. (1999). Emotional affinity toward nature as a motivational basis to protect nature. Environment and Behavior,31, 178–202.
Kaplan DM Hughes CD Schatten HT Mehl MR Armey MF Nugent NR Emotional change in its “natural habitat”: Measuring everyday emotion regulation with passive and active ambulatory assessment methods Journal of Psychotherapy Integration. 2022 33 2 123 140 37588252 10.1037/int0000291 Kaplan, D. M., Hughes, C. D., Schatten, H. T., Mehl, M. R., Armey, M. F., & Nugent, N. R. (2022). Emotional change in its “natural habitat”: Measuring everyday emotion regulation with passive and active ambulatory assessment methods. Journal of Psychotherapy Integration.,33(2), 123–140.10.1037/int0000291 PMC10427127 37588252
Kaplan, D. M., Arconada Alvarez, S. J., Palitsky, R., Choi, H., Clifford, G. D., Crozier, M., Dunlop, B. W., Grant, G. H., Greenleaf, M. N., Johnson, L. M., Maples-Keller, J., Levin-Aspenson, H. F., Mascaro, J. S., McDowall, A., Pozzo, N. S., Raison, C. L., Zarrabi, A. J., Rothbaum, B. O., & Lam, W. A. (2025). Fabla: A voice-based ecological assessment method for securely collecting spoken responses to researcher questions. Behavior Research Methods, 57, Article 257. 10.3758/s13428-025-02777-1 PMC12343679 40797121
Kassambara, A. (2022). ggcorrplot: Visualization of a correlation matrix using 'ggplot2' [R package]. Comprehensive R Archive Network (CRAN). https://CRAN.R-project.org/package=ggcorrplot
Kassambara, A. (2023a). ggpubr: 'ggplot2' based publication ready plots [R package]. Comprehensive R Archive Network (CRAN). https://CRAN.R-project.org/package=ggpubr
Kassambara, A. (2023b). rstatix: Pipe-friendly framework for basic statistical tests [R package]. Comprehensive R Archive Network (CRAN). https://CRAN.R-project.org/package=rstatix
Kjell, O. N. (2011). Sustainable well-being: A potential synergy between sustainability and well-being research. Review of General Psychology, 15(3), 255–266.
Kjell ONE Kjell K Garcia D Sikström S Semantic measures: Using natural language processing to measure, differentiate, and describe psychological constructs Psychological Methods 2019 24 1 92 115 10.1037/met0000191 29963879 Kjell, O. N. E., Kjell, K., Garcia, D., & Sikström, S. (2019). Semantic measures: Using natural language processing to measure, differentiate, and describe psychological constructs. Psychological Methods,24(1), 92–115. 10.1037/met0000191 29963879
Kjell O Daukantaitė D Sikström S Computational language assessments of harmony in life—not satisfaction with life or rating scales—correlate with cooperative behaviors Frontiers in Psychology 2021 12 601679 34045988 10.3389/fpsyg.2021.601679 Kjell, O., Daukantaitė, D., & Sikström, S. (2021a). Computational language assessments of harmony in life—not satisfaction with life or rating scales—correlate with cooperative behaviors. Frontiers in Psychology,12, 601679.10.3389/fpsyg.2021.601679 PMC8144476 34045988
Kjell O Giorgi S Schwartz HA Text: An R-package for analyzing and visualizing human language using natural language processing and deep learning Psychological Methods 2021 28 6 1478 1498 10.1037/met0000542 PMC12803442 37126041 Kjell, O., Giorgi, S., & Schwartz, H. A. (2021b). Text: An R-package for analyzing and visualizing human language using natural language processing and deep learning. Psychological Methods,28(6), 1478–1498.10.1037/met0000542 PMC12803442 37126041
Kjell K Johnsson P Sikström S Freely generated word responses analyzed with artificial intelligence predict self-reported symptoms of depression, anxiety, and worry Frontiers in Psychology 2021 10.3389/fpsyg.2021.602581 34149500 PMC8212927 Kjell, K., Johnsson, P., & Sikström, S. (2021c). Freely generated word responses analyzed with artificial intelligence predict self-reported symptoms of depression, anxiety, and worry. Frontiers in Psychology. 10.3389/fpsyg.2021.602581 PMC8212927 34149500
Kjell ONE Sikström S Kjell K Schwartz HA Natural language analyzed with AI-based transformers predict traditional subjective well-being measures approaching the theoretical upper limits in accuracy Scientific Reports 2022 10.1038/s41598-022-07520-w 35273198 Kjell, O. N. E., Sikström, S., Kjell, K., & Schwartz, H. A. (2022). Natural language analyzed with AI-based transformers predict traditional subjective well-being measures approaching the theoretical upper limits in accuracy. Scientific Reports. 10.1038/s41598-022-07520-w PMC8913644 35273198
Kjell, O. N. E., Kjell, K., & Schwartz, H. A. (2023, January 4). AI-based large language models are ready to transform psychological health assessment. PsyArXiv. 10.31234/osf.io/yfd8g
Kjell, O. N. E., Ganesan, A. V., Boyd, R. L., Oltmanns, J. R., Rivero, A., Feltman, S., Carr, M. A., Luft, B., Kotov, R., & Schwartz, H. A. (2024). Demonstrating high validity of a new AI-language assessment of PTSD: A sequential evaluation with model pre-registration [Preprint]. PsyArXiv. 10.31234/osf.io/xw24e
Krosnick JA Survey research Annual Review of Psychology 1999 50 1 537 567 15012463 10.1146/annurev.psych.50.1.537 Krosnick, J. A. (1999). Survey research. Annual Review of Psychology,50(1), 537–567.10.1146/annurev.psych.50.1.537 15012463
Kučera D Haviger J Using mood induction procedures in psychological research Procedia - Social and Behavioral Sciences 2012 69 31 40 10.1016/j.sbspro.2012.11.380 Kučera, D., & Haviger, J. (2012). Using mood induction procedures in psychological research. Procedia - Social and Behavioral Sciences,69, 31–40. 10.1016/j.sbspro.2012.11.380
Kuhn, M., Wing, J., Weston, S., Williams, A., Keefer, C., Engelhardt, A., ... & Team, R. C. (2020). Package ‘caret’. The R Journal, 223(7), 48.
Kumar A Killingsworth MA Gilovich T Spending on doing promotes more moment-to-moment happiness than spending on having Journal of Experimental Social Psychology 2020 88 103971 10.1016/j.jesp.2020.103971 Kumar, A., Killingsworth, M. A., & Gilovich, T. (2020). Spending on doing promotes more moment-to-moment happiness than spending on having. Journal of Experimental Social Psychology,88, Article 103971.
Kuo, M. (2015). How might contact with nature promote human health? Promising mechanisms and a possible central pathway. Frontiers in Psychology, 6, 1093.10.3389/fpsyg.2015.01093 PMC4548093 26379564
Li, X., & Li, J. (2023). AnglE-optimized text embeddings [Preprint]. arXiv. 10.48550/arXiv.2309.12871
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M. S., Chen, D., Levy, O., ..., & Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv(Cornell University). 10.48550/arxiv.1907.11692
Löwe B Kroenke K Gräfe K Detecting and monitoring depression with a two-item questionnaire (PHQ-2) Journal of Psychosomatic Research 2005 58 2 163 171 10.1016/j.jpsychores.2004.09.006 15820844 Löwe, B., Kroenke, K., & Gräfe, K. (2005). Detecting and monitoring depression with a two-item questionnaire (PHQ-2). Journal of Psychosomatic Research,58(2), 163–171. 10.1016/j.jpsychores.2004.09.006 15820844
Mailund T Functional programming: Purrr R 4 data science quick reference: A pocket guide to APIs, libraries, and packages 2022 Apress89 110 10.1007/978-1-4842-8780-4_7 Mailund, T. (2022). Functional programming: Purrr. R 4 data science quick reference: A pocket guide to APIs, libraries, and packages (pp. 89–110). Apress.
Marcusson-Clavertz D Kjell ONE Persson S Cardeña E Online validation of combined mood induction procedures PLoS One 2019 14 6 e0217848 10.1371/journal.pone.0217848 31163062 Marcusson-Clavertz, D., Kjell, O. N. E., Persson, S., & Cardeña, E. (2019). Online validation of combined mood induction procedures. PLoS One,14(6), e0217848. 10.1371/journal.pone.0217848 PMC6548374 31163062
Merz EL Malcarne VL Roesch SC Ko CM Emerson MA Roma VG Sadler GR Psychometric properties of Positive and Negative Affect Schedule (PANAS) original and short forms in an African American community sample Journal of Affective Disorders 2013 151 3 942 949 10.1016/j.jad.2013.08.011 24051099 Merz, E. L., Malcarne, V. L., Roesch, S. C., Ko, C. M., Emerson, M. A., Roma, V. G., & Sadler, G. R. (2013). Psychometric properties of Positive and Negative Affect Schedule (PANAS) original and short forms in an African American community sample. Journal of Affective Disorders,151(3), 942–949. 10.1016/j.jad.2013.08.011 PMC3934411 24051099
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2023). Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 56(2), 1–40. 10.1145/3605943
Mohammad SM Turney PD Crowdsourcing a word–emotion association lexicon Computational Intelligence 2013 29 3 436 465 10.1111/j.1467-8640.2012.00460.x Mohammad, S. M., & Turney, P. D. (2013). Crowdsourcing a word–emotion association lexicon. Computational Intelligence,29(3), 436–465.
Muchinsky PM The correction for attenuation Educational and Psychological Measurement 1996 56 1 63 75 10.1177/0013164496056001004 Muchinsky, P. M. (1996). The correction for attenuation. Educational and Psychological Measurement,56(1), 63–75.
Nielsen MD Kajonius P Beyond the big five factors: Using facets and nuances for enhanced prediction in life outcomes Current Psychology 2024 43 18621 18630 10.1007/s12144-024-05662-w Nielsen, M. D., & Kajonius, P. (2024). Beyond the big five factors: Using facets and nuances for enhanced prediction in life outcomes. Current Psychology,43, 18621–18630. 10.1007/s12144-024-05662-w
Nilsson A Hellryd E Kjell ONE Doing well-being: Self-reported activities are related to subjective well-being PLoS One 2022 17 6 e0270503 10.1371/journal.pone.0270503 35749555 Nilsson, A., Hellryd, E., & Kjell, O. N. E. (2022). Doing well-being: Self-reported activities are related to subjective well-being. PLoS One,17(6), e0270503. 10.1371/journal.pone.0270503 PMC9232132 35749555
Nilsson AH Schwartz HA Rosenthal RN McKay JR Vu H Cho YM Mahwish Syeda Ganesan Adithya V. Ungar L Language-based EMA assessments help understand problematic alcohol consumption PLoS One 2024 19 3 e0298300 38446796 10.1371/journal.pone.0298300 Nilsson, A. H., Schwartz, H. A., Rosenthal, R. N., McKay, J. R., Vu, H., Cho, Y. M., Mahwish, Syeda, Ganesan, Adithya V.., & Ungar, L. (2024a). Language-based EMA assessments help understand problematic alcohol consumption. PLoS One,19(3), e0298300.10.1371/journal.pone.0298300 PMC10917301 38446796
Nilsson, A., Boyd, R. L., Ganesan, A. V., Kjell, O. N. E., Mahwish, S., Huang, H., Rosenthal, R. N., Ungar, L. H., & Schwartz, H. A. (2024b). Language-based assessments for experienced well-being: Accuracy and external validity across behaviors, traits, and states [Preprint]. PsyArXiv. 10.31234/osf.io/dgnaf
Nilsson AH Eijsbroek VC Gu Z Kjell K Giorgi S Kotov R Ganesan Adithya V. Schwartz H. Andrew Kjell Oscar N. E. Kjell ON The language-based assessment model library: Open model sharing for independent validation and broader applications Advances in Methods and Practices in Psychological Science 2026 9 2 25152459261419036 10.1177/25152459261419036 Nilsson, A. H., Eijsbroek, V. C., Gu, Z., Kjell, K., Giorgi, S., Kotov, R., ..., & Kjell, O. N. (2026). The language-based assessment model library: Open model sharing for independent validation and broader applications. Advances in Methods and Practices in Psychological Science,9(2), 25152459261419036.
Park GS Schwartz HA Eichstaedt JC Kern ML Kosinski M Stillwell D Ungar LH Seligman MEP Automatic personality assessment through social media language Journal of Personality and Social Psychology 2015 108 6 934 952 10.1037/pspp0000020 25365036 Park, G. S., Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Kosinski, M., Stillwell, D., Ungar, L. H., & Seligman, M. E. P. (2015). Automatic personality assessment through social media language. Journal of Personality and Social Psychology,108(6), 934–952. 10.1037/pspp0000020 25365036
Pennebaker JW The secret life of pronouns New Scientist 2011 211 2828 42 45 10.1016/s0262-4079(11)62167-2 Pennebaker, J. W. (2011). The secret life of pronouns. New Scientist,211(2828), 42–45. 10.1016/s0262-4079(11)62167-2
Pennebaker JW Mehl MR Niederhoffer KG Psychological aspects of natural language use: Our words, our selves Annual Review of Psychology 2003 54 1 547 577 10.1146/annurev.psych.54.101601.145041 12185209 Pennebaker, J. W., Mehl, M. R., & Niederhoffer, K. G. (2003). Psychological aspects of natural language use: Our words, our selves. Annual Review of Psychology,54(1), 547–577. 10.1146/annurev.psych.54.101601.145041 12185209
Posit TeamRStudio: Integrated Development Environment for R 2023 Posit Software PBChttp://www.posit.co/Posit Team. (2023). RStudio: Integrated Development Environment for R. Posit Software PBC. http://www.posit.co/
R Core Team. (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
Reif E Yuan A Wattenberg M Visualizing and measuring the geometry of BERT Proceedings of the Advances in Neural Information Processing Systems 2019 Vancouver8592 8600 Reif, E., Yuan, A., Wattenberg, M., et al. (2019). Visualizing and measuring the geometry of BERT. Proceedings of the Advances in Neural Information Processing Systems (pp. 8592–8600). Vancouver.
Revelle W psych: Procedures for Psychological, Psychometric, and Personality Research (2.3.3) 2023 Northwestern Universityhttps://CRAN.R-project.org/package=psychRevelle, W. (2023). psych: Procedures for Psychological, Psychometric, and Personality Research (2.3.3). Northwestern University. https://CRAN.R-project.org/package=psych
Russell JA A circumplex model of affect Journal of Personality and Social Psychology 1980 39 6 1161 1178 10.1037/h0077714 Russell, J. A. (1980a). A circumplex model of affect. Journal of Personality and Social Psychology,39(6), 1161–1178.
Russell JA A circumplex model of affect Journal of Personality and Social Psychology 1980 39 6 1161 1178 10.1037/h0077714 Russell, J. A. (1980b). A circumplex model of affect. Journal of Personality and Social Psychology,39(6), 1161–1178. 10.1037/h0077714
Sametoğlu S Pelt DHM Eichstaedt JC Ungar LH Bartels M The value of social media language for the assessment of wellbeing: A systematic review and meta-analysis Journal of Positive Psychology 2024 19 3 471 489 10.1080/17439760.2023.2218341 Sametoğlu, S., Pelt, D. H. M., Eichstaedt, J. C., Ungar, L. H., & Bartels, M. (2024). The value of social media language for the assessment of wellbeing: A systematic review and meta-analysis. Journal of Positive Psychology,19(3), 471–489.
Schwartz HA Eichstaedt JC Kern ML Dziurzynski L Ramones SM Agrawal M Shah A Kosinski M Stillwell D Seligman MEP Ungar LH Personality, gender, and age in the language of social media: The open-vocabulary approach PLoS One 2013 8 9 e73791 10.1371/journal.pone.0073791 24086296 Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Dziurzynski, L., Ramones, S. M., Agrawal, M., ..., & Ungar, L. H. (2013). Personality, gender, and age in the language of social media: The open-vocabulary approach. PLoS One,8(9), e73791. 10.1371/journal.pone.0073791 PMC3783449 24086296
Schwartz H Eichstaedt JC Kern ML Park GS Sap M Stillwell D Kosinski M Ungar LH Resnik Philip Resnik Rebecca Mitchell Margaret Towards Assessing Changes in Degree of Depression through Facebook Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality 2014 Association for Computational Linguistics118 125 10.3115/v1/w14-3214 Schwartz, H., Eichstaedt, J. C., Kern, M. L., Park, G. S., Sap, M., Stillwell, D., Kosinski, M., & Ungar, L. H. (2014). Towards assessing changes in degree of depression through facebook. In P. Resnik, R. Resnik, & M. Mitchell (Eds.), Proceedings of the workshop on computational linguistics and clinical psychology: From linguistic signal to clinical reality (pp. 118–125). Association for Computational Linguistics. 10.3115/v1/w14-3214
Seeboth A Mõttus R Successful explanations start with accurate descriptions: Questionnaire items as personality markers for more accurate predictions European Journal of Personality 2018 32 3 186 201 10.1002/per.2147 Seeboth, A., & Mõttus, R. (2018). Successful explanations start with accurate descriptions: Questionnaire items as personality markers for more accurate predictions. European Journal of Personality,32(3), 186–201. 10.1002/per.2147
Son Y Clouston SA Kotov R Eichstaedt JC Bromet EJ Luft BJ Schwartz HA World Trade Center responders in their own words: Predicting PTSD symptom trajectories with AI-based language analyses of interviews Psychological Medicine 2023 53 3 918 926 34154682 10.1017/S0033291721002294 PMC8692489 Son, Y., Clouston, S. A., Kotov, R., Eichstaedt, J. C., Bromet, E. J., Luft, B. J., & Schwartz, H. A. (2023). World Trade Center responders in their own words: Predicting PTSD symptom trajectories with AI-based language analyses of interviews. Psychological Medicine,53(3), 918–926.10.1017/S0033291721002294 PMC8692489 34154682
Spearman C Jenkins JJ Paterson DG The proof and measurement of association between two things Studies in individual differences: The search for intelligence 1961 Appleton-Century-Crofts45 58 Spearman, C. (1961). The proof and measurement of association between two things. In J. J. Jenkins & D. G. Paterson (Eds.), Studies in individual differences: The search for intelligence (pp. 45–58). Appleton-Century-Crofts.
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agarwal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., ..., & Wu, Z. (2022). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models [Preprint]. arXiv. 10.48550/arXiv.2206.04615
Stewart RD Mõttus R Soto CJ Seeboth A Johnson W The finer details? The predictability of life outcomes from big five domains, facets, and nuances Journal of Personality 2022 19 2 167 182 10.1111/jopy.12660 34236710 Stewart, R. D., Mõttus, R., Soto, C. J., Seeboth, A., & Johnson, W. (2022). The finer details? The predictability of life outcomes from big five domains, facets, and nuances. Journal of Personality,19(2), 167–182. 10.1111/jopy.12660 34236710
Sun J Kern ML Schwartz HA Son Y Vazire S The language of well-being: Tracking fluctuations in emotion experience through everyday speech Journal of Personality and Social Psychology 2019 118 2 364 387 30945904 10.1037/pspp0000244 Sun, J., Kern, M. L., Schwartz, H. A., Son, Y., & Vazire, S. (2019). The language of well-being: Tracking fluctuations in emotion experience through everyday speech. Journal of Personality and Social Psychology,118(2), 364–387.10.1037/pspp0000244 30945904
Terracciano A McCrae R Costa PT Factorial and construct validity of the Italian Positive and Negative Affect Schedule (PANAS) European Journal of Psychological Assessment 2003 19 131 141 20467578 10.1027//1015-5759.19.2.131 Terracciano, A., McCrae, R., & Costa, P. T. (2003). Factorial and construct validity of the Italian Positive and Negative Affect Schedule (PANAS). European Journal of Psychological Assessment,19, 131–141.10.1027//1015-5759.19.2.131 PMC2868265 20467578
Vainre, M., Anni, K., Vainik, U., & Mõttus, R. (2025). How satisfaction varies among 263 occupations [Preprint]. PsyArXiv. 10.31234/osf.io/8zqgd_v1
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems (Vol. 30). Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
von Humboldt S Monteiro A Leal I Validation of the PANAS: A measure of positive and negative affect for use with cross-national older adults Review of European Studies 2017 9 2 10 10.5539/res.v9n2p10 von Humboldt, S., Monteiro, A., & Leal, I. (2017). Validation of the PANAS: A measure of positive and negative affect for use with cross-national older adults. Review of European Studies,9(2), 10. 10.5539/res.v9n2p10
Warriner AB Kuperman V Brysbaert M Norms of valence, arousal, and dominance for 13,915 English lemmas Behavior Research Methods 2013 45 1191 1207 23404613 10.3758/s13428-012-0314-x Warriner, A. B., Kuperman, V., & Brysbaert, M. (2013). Norms of valence, arousal, and dominance for 13,915 English lemmas. Behavior Research Methods,45, 1191–1207.10.3758/s13428-012-0314-x 23404613
Watson D Clark LA The PANAS-X: Manual for the positive and negative affect schedule—expanded form 1999 University of IowaWatson, D., & Clark, L. A. (1999). The PANAS-X: Manual for the positive and negative affect schedule—expanded form. University of Iowa.
Watson D Clark LA Tellegen A Development and validation of brief measures of positive and negative affect: The PANAS scales Journal of Personality and Social Psychology 1988 54 6 1063 1070 10.1037/0022-3514.54.6.1063 3397865 Watson, D., Clark, L. A., & Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology,54(6), 1063–1070. 10.1037/0022-3514.54.6.1063 3397865
Wei, T., & Simko, V. (2021). corrplot: Visualization of a correlation matrix [R package]. https://github.com/taiyun/corrplot
Westermann R Spies K Stahl G Hesse FW Relative effectiveness and validity of mood induction procedures: A meta-analysis European Journal of Social Psychology 1996 26 4 557 580 10.1002/(SICI)1099-0992(199607)26:4<557::AID-EJSP769>3.0.CO;2-4 Westermann, R., Spies, K., Stahl, G., & Hesse, F. W. (1996). Relative effectiveness and validity of mood induction procedures: A meta-analysis. European Journal of Social Psychology,26(4), 557–580.
Whitehead BR Bergeman CS Coping with daily stress: Differential role of spiritual experience on daily positive and negative affect Journals of Gerontology Series B: Psychological Sciences and Social Sciences 2012 67 4 456 459 22193637 10.1093/geronb/gbr136 PMC3530403 Whitehead, B. R., & Bergeman, C. S. (2012). Coping with daily stress: Differential role of spiritual experience on daily positive and negative affect. Journals of Gerontology Series B: Psychological Sciences and Social Sciences,67(4), 456–459.10.1093/geronb/gbr136 PMC3530403 22193637
Wickham H Averick M Bryan J Chang WW McGowan LD François R Grolemund G Hayes AG Henry L Hester J Kuhn M Pedersen TG Miller EW Bache SM Müller K Ooms J Robinson D Seidel DP Spinu V Chang Winston McGowan Lucy Hayes Alex Pedersen Thomas Miller Evan Bache Stephan Seidel Dana Takahashi Kohske Vaughan Davis Wilke Claus Woo Kara Yutani H Welcome to the tidyverse Journal of Open Source Software 2019 4 43 1686 10.21105/joss.01686 Wickham, H., Averick, M., Bryan, J., Chang, W. W., McGowan, L. D., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., Kuhn, M. (2019). Welcome to the tidyverse. Journal of Open Source Software,4(43), 1686. 10.21105/joss.01686
Wickham, H., & Bryan, J. (2019). readxl: Read excel files [R package]. Comprehensive R Archive Network (CRAN). https://CRAN.R-project.org/package=readxl
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2023). Can large language models transform computational social science? [Preprint]. arXiv. 10.48550/arXiv.2305.03514
Zumbach, D., & Bauer, P. C. (2021). deeplr: Interface to the 'DeepL' translation API [R package]. Comprehensive R Archive Network (CRAN). https://CRAN.R-project.org/package=deeplr
Come citare questo studio
Bång O, Gu Z, Nilsson AH, Eijsbroek VC, Wiebel C, Ganesan AV, Kjell K, Schwartz HA, Kjell ONE. (2026). Language-based affect assessments capture experiment-induced changes beyond rating scales.. Behavior research methods. DOI: 10.3758/s13428-026-03200-z
Consulta la pubblicazione originale ↗