If there’s a Trigger Warning, then where’s the Trigger? Investigating Trigger Warnings at the Passage Level (2024)

Trigger Diversity

The trigger warnings used in practice are based on the personal opinions and experiences of many authors and thus very diverse. The WTWC-22 organizes them into seven categories containing 36warnings, encompassing hundreds of thousands of variants. Annotating a given passage with all 36warnings, let alone all variants, has proven infeasible at scale under reasonable budget constraints. We therefore select eight common warnings for annotation, four each from the two most frequently assigned warning categories Aggression and Discrimination of the WTWC-22 taxonomy. From Aggression we use all four warnings Death, Violence, Abduction and War. From Discrimination we use the four most frequently assigned warnings Misogyny, Racism, hom*ophobia and Ableism. The former relate primarily to physical harm, the latter to psychological harm.

Trigger Sparsity

Harmful passages are often sparsely distributed over long documents,²²2E.g., many fan fiction works are as long as books.leading to class imbalance between positive an negative cases. We therefore resort to the commonly used approach of dictionary-based retrieval to obtain a sufficient number of positive candidate passages as detailed below.³³3Retrieval-guided annotation have previously been employed to create NLP resources.This entails selection bias, since very subtle cases of triggering content may not contain any explicit mention of one of the dictionary’s phrases, and since some important phrases might be missing from our dictionaries, both leading to false negatives. We analyze this possibility with a targeted out-of-distribution experiment.

Trigger Severity

Authors and readers are not equally sensitive to a potentially triggering concept (see Section4.4). A reader who enjoys horror may find fewer references to Death in fiction triggering than readers who do not. For instance, the positive example in Figure1 might be considered enticing rather than harmful. Therefore, when annotators are asked to make a descriptive binary decision, they often disagree. This is not uncommon in work on harmful content and, according to rottger:2022, even desirable in an initial study like ours. However, this presents difficulties when assessing annotations and annotators, as inter-annotator agreement measures become less reliable due to different reader sensitivity to a particular trigger. This prevents the use of crowdsourcing platforms for annotation, as we cannot(i)quantitatively evaluate annotators,(ii)train them reliably ahead of time, and(iii)provide appropriate support during annotation.We therefore recruited local annotators and provided them with personal support.

4.1 Retrieving Triggering Passage Candidates

We collected the passages for annotation using a keyword-based retrieval approach. We first constructed a keyword list for each of the eight considered warnings, then retrieved matching documents from WTWC-22 using a BM25-based initial retrieval⁴⁴4We retrieved the documents via elasticsearch 7.17.4 with the default BM25 scoring on the chapter text. We applied html-removal and stemming after the default pre-processing. where the keywords served as query terms. From each document, we extracted the first sentence with a keyword match and added the two pre- and succeeding sentences as context. Adding context is often necessary understand the central sentence. We did not use the original (paragraph) segmentation of the documents, since they greatly varied in length, particularly since dialog turns are often written as single paragraph. The resulting five-sentence passages were de-duplicated across warnings and annotated in order of the BM25 score of the originating document.

4.2 Passage Annotation

Figure1 shows two passages, one for each annotation decision. The collected passages were annotated as either positive (requires a warning) or negative (does not require a warning) by three different annotators each. We decided on the final annotation based on the majority of the votes.

Instead of annotating a fixed number of passages for each warning, the first annotator of each warning rated continuously, in order of the BM25 score of the source document, until 50 passages were marked positive in each set (100 per warning). All passages marked either positive or negative by the first annotator were then also rated by the other annotators. This step is necessary because the ratio of positive-to-negative passages varies between the warnings and is very low (1:10) for some warnings like Death and War in set 2, so having an equal number of passages for each warning would results either in a very high number of annotations or a low number of positive examples for some classes.

4.3 Evaluation

We evaluate(i)if the keyword-based passage retrieval is effective,(ii)if the dataset is large and balanced enough for machine learning, and(iii)if the task design facilitates high-quality annotations.

4.4 Subjectivity and Annotator Beliefs

Table1b shows the passage count by number of positive votes. Out of all passages, 55 %were unanimously negative, but only9 % unanimously positive. The negatives are partially explained by off-topic passages retrieved by ambiguous keywords, although this does not account for all negatives (see Figure5). The variance across positive votes is explained by varying annotator sensitivity and a different belief of when a warning is required. Figure3 shows typical positive passages. Unanimous positive passages are often severe cases with heavy use of slurs and very graphic descriptions. Non-unanimous examples feature unfamiliar or unrelatable settings (fantasy, science fiction) or concepts (‘conversion therapy’), background or implied mentions, or a mix between warnings which confuse the annotators. Contributing to this are keywords like ‘undead’, ‘reanimation’, or ‘necromancy’ often contained in fantastical scenes (see Table3). Removing those keywords would reduce annotation load and annotator uncertainty but would also reduce recall, remove positive passages, and further limit the scope of the concepts for model training.

The vote aggregation threshold impacts the characteristics of a classifier. A low threshold (1vote) produces recall-oriented classifiers for warning assignment, while a high threshold (2–3votes) produces precision-oriented classifiers suitable for moderation tools. We assume that resolving annotator disagreement will require either a form of personalization with a per-warning known sensitivity, or a prescriptive transformation of the annotation task, where the triggers are intensionally defined.

Class Modeling

We comparatively evaluate binary, multiclass, and multilabel modeling via fine-tuned classification. We formulated the annotation task as binary classification: given a passage and a warning, decide if the warning should be assigned. Hence, our reference baseline and our evaluation setting is binary classification. However, binary classification is expensive to scale since each new warning needs a new classifier, and new training data, and each classifier does not use most of the annotated data. Under multiclass modeling, only one classifier is needed so the training data is better utilized, although such a classifier can only predict one warning per passage. Multilabel modeling combines these advantages of binary and multiclass, but is more difficult to train and our (binary) data might not be sufficient.

Fine-tuning vs. Few-shot Learning

We comparatively evaluate six few-shot prompted generative LLMs which can scale to a high-dimensional warning taxonomy like NEON. Trigger warnings are potentially open-class and might require personalization, which is difficult to scale to with model fine-tuning.

Vote Aggregation

We comparatively evaluate majority (2 votes count as positive) and minority voting (1 vote counts as positive) across all other experimental configurations. Our dataset evaluation suggested that examples containing clear and intense harm often receive multiple positive votes (24% with 2–3), while examples with one vote (23%) often contain only mild or implicit harm. The minimum number of votes required for an example to count as positive for classification will thus influence the sensitivity of the classifier.

Keyword Distribution

We evaluate all experiments across in-distribution (ID) and out-of-distribution (OOD) samples of the dataset. Using keyword-based filtering to pre-select passages may limit how well the classifiers generalize to unseen examples that match the respective warnings but not the concepts captured by the keywords. We simulate this situation by splitting the keywords (and the passages retrieved using them) into two non-overlapping sets (cf.4.1) and sampling datasets once from both sets combined and once separately.

5.1 Experiment Datasets

We compiled a total of 16 initial, unbalanced datasets, one for each of the eight warnings with majority and minority vote aggregation respectively. To get comparable results, we sampled balanced test sets with 20 positive and 20 negative instances for each of the 16 unbalanced datasets, which is the limit imposed by the smallest set (Misogyny Set 2). Since 40 instances are too few to get stable results, we created six random folds for a 5-fold Monte Carlo cross-validation with a sixth fold for parameter tuning.

For the in-distribution experiments, the test data are randomly drawn from all examples (i.e. they share passages retrieved by both keyword sets), while all other instances remain for training. In the out-of-distribution experiments, the test data are randomly drawn from all passages retrieved by keyword set 2, all other set 2 instances are discarded and all set 1 instances remain for training. Table1c shows the number of examples in the training and test datasets.

5.2 Models and Training

We evaluate five fine-tuning-based classification models Binary, Binary+ (which includes negative instances from other classes), Multiclass, Multilabel (without any all-negative instances) and Multilabel+ (with all-negative instances) and six generative LLMs GPT 3.5, GPT 4, Mistral 7B, Mixtral 7x8B, Llama 7B, and Llama 13B. We implemented the models using Huggingface’s transformers library (4.35.0). We prompted GPT via OpenAI’s API.

Llama 7B and 13B

We implemented both Llama models via Huggingface’s transformers library, using the ‘meta-llama/Llama-2-7b-chat-hf’ and ‘meta-llama/Llama-2-13b-chat-hf’ checkpoints respectively. We used the prompt directly and without the chat template, because the models produces difficult-to-parse output when using the chat template as we did with Mistral. Llama 7B was tested on two A100 40GB with a batch size of 12. Llama 12B was loaded using the A100 GPUs and batch sizes of 8.

5.3 Evaluation

We test all models on the balanced, binary test datasets for ease of comparison and report accuracy and positive rate. For multiclass and multilabel models we only count the predictions relating to the respective warning (cfLABEL:model-implementation). We report the mean performance across all five folds for each model and macro-averages across models and warnings.

Class Modeling

Model efficiency varies significantly between modeling strategies: Multiclass outperforms Binary (0.11 pp. for ID/MinV;) and Multilabel+ (0.23 pp. for ID/MinV; 0.07 pp. ID/MajV) and Binary outperforms Multilabel (0.09 for OOD/MinV). All other differences are not significant. Adding more negative examples to Binary shows no significant difference and adding negative examples to Multilabel improves the measured performance within the 95% CI by 0.01–0.06. Although Multiclass is the most effective fine-tuned model, it works only if the warnings are separable and rarely overlap. Since this is often not the case (cf. Figure3 (5)), the binary model is a more realistic alternative.

Fine-tuning vs. Few-shot Learning

The results show that all fine-tuned models perform equal or worse than the best few-shot models (Mixtral) in the OOD configuration. In the ID configuration, the fine-tuned models score comparative or higher (Multilabel for ID/MinV). The fine-tuned models are also more efficient in time and energy and should be preferred whenever possible.

Another notable difference is that all few-shot models have a higher average positive rate (PR) than the fine-tuned model (9–49 pp.), so the generative LLMs can be considered more sensitive. The reason for this is not clear, although explicit harm avoidance during training is a possible explanation.

Vote Aggregation

Increasing the aggregation threshold from one to two significantly increases the accuracy for Multilabel+ and decreases the accuracy for Multiclass, for Ableism, and for Abduction (all ID). In addition, the inter-quartile range of the folds (box plots in Figure4) is smaller for MajV. This is likely explained by the reduced number of positive examples, which reduced the variance of the examples in the folds since there are much fewer positive examples to draw from.

Increasing the threshold also reduces the PR (3–28 pp.) across all fine-tuned models and increases the PR (6–10 pp.) across all few-shot models. The increase is likely explained by the overall increases of instances with 1+ votes increases with higher thresholds, since these get samples as the negative class and, since the the prompt did not change, this increases the few-shot models’ positive rates.

Keyword Distribution

Overall, all models are less effective in the OOD configuration. This is expected behavior for fine-tuned models, since they can rely less on learned lexical cues, but not necessarily for few-shot models. A possible explanation is that we split the keywords in the order of generation, so keywords that are strongly associated with a trigger are in set 1. The OOD test set then contains passages that are less strongly associated with the trigger, thus reducing the few-shot scores.

In the OOD configurations, all fine-tuned models are significantly worse by ca. 0.1–0.2 pp. on average with an increased spread of per-warning scores while the few-shot models score as on ID (except for a reduction on Racism). Fine-tuned models are still competitive for most Aggression but ineffective for Discrimination warnings with barely above random accuracy and very low PRs. This is likely due to systematic differences between the keywords (cf. 3). While the Aggression keywords are somewhat consistent between sets, Discrimination keywords often start with a series of slurs (Set 1) and become more conceptual later (Set 2), leading to a topical gap between OOD training and test. This was the goal of the configuration and shows that fine-tuned models generalize poorly in this case, highlighting the importance of diverse training data.

6.1 Causes of Misclassification

Table5 shows selected passages that are always misclassified. Cases of systematic misclassification contain passages that (i)are on-topic but are never rated as triggering by the annotators, (ii)mention the topic implicitly or in the background (like historic discrimination, descriptions of the setting) but are never rated as triggering by the annotators, and (iii)with edge cases which are rated positively by only one annotator. Misclassified instances seem to align with annotator disagreement. Across all few-shot models, many (56%/75%) instances with 1 or 3 votes are always classified correctly, but only 15% with 1 vote. Mixtral classifies 70–91% of passages with 1/2/3-votes correctly, but only 30% of 1-vote instances. The fine-tuned models, too, more often misclassify examples with disagreement but less extreme: Binary (ID) is correct for 66% of 3-vote and only 44% on 2-vote instances.

A.1 Dataset Evaluation

A.2 Classification Evaluation

B.1 Language Modeling Fine-tuning of RoBERTa for Fan Fiction

We fine-tuned the ‘roberta-base’ checkpoint on fan fiction documents via masked language modeling using Huggingface’s ‘Trainer’ routine. As data, we extracted all English fan fiction documents from WTWC-22 that were marked as recommended for model training, i.e. where a trigger warning was assigned and where overly long, short, or unpopular works were removed (wiegmann:2023a list the precise parameters). All documents were split into training examples of ca. 450-500 words, respecting sentence and paragraph boundaries. We trained the checkpoint on the resulting 19 million examples for ca. 470,000 steps with a batch size of 32, the point of loss convergence on a 10,000 example hold-out validation set. As parameters, we used largely the standard parameters of the ‘Trainer’, with a random masking function, masking probability of 0.2, and an initial learning rate of 2e-5.

B.2 Fine-tuning Parameter Sweep

We conducted a grid-based parameter sweep on the validation sample (cf. Section5.1) for all fine-tuning strategy (binary, multi-label, multi-class, with and without extended training data) across three dimensions: in vs out-of-distribution, minority vs. majority vote aggregation, and learning rate within $(1e-5,2e-5,5e-5)$. All models were trained for 20 epochs. The parameters did not vary between warnings. Table6 shows the final parameter settings each model was trained on.

B.3 Prompt Ablation

Figure2 shows the 5 parts of the instructions we used for both the annotators and as the prompt for the few-shot models: Instruction, Passage, Persona, Definition, and Demonstrations (one positive and one negative in that order). We counted model responses as positive when they contained ‘yes’ within the first 5 response tokens and negative if they contained ‘no’

For the prompt ablation, we queried all generative LLMs with open weights (excluding GPT) across all 4 test settings (aggregation and distribution), averaged the scores across all settings per model, and averaged the scores across all models to determine the best prompt template. We tested eight prompt template variations (cf. Table7). Each variation started with Instruction and Passage, followed by all combinations of Persona, Definition, and Demonstrations, including neither. We manually selected the demonstrations that we rated as clear and representative for the respective trigger out of all unanimously annotated passages. We selected the prompt with all parts, which had with the highest average accuracy on the validation sample across all models and was also the one shown to annotators.