Human values are notoriously dicfiult to predict as they are often of nuanced nature, culturally embedded and varying across geographies. Generative Large Language Models (LLMs) have become very powerful to mimic how people use language, including value-laden content. We explore the opportunities for supervised fine-tuning and prompt engineering the LLMs in order to better perform a downstream task such as finding value-laden content in text. We compare fine-tuning, which heavily relies on labeled data, to the more flexible approach of prompt engineering that requires less or no labeled data at all. Our goal in this paper is three-fold: 1) assess the capabilities of closed source (GPT-3.5 and GPT-4o) versus open source (Gemini and Llama3) LLMs, 2) analyse the influence of domain-specific information by comparing fine-tuning with prompts, and 3) compare multi-label with single-label approaches.
Higher-order constructs such as human values are likely picked up by transformer models 1. One way to efectively use these pre-trained transformer models to capture nuances in texts containing human values is through fine-tuning. It involves taking a pre-trained model and adopting to a specific language modeling task by further training on a smaller task-specific dataset. The idea is that, through ifne-tuning, the new model captures both the general linguistic features from the pre-training phase as well as improve performance on the specific task by adjusting the model’s parameters during fine-tuning.
Fine-tuning can be done in various ways, including supervised, unsupervised and semi-supervised approaches. For our case, we use supervised fine-tuning (SFT) which involves further training the model on a labelled dataset to adapt it to a new task. SFT involves selecting a relevant pre-training model for the task, preparing a labeled dataset tailored to the task and then extracting a new model with adjusted parameters that captures the specific nuances of the ask. The main advantage of SFT is the improved performance on the specific task while also being highly resource eficient, requiring less data and computational capacity compared to training a new model from scratch.
In our paper, we use four models, two from OpenAI (GPT 3.5, GPT 4o), Gemini-1.0-pro and
Llama370B-Instruct, all of which are based on the transformer architecture introduced by Vaswani et al. (2017)
[
To maximise the benefit from these capabilities of the LLMs, we integrate description of human values
directly into the prompts. By including the information about all possible values in the prompt but
instructing the model to only report one value per sentence, we ensure that the model assigns a single
value to each sentence. Prompting is quite sensitive to the information fed, meaning even small changes
in prompts can lead to significantly diferent results which emphasizes the importance of an efective
prompt design [
Prompting also allows us to give examples with this description which is a bridge between fine-tuning (giving many examples) and zero-shot prompting (giving only description). We apply few-shot SL prompting by carefully selecting examples from the training set so that the model is able to learn to distinguish between the positive and negative examples for each value. After some experimentation, we sufice with 3 positive examples, and 3 negative examples. The example sentences are selected from a dataset of sentences based on the words they have in common with the value-labeled sentences. Before matching sentences based on words, we remove the words that are common across all sentences (see Appendix B.2). For the negative examples, we select sentences that are a) randomly drawn from those sentences not annotated by the vocal value, b) annotated with a related value adjacent in the circle, and c) annotated with an opposed value. This way, we hope that we show the algorithm specialized information on what constitutes a value and what does not.
From the training set, we selected sentences to be used as examples in prompts as well as labeled data used for fine-tuning. We remove sentences with fewer than 15 characters are excluded from this selection as they are less likely to be informative about a human value, reducing the training set by 44123 sentences. when we also remove those labeled by 0.5, which might be less clear, our final training dataset is 42210. Second, we remove the stopwords (we augmented the nltk list with 124 words, see B.1), connector words (gensim), numbers (both alphabetically and numerically written), and tokens smaller than 2. We keep hyphened words and nouns, adjectives, and adverbs. On this subset we run a phrase model to identify frequently co-occurring words. From this final vocabulary of 24172 tokens, we identify the most frequent words occurring across all sentences (see B.2). Excluding these overall common words, we search per value for the most frequent words and match the negative and positive examples based on these words.
To explore various approaches to zero-shot and few-shot prompting and compare with fine-tuning, we select a subset from the validation sample. For prompting, we selected max 600 sentences per value of which 300 were positive examples, and the other 300 were divided among 4 sets of negative examples (of which 2 were random negative examples, 1 was related negative example, and 1 was opposed negative example). If there were fewer than 300 positive examples, we selected all positive examples, and matched with an equal set of negative examples (divided across the random, opposed and related values). Since the negative examples could be labeled for values other than the vocal value, the total subset contained more than 300 positive examples for some values. In total we have .. sentences for the validation subset used for testing (see appendix A.3 and A.1. All of our models are tested on these subsamples from the original validation set.
To fine-tune the models, we used the training set to select sentences. We used the same approach as above but only for maximally 240 positive examples per value (for SL), or 20 positive examples (for ML) to reduce the computational resources needed. This way, we cap the dataset used for fine-tuning at 480 sentences. Again, we tested the models on the sub samples from the validation set. For fine-tuning Gemini, we convert the training data into a jsonl format and use the VertexAI API to initiate and run a fine-tuning job. When completed, the job returns evaluation metrics for the training data which includes the training loss, token accuracy at training step and number of predicted tokens at a training step 2. These metrics can be visualised both using an API call as well as the Vertex AI Dashboard. For ifne-tuning in OpenAI using the Davinci model, we used the 480 sentences and the labels with hyphens between (so self-direction-thought). This resulted in fewer random responses. But still there were responses such as: ’ self-direction-direction-thought’,’-thoughtominityetal’ or ’freedom-dominance-th’. After performing the fine-tuning job for both single and multi-label, we evaluate their performance on the validation set.
We present our validation set results and discuss the influence of diferent choices on model
performances: 1) open vs closed source 2) fine-tuning vs prompting using single and multi-label approaches.
As can be seen in 2, our best performing model is the open source Llama3-70b-instruct with an overall
f1-score of 0.70, 6 points higher than the best performing closed source models (gemini-1.0-pro SFT
(SL) and GPT-4o few-shot (SL). This signifies that even open-access models can deliver state of the art
performance in a task like human values detection that requires nuanced and contextual understanding
of language. In comparing fine-tuning to prompting, we first analyse it on the single label. Here, the
ifne-tuning for single label for Gemini seems to relatively match the performance of the best performing
prompting approach, which Llama3 being an exception. This highlights that creating a fine-tuned
model for single labels and aggregating them for predicting all labels might give good results. However,
prompting still seems to be the best performing approach for predicting single labels. In contrast, the
multi-label approaches seem to be the worst performing for both the fine-tuning as well as prompting.
For fine-tuning, this can be caused by the lack of suficient training data for each value for the model to
properly understand the nuances in them. For prompting, we think that this can be partly explained by
the fact that these language models do not robustly make use of information presented in long input
contexts[
We also see that compared to zero-shot, few-shot approaches lead to a slight gain in performance, indicating the usefulness of including positive and negative examples for each value in prompt design. For GPT, it seems that the more recent models are more efective with a higher F1 score for GPT4o compared to GPT3.5. We only tested this for zero-shot single label). Adding positive and negative examples to the prompt increases the performance. When adding on top of the examples, the context of the sentence (i.e. the three sentences in the text preceding the sentence that was labeled), the 2https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini-use-supervised-tuning :iitttrcegoodunhh :iiitt-frcceaoodnn iltaon isnm teeevnm :irceeaodnnm :ssrrrceeeou :iltsrreaoypn :iilttsrceaoy iiton :iltsrreoyum :iilttfsrrreeaooypnnm iilty :ilrcceeagvonn :iilltceeeeavboyddpnn :ilssrrcceeaonnm :iltssrreeaaunm :illtssrrceeeaaonm
EN llA l-feS leS itSum eodH ichA oPw oPw ceaF ceSu ceSu radT fonC onC uHm eenB eenB ivnU ivnU ivnU Submission(testset) GPT3.5fewshot(SL) ✓ 23 08 12 13 20 27 18 27 12 15 32 31 33 07 03 19 19 35 50 11 GPT-4oinformedzero-shot(ML) ✓ 25 15 10 10 18 25 18 09 24 21 30 46 33 09 15 26 15 41 55 20 valueeval24-bert-baseline-en ✓ 24 00 13 24 16 32 27 35 08 24 40 46 42 00 00 18 22 37 55 02 performance deteriorates slightly. Note that Llama3 has only been used as a zero-shot due to time constraints.
If we zoom in on the values, we see that Llama3 performs well across all values, while other generative LLMs perform worse. For instance, GPT3.5 has a much lower F1 score for values across the board, except for tradition and universalism-nature. However, Llama3 also outperforms GPT3.5 here with a very impressive F1 score of 85, respectively 91. Some values are notoriously dificult to predict, such as self-direction thought. Even Llama3 was unable to achieve a higher F1 score than .49. Surprisingly, ifne-tuning with GEMINI proved to be very successful and obtained an F1 of .57 for this value. We can hypothesize that for some values such as self-direction thought, fine-tuning leads to a better result as the model better learns the nuances in the value through suficient training examples whereas for some an efective prompt design seems to give the best results. This also highlights the importance of combining these two approaches to achieve an overall better result.
Table 2 shows that our ML model does slightly better above the baseline model on the test set C.4. As the SL predictions took a lot of time (for GPT3.5, it took about 3-4 hours per value for the single-label model, and Llama3 it took about 7-8 hours per value), we only include the results of our best performing GPT3.5 model: the few-shot single label prompt. Contrary to our expectations the prompt did not do much better than our previous multilabel prompt using GPT4o. The value self-direction-thought was not completely finished which could explain the low F1 score here, but even without this one value, we don’t see an improvement of SL-GPT3.5 over ML-GPT4o. Despite these results, the single label few shot outperformed this model in our validation subsets. The most likely reason could be that our validation subsets have very diferent distributions of values and words than the test set.
In our paper, we looked at the capabilites of open and closed source models as well as the influence of ifne-tuning and prompting with diferent single and multi-label approaches. Based on our validation set, prompting gives the best results when trying to predict human values using text, hence signifying the importance of an efective prompt design, with few-shot approaches showing slight gain in performances compared to zero shot approaches. Given the time-limitedness, further research can be focused on looking at the text similarities and diferences between test and our validation subset. We can also estimate SL prompting approaches with GPT-4o as well as run LLama3 SL on the entire test set given enough computational capacity and finally compare SL SFT for openai with ML SFT and note the gain in performance or lack thereof.
Assess which value relates to text. Follow description below in format VALUE: description. SELFDIRECTION–THOUGHT: Freedom to cultivate one’s own ideas and abilities SELF-DIRECTION–ACTION: Freedom to determine one’s own actions STIMULATION: Excitement, novelty, and change HEDONISM: Pleasure and sensuous gratification ACHIEVEMENT: Success according to social standards POWER–DOMINANCE: Power through exercising control over people POWER–RESOURCES: Power through control of material and social resources FACE: Security and power through maintaining one’s public image and avoiding humiliation SECURITY–PERSONAL: Safety in one’s immediate environment SECURITY–SOCIETAL: Safety and stability in the wider society TRADITION: Maintaining and preserving cultural, family, or religious traditions CONFORMITY–RULES: Compliance with rules, laws, and formal obligations CONFORMITY–INTERPERSONAL: Avoidance of upsetting or harming other people HUMILITY: Recognizing one’s insignificance in the larger scheme of things BENEVOLENCE–DEPENDABILITY: Being a reliable and trustworthy member of the in-group BENEVOLENCE–CARING:Devotion to the welfare of in-group members UNIVERSALISM–CONCERN: Commitment to equality, justice, and protection for all people UNIVERSALISM–NATURE: Preservation of the natural environment UNIVERSALISM–TOLERANCE: Acceptance and understanding of those who are diferent from oneself Return VALUE. If text reflects no value, return NEUTRAL.
Assess if the text relates to UNIVERSALISM–TOLERANCE: Acceptance and understanding of those who are diferent from oneself. Return 1 if it does, 0 if not.
Assess if the text relates to SELF–DIRECTION–THOUGHT: Freedom to cultivate one’s own ideas and abilities. Return 1 if it does, 0 if not. Here are some examples: Haimov explains that it is important for the child to be involved in the process, so that he understands that even if he is headed for a certain institution, sometimes it is not the right step for him. : 1 President Donald Trump says the US Supreme Court has not properly addressed mass election fraud. : 1 Stabilize eco-bonuses and support eficient district heating for upgrading and decarbonization of public and private heritage buildings.: 0 People who wanted to obtain information on the issue accelerated their research.: 0 This series of experiments is the first step in a multi-year experiment program of the Ministry of Defense (the directorate for research and development of the military and technological infrastructure AB) and the defense industries to develop a land and air laser system to deal with threats at diferent ranges at high powers.: 0
im b liito tsa loop ap em ssu bpu iag irP im
p p a R l ca te gy try iad ian lic n em eag trap ftca reev lreo try s i n n i
M m a c r a m m m on en con R cu m tse o e d v ce ro in
p t s y y ia s t y ll r s ek ic a g y U n a e to ie g re om ssu reou iton eon E e w w c n e a s p m o c l t e e s l l ea r an isa SU EU en ry in ic n ro ca re ey Irs eow issu suR ised lita rak lop itcon tcon liito rsseu rkuT p R r i U a p p
P m s b p a a r y w
e s m m p m la s u o m n c o
c t lsa re ayn ise tek ady tem ay e ea od rk le ts t n r ll l w Irs go ow ab eb cen r e p od n lly tn en ev od h llit yn e eu lte ay yp rP ed d e e r f
n e th op ito ra edn ew TAO itson reky itray good ayw rseb EU irem tego spu reap Is is N la u d m P re e T li e r o
s m e h y t e n t n l l n c a n l h o am fu fu son ig le to o u d ta bm irca ig it e n o l ry m r n a te kn ta ea th h e v e op uh em itu tah rg s yn w im A s a EU son ce g s n se sh ey tn re tse ce e r litrea reeG iteenm ltak itreaop ttsa irkuT rkuT reeem evn itren cahn revoyn tteegoh o g e o a c e d
Self-direction: thought 0.389098 Self-direction: action 0.382206 Stimulation 0.444883 Hedonism 0.548611 Achievement 0.641937 Power: dominance 0.523553 Power: resources 0.458125 Face 0.365027 Security: personal 0.590541 Security: societal 0.499145 Tradition 0.550340 Conformity: rules 0.359992 Conformity: interpersonal 0.351852 Humility 0.386602 Benevolence: caring 0.493737 Benevolence: dependability 0.354696 Universalism: concern 0.561806 Universalism: nature 0.791186 Universalism: tolerance 0.378788 Mean 0.477480
Self-direction: thought 0.475630 Self-direction: action 0.581450 Stimulation 0.586714 Hedonism 0.484685 Achievement 0.616815 Power: dominance 0.614849 Power: resources 0.506604 Face 0.401412 Security: personal 0.550000 Security: societal 0.594653 Tradition 0.709248 Conformity: rules 0.573897 Conformity: interpersonal 0.628830 Humility 0.563636 Benevolence: caring 0.399468 Benevolence: dependability 0.534250 Universalism: concern 0.473877 Universalism: nature 0.690581 Universalism: tolerance 0.753754 Mean 0.565282 Self-direction: thought 0.448520 Self-direction: action 0.620388 Stimulation 0.674399 Hedonism 0.672856 Achievement 0.600360 Power: dominance 0.716752 Power: resources 0.591579 Face 0.578354 Security: personal 0.600137 Security: societal 0.561422 Tradition 0.788847 Conformity: rules 0.730796 Conformity: interpersonal 0.670071 Humility 0.615385 Benevolence: caring 0.583410 Benevolence: dependability 0.610371 Universalism: concern 0.619124 Universalism: nature 0.819764 Universalism: tolerance 0.746444 Mean 0.644683
Self-direction: thought 0.454714 Self-direction: action 0.517328 Stimulation 0.722974 Hedonism 0.760369 Achievement 0.624210 Power: dominance 0.433450 Power: resources 0.548872 Face 0.540455 Security: personal 0.596154 Security: societal 0.585175 Tradition 0.746617 Conformity: rules 0.683777 Conformity: interpersonal 0.608245 Humility 0.573333 Benevolence: caring 0.535228 Benevolence: dependability 0.618975 Universalism: concern 0.649815 Universalism: nature 0.785372 Universalism: tolerance 0.730263 Mean 0.616596
F1 Accuracy Precision Recall N
F1 Accuracy Precision Recall N F1 Accuracy Precision
Recall N