Normative Data for 111 Compound Remote Associates Test Problems in Romanian

The Remote Associates Test (RAT) is a classical creativity test developed by Mednick and Mednick in 1967. RAT problems and their norms so far exist only in a few languages, including English, Dutch, Japanese and Italian. In this paper, we describe our process of constructing a set of Remote Associates Test problem in Romanian. 63 native speaking Romanian participants have solved this set. The set of items shows high internal consistency. Normative data pertaining to each problem is provided, together with a description of RAT problems peculiarities in Romanian.


INTRODUCTION
The Remote Associates Test is a widely used creativity test (Ansburg and Hill, 2003;Ward et al., 2008;Cai et al., 2009;Cunningham et al., 2009) by Mednick and Mednick (1971). A RAT problem consists of the following: three words are given to the participant, like SAFETY, CUSHION, POINT; the participant is asked to come up with a fourth word, which relates to all of the three given words. For example, for the problem above, PIN would be a good answer, because of the compounds SAFETY PIN, PIN CUSHION, AND PINPOINT, which relate each of the three given words to the answer.
Associative abilities are considered an important factor in creativity (Benedek et al., 2012). Various versions of the RAT exist-compound (Bowden and Jung-Beeman, 2003), functional (Worthen and Clark, 1971;Olteţeanu et al., 2019), and visual . The compound RAT is so far the most widely used.
In order to measure associative ability, the language in which the test is given is important; native speakers are much more likely to know the expressions and compound words in their language, thus to have the knowledge required to solve the test. The associative ability process can thus be measured reliably when the test is administered in the native language of the speaker.
In previous work, Remote Associates items have been both solved computationally  and also created computationally in American English . While a computational approach to item creation could also work for other languages, the manual construction of a RAT in that language is generally indicated initially, in order to first understand the peculiarities of creating the test in that language.
In order for researchers to be able to use the RAT to assess native Romanian speakers, this paper focuses on the construction and norming of a variant of the RAT in Romanian. The authors, all native Romanian speakers, have collaboratively created the items. The rest of the paper is organized as follows: the process of constructing the queries is described in section 2. The process of norming the queries, together with descriptive statistics on the results obtained, are provided in section 3. A discussion on the specific difficulties in creating RAT queries in Romanian is presented in section 4. The last section presents future work and conclusions, followed by an Appendix containing the normative data on each of the question items, and their partial translation to English.

CONSTRUCTING QUERIES
The query construction process was done in four steps. These are: 1. Direct translation attempt; 2. Adaptation of items; 3. Creation of new items (including check of expressions); 4. Initial item evaluation and clean-up.
In the rest of this section we will present our process, for the benefit of researchers that are native speakers of other languages-they might be able to use this process to create RAT repositories in their own language.

Direct Translation Attempt
We initiated the procedure of constructing queries by exploring the English and Italian stimuli provided in the normative data of Bowden and Jung-Beeman (2003) and Salvi et al. (2016). As the co-authors were familiar with both of these languages, we first explored whether some of these items would translate in RAT items in Romanian. In the following we will exemplify how some queries may or may not be directly translatable.
Let us take as an example a formal query w 1 , w 2 , and w 3 , where ans x is considered a correct answer. For this query to be possible in language L 1 , at least three expressions or compound words must exist in that language: (i) (w 1 , ans x ) or (ans x , w 1 ); (ii) (w 2 , ans x ) or (ans x , w 2 ) and (iii) (w 3 , ans x ) or (ans x , w 3 ).
For this query to be directly translatable from L 1 to another language L 2 , these expressions or compound words, either in the same or reverse order, must also exist in L 2 (though they need not have the same meaning in L 2 ). For example, the query CASELLA PRIORITARIA ELETTRONICA, answer POSTA, can be directly translated from Italian to the English query BOX PRIORITY ELECTRONIC, answer MAIL, because all three compounds MAIL BOX, PRIORITY MAIL, and ELECTRONIC MAIL are valid. The answer POSTA could also be translated more directly to the word POST, however using this word as a potential answer for the RAT does not work very well, because ELECTRONIC POST is not a compound used in English.
Most queries are not directly translatable, because one of the expressions does not exist in the second language. For example, for the above mentioned query, a direct translation in Romanian would involve the words CȂSUŢȂ (box), PRIORITARȂ (with priority), and ELECTRONICȂ (electronic) with the answer POŞTȂ (post). However, the answer word has to be an adjective in order fit with CȂSUŢȂ (i.e., the correct form is CȂSUŢȂ POŞTALȂ and not CȂSUŢȂ POŞTȂ). On the other hand, changing the word POŞTȂ to POŞTALȂ makes the other two compounds invalid. If two out of the three queries existed, we proceeded to then adapt the query to the new language.

Adaptation of Items
Adaptation of an existing query to a new language happened as follows. If when attempting to translate a RAT query from L 1 to L 2 only items (i) (w 1 , ans x ) or (ans x , w 1 ) and (ii) (w 2 , ans x ) or (ans x , w 2 ) were valid expressions or compound nouns in L 2 , but (w 3 , ans x ) or (ans x , w 3 ) did not exist, we replaced w 3 with a new word, which formed a valid expression or compound noun with ans x in L 2 .
For example, in Salvi's (Salvi et al., 2016) set of items, we encountered the item: BIANCA CREDITO IDENTITÀ, answer CARTA. In Romanian, the expressions formed with the answer word by the second and third terms exist: CARTE DE CREDIT (credit card) and CARTE DE IDENTITATE (identity card). However, for CARTA BIANCA (roughly meaning a free pass), Romanian uses the French CARTE BLANCHE. We thus had to replace w 1 BIANCA-in Romanian ALBȂ with SȂNȂTATE (health), which forms a valid expression with the answer CARD DE SȂNȂTATE (health card).
The RAT item thus obtained after adaptation was SȂNȂTATE, CREDIT, IDENTITATE, answer CARD.

Creation of New Items
In many cases, not just one of the three terms, but two of the three terms did not form a good compound. However, one of the query items and the answer could still be taken as a seed for creating new queries. We developed this technique when observing the similarity of one query word and the answer word between the two sets of items in English and Italian. For example, in Bowden's set of queries, COTTAGE, SWISS, CAKE, answer CHEESE appears. In Salvi's set of queries, we encounter item CAPRA, SVIZZERO, BUCHI, answer FORMAGGIO. The second word of the query and the answer form a compound (SWISS CHEESE-FORMAGGIO SVIZZERO) which is common to the two queries. However, the two other queries in Italian translate roughly as goat cheese and cheese with holes. In Romanian we made the query: CAPRȂ (goat), TOPITȂ (melted), BURDUF (bellows), answer: BRÂNZȂ (cheese).
While creating new RAT items, we sometimes used one query word-answer word pair, or just a query word or an answer word as a seed. That is to say, we formed a completely new query using that seed word or compound, upon finding a word we considered would have suitable other compounds in Romanian. During query creation, we used an online Romanian dictionary 1 to check whether other compounds existed for a particular word, and whether some compounds where as usual as we thought them to be.

Initial Item Evaluation and Clean Up
After the steps above, a list of 198 RAT items was established. The authors and five Romanian student volunteers rated all the items on a 1-5 Likert scale (1 meant an excellent item), and made notes on items which did not seem suitable. Thus eight ratings were obtained for each of the 198 items. The top rated items where kept, and some weaker items improved based on suggestions from the various raters.
The resulting set of items was then cleaned of query word repetitions. This was performed as not to bias participants to associate a particular query word with a particular compound, because of having encountered that query word earlier during the solving in a different potential compound. Where two queries both contained a particular query word, we kept the stronger rated query. After this process, a set of 111 RAT queries in Romanian was obtained.

NORMING QUERIES
The 111 queries thus created where given to participants in order to evaluate and norm.

Method
Participants where invited to take part in the norming via the online platform CrowdFlower 2 , a platform similar to Mechanical Turk. The call invited native level Romanian speakers alone, and restricted IP addresses of potential participants to Romania. Volunteer students from Babeş-Bolyai University, Cluj, Romania also participated. In accordance with local legislation, the study was exempt from an ethics committee approval because it only involved the gathering of normative data on a task that posed no danger to human participants. Participants approved to having their data used in anonymised form for scientific purposes as part of continuing the online data gathering process.

Participants
There were 63 participants (31 females, 32 males), aged between 18 and 70 years (3 under 20; 27 between 20 and 30 years; 19 between 30 and 40 years; 9 between 40 and 50 years; 4 between 50 and 60 years of age; and 1 between 60 and 70 years of age). With regard to their educational level, 9 of them graduated high-school, 11 were university students, 32 had a bachelor degree, 3 were graduate students, and 8 had a post graduate degree.  Four training queries where then provided, to familiarize the participant with the task. After the participant answered each of the training queries, the expected answer for the query was provided. It was explained that word x can be associated with each of the three given words: and then the three required compounds where provided. The four training queries were the following: -Training item 1: EXPRES (Express), REGIONAL (Regional) Training queries were also used to assess whether each participant understood the task. After the training queries, the 111 queries for which normative data was sought were then given in randomized order. We measured accuracy and response times. Response times were recorded using jsPsych, a JavaScript library for running behavioral experiments in a web browser 3 . The trials were administered without any breaks.

Results
The overall average accuracy to complete all the items was M = 0.54 (SD = 0.43). Response times (RT) were calculated in seconds. RTs were recorded for both correctly answered and incorrectly answered items. The average reaction time for the items solved correctly was M = 15.37 (SD = 10.53). The average reaction time for solving all the items (including the incorrect responses) was M = 22.41 (SD = 22.39). Descriptive data on accuracy and response times, together with confidence intervals, is shown in Table 1.
The accuracy and response times to each of the query items are shown in the Appendix. Accuracy per question can be seen in Figure 1.
The easiest query was item number 84: GURȂ, MINERALȂ, IZVOR, answer APȂ (Mouth, Sparkling, Spring, answer Water), with an average accuracy of 0.98. The hardest query was item number 28: FÂN, RÂU, MUNCȂ, answer BRAŢ (Hay, River Work, answer Hand/Arm) with an average accuracy of 0.02. Overall, the fastest query solved was item number 76: LUMINȂ, The scale reliability of the newly created set of items was measured using Cronbach's alpha as an internal consistency measure. Cronbach's alpha on Accuracy was 0.93; Cronbach's alpha on reponse times (on both correct and incorrect items) was 0.97.
Seventy three percent of the participants had ages between 20 and 40, 22% were aged between 40 and 70 and 4% of the participants were younger than 20 years. There were no age effects on accuracy, F (2,60) = 0.048, p = 0.953, nor on reaction times for correct responses, F (2,60) = 0.998, p = 0.374.
Gender differences were observed on the accuracy of responses, females being more accurate than males, F (1, 61) = 5.225, p = 0.026. There were no gender effects on the reaction times of the correct responses, F (1, 61) = 0.359, p = 0.551. Although the number of participants is relatively low this result is very interesting as it shows some cultural differences in the effects of gender on RAT responses.
No significant correlation between self-assessed creativity and accuracy or response times was observed.
At the request of one of our reviewers, we also provided a partial translation of the query items in English. The translations are followed by notes. The translations are of course not meant to be RAT items, but rather help give insight into language differences.

DISCUSSION
A set of 111 compound Remote Associates Test problems was created in Romanian. The high Cronbach alpha of the answers to these items, measured for both accuracy and response times, shows the scale has high internal consistency. This validates our approach to item creation.
On a general note, the intent of this manuscript is to showcase an approach to RAT items creation for languages in which the RAT does not exist. We believe that our general process of item creation, described in section 2, is applicable to most other languages too. This process can be summarized as starting from existing sets of RAT items in languages the set creators understand, followed by: (1) a direct translation attempt from the source language(s); (2) an adaptation of the items that cannot be fully translated, using the target language's expressions; (3) creation of new items in the target language and (4) initial item evaluation and clean-up. We would generally like to encourage other researchers to create sets of RAT items in their native languages, as this will provide more culturally diverse studies of creativity, and cross-cultural comparison. Such cross-cultural comparisons can further support cross-modality studies of the associative process Toivainen et al., 2019). A bigger picture of the role associative processes play in creativity can then be obtained.
After the description of this approach, we hope computational approaches can be implemented to generate items in different languages, like the one proposed by , Olteţeanu et al. (2019), and Olteţeanu et al. (2018), offering more parameter control . However, we would like to emphasize that computational creation acts best when based on direct observations of expert researchers regarding peculiarities of that language, as language differences will involve tweakings of the general process of item creation; such observations can only be made via studying or creating and validating an initial set of RAT items in that language.
One peculiarity of the Romanian language is that compounds often use prepositions (like PERIE DE  Another peculiarity is that Romanian nouns with determiners (e.g., CARTEA vs. CARTE, translated to THE BOOK vs. BOOK; see FLOAREA-SOARELUI above) cannot be used in the same format as answers because they are not appropriate in this format to other compounds. Again, respondents were not instructed when or whether to use determiners: for example, they had to come up with the answer SOARE (sun), but they had to think that in combination with FLOARE (flower) both must change to nouns with determiners, FLOAREA-SOARELUI (literally, the flower of the sun in Romanian).
These two features of compounds in Romanian language also made adapting the original queries difficult.
The easiest and fastest questions are not the same. This may be due to overt processes of checking the intuited answer fits with all query items increasing Accuracy for some participants and queries. It may also be due to the fact that easier queries to answer (queries which are answered correctly with higher frequency) include RT contributions from relatively slower solvers, while queries that are harder do not include such RTs because slower solvers may be unable to accurately solve these queries.
An interesting question which one of our reviewers brought forth was what makes an answer a correct answer. For the context of this paper, we have assessed correctness of the answer word to represent the fact that a relationship existed between the answer word and each of the query words. The uniqueness of the correct response is, however, a different matter. As one of the authors has pointed out in , the computational solver they built for the solving of the RAT in English showed that other answer were possible and correct, besides the answers rated as correct in the norms of Bowden and Jung-Beeman (2003). Thus while the Romanian queries manually created here did not evoke any oher answers to the authors, a more complete exploration of whether other correct answers are possible could only be done computationally, as part of future work.

FUTURE WORK AND CONCLUSION
In this paper, a set of 111 compound Remote Associates Test problems was developed in Romanian. This set of queries was normed using Romanian participants. These queries can now be used in experiments which involve creativity measurements for Romanian participants. A formal description of the query creation process was also provided, so that computational forms of this process can be implemented in the future.
As future work, we plan to: a) prototype a computational form of constructing Romanian RAT queries, based on our process description and on processes of computational query construction already deployed in English ; b) explore queries which are common across multiple languages with bilingual participants.

ETHICS STATEMENT
The study was exempt because it only involved the gathering of normative data on a task that posed no danger to human participants.

AUTHOR CONTRIBUTIONS
A-MO, MT, and TI contributed to design of queries in Romanian, data gathering, and data analysis.