Psychometric Evaluation of the Nine-Item Problematic Internet Use Questionnaire (PIUQ-9) in Nine European Samples of Internet Users

Objectives: The nine-item Problematic Internet Use Questionnaire (PIUQ-9) is a brief self-report screening instrument for problematic internet use. The main objective of the present study was to explore the psychometric properties of the PIUQ-9 among nine different language-based samples of European internet users (Italian, German, French, Polish, Turkish, Hungarian, English, and Greek).

Methods: The total sample comprised 5,593 internet users (38.1% men), aged between 18 and 87 years (M = 25.81; SD = 8.61). Via online recruitment, participants completed the PIUQ-9, the Brief Symptom Inventory (BSI) and items about time spent online.

Results: Confirmatory factor analysis demonstrated that the bifactor model with one general factor (i.e., general problem) and two-specific factors (i.e., obsession and neglect + control disorder) yielded acceptable or good fit indices in all subsamples except for one. The common variance index in the bifactor model indicated that the general problem factor explained from 57.0 to 76.5% of common variance, which supports the presence of a strong global factor. According to the multiple indicators multiple causes (MIMIC) model, psychiatric symptoms had a moderate-to-strong direct effect on the general problem factor in all subsamples, ranging from β = 0.28 to β = 0.52 supporting the construct validity of the scale. Furthermore, in a majority of the subsamples, time spent online during the weekend had considerably higher effect sizes on the general problem factor than time spent online during weekdays.

Conclusion: The present study highlights the appropriate psychometric properties of the PIUQ-9 across a number of European languages and cultures.

Introduction

Internet addiction or problematic internet use (PIU) have both been defined as an excessive and/or inappropriate use of the internet which can lead to psychological, social, academic, and/or professional difficulties among a small minority of users and which shows high comorbidity with other mental disorders (1–6). PIU has not been consensually recognized as an official disorder and it lacks a consensual definition and agreed upon diagnostic criteria despite the recent introduction of Internet Gaming Disorder in the third section of the (fifth) Diagnostic and Statistical Manual of Mental Disorders [DSM-5; (7)] and Gaming Disorder in the most recent 11th edition of the International Classification of Diseases [ICD-11; (8)]. Nevertheless, many studies have already demonstrated the extent to which PIU is similar to other addictive disorders, such as gambling disorder and psychoactive substance use disorder. Consequently, a significant number of PIU definitions and diagnostic criteria are based on these two disorders (9–12).

PIU has frequently been comorbid with psychopathology, including depressive disorders (13), ADHD (14) and/or other psychiatric disorders and symptomatology (15, 16). Research has also found that psychopathology, operationalized using the Global Severity Index, as well as dysfunctional coping strategies, are predictive of addictive internet use (17).

PIU is typically viewed as an umbrella term referring to several problematic online behaviors (18) (e.g., gaming, gambling, sexual activity, social media use, and shopping), and as such, it has been criticized for being conceptually too heterogeneous [e.g., (19, 20)]. Nevertheless, arguably such umbrella terms can still be useful for screening the general population. Such a screening can identify groups at risk, which can then be explored more closely for the specific behaviors, while avoiding using resources to groups that are not at risk of any problematic online behaviors.

Research into internet addiction has led to the development of numerous assessment tools. Moreover, the use of a large number of different screening instruments led to major inconsistencies in the assessment and prevalence estimations in PIU studies (9, 11). This condition could be improved by using similar methods and screening instruments, as well as tools with appropriate psychometric properties, in order to reliably compare results across studies. However, relatively few validation studies have been conducted (11, 21, 22). In a systematic review published in 2014, Laconi et al., identified 45 measurement instruments for PIU, of which only 17 (38%) had been evaluated more than once in terms of their psychometric properties. Therefore, they concluded that most of the existing scales for PIU require further validation work but some of them have already demonstrated promising psychometric properties relative to other scales, given their solid theoretical basis and/or their rigorous validation studies. In addition, given the global nature of PIU, measurement instruments should also be explored across different cultural and linguistic contexts (5).

The Problematic Internet Use Questionnaire [PIUQ; (23)] is one of the aforementioned screening instruments that has previously demonstrated promising psychometric properties. It was developed to comprehensively assess the main aspects of PIU and several studies have examined its psychometric properties (5, 23, 24). Factorial investigation of the original scale revealed a three-factor solution (5, 23, 24). The three factors identified in the scale were obsession, neglect, and control disorder, reflecting solid content validity. Nevertheless, a French study also explored the validity of a 12-item form and reported a four-factor solution comprising preoccupation, withdrawal, negative outcomes, and self-control (25). Internal consistency of the different forms of the PIUQ has been high in all validating studies, with Cronbach's alphas ranging between 0.77 and 0.91. Its test-retest reliability evaluated in one study was also high (23). Validity was found to be satisfactory with regards to depression [rs ranged from 0.20 to 0.43; (21, 24)], time spent online per day [r = 0.47; (21)], and anxiety [r = 0.73; (25)]. However, there is a need to further test the reliability and validity of the PIUQ in different cultural and linguistic contexts.

Given that the length of the scale is often an important issue in large-scale studies in regard to survey fatigue, brief but comprehensive screening instruments are usually preferred. For this reason, two brief versions have also been proposed for the PIUQ by the original authors including a nine-item (5) and a six-item (21) version, both of which preserved the same three-factor structure as the original 18-item scale. According to the previous validation studies, the PIUQ and its short forms, appear to be promising screening instruments, which are worth exploring further from a psychometric perspective. Based on these outlined concerns and issues, the main objective of the present study was to explore the psychometric properties of the PIUQ 9-item version (still a reasonably comprehensive scale) across nine different language-based samples of European internet users.

Material and Methods

Participants and Procedure

All participants were recruited during December 2015 and May 2016 via an online dedicated website which provided information about the study (e.g., objectives and assurances of anonymity and confidentiality). A specific Facebook page was also created and was used to recruit participants. Authors of the present paper used their own social networking profiles and the online communication networks of their university or institute to further publicize the study. This recruitment strategy resulted in considerable differences among the samples in terms of gender, age, and other socio-economic variables, as well as differing internet use patterns (see Table 1 in the Results section). More details about the recruitment procedure are presented elsewhere [see (26)]. The present study received the approval from the ethics committee of [masked for review purposes]. Participants who did not give their consent and who were under the age of 18 years were excluded (n = 76), as were participants who did not complete the socio-demographic information (n = 1,048). For participants who did not complete the PIUQ and the Brief Symptom Inventory (BSI), if <10% of the data were missing, empty cells were replaced by the mean scores of the scale (n = 1,249). The completion rate was 70%. Of the total of 7,969 participants, 2,376 participants were excluded. Consequently, the final sample included 5,593 internet users speaking nine languages (Italian, German, French, Polish, Turkish, Hungarian, English, and Greek).

TABLE 1

Table 1. Descriptive statistics of studied variables.

Measures

Basic sociodemographic information (gender, age, education, working status, the size of the residence) along with information regarding internet use were collected. Using an open question, time spent online was assessed separately for the weekdays (from Monday to Friday) and for the weekend days (Saturday and Sunday) in hours per day.

Problematic internet use was assessed using the nine-item version of the Problematic Internet Use Questionnaire [PIUQ-9; (5); see all nine language versions in the Appendix]. The original scale comprised 18 items consisting of three subscales: obsession, neglect, and control disorder (23). During the development of the short nine-item version, the authors aimed to retain the original three-factor structure assessed by three items, respectively. A 5-point Likert scale (ranging from “never” to “always/almost always”) was used to estimate how much the given statements characterized the respondents. Scores range from 9 to 45, with higher scores indicating higher risk of PIU. In the present study, internal consistency of the PIUQ-9 ranged from 0.81 (German subsample) to 0.90 (Turkish subsample).

Psychiatric symptoms were assessed using the Brief Symptom Inventory [BSI; (27)], an instrument that assesses self-reported clinically relevant psychological symptoms. The 53-item questionnaire uses a 5-point Likert scale (from 0 = “not at all” to 4 = “extremely”). The BSI comprises nine symptom dimensions: somatization, obsession compulsion, interpersonal sensitivity, depression, anxiety, hostility, phobic anxiety, paranoid ideation, and psychoticism. However, in the present study, the summarized measure—the Global Severity Index (GSI)—was applied to capture the intensity of general psychiatric distress. In the present study, internal consistency of the BSI ranged from 0.95 (German subsample) to 0.98 (Turkish subsample).

Survey questions were translated from English to the other languages using the double back-translation procedure (28). Translation was conducted by research colleagues from the collaborating countries and back translated by another expert. Back translations were then compared with the original English version and differences were discussed until a final consensus was reached.

Statistical Analysis

Confirmatory factor analysis (CFA) with maximum likelihood estimation with robust standard errors (MLR) in Mplus 8 (29) was used to estimate the degree of fit of the previously proposed three-factor model and three alternative models in all language-based subsamples. Alternative models were tested because the original three-factor model did not yield acceptable fit in all subsamples, and two of the three factors (neglect and control disorder) were highly correlated with each other. The three alternative models were the following: (i) a two-factor model where neglect and control disorder factors were merged; (ii) a bifactor model representing a global severity dimension on which each item is loaded plus three specific factors on which the items belonging to the three original factors (obsession, neglect and control disorder) were loaded (and where the correlations between specific factors were fixed at zero); (iii) a bifactor model representing a global severity dimension on which each item was loaded plus two specific factors on which the items belonging to the two newly created factors (obsession and neglect + control disorder) were loaded (where the correlations between specific factors were also fixed at zero). Missing data were treated with the full information maximum likelihood estimation method (29). The following fit indices were used in the CFA analysis: chi-square test (χ²), comparative fit index (CFI), Tucker-Lewis index (TLI), root mean square error of approximation (RMSEA) and its pclose value, and standardized root mean residual (SRMR). According to Brown and Moore (30), in practice, it is suggested that researchers report and consider each of the aforementioned fit indices because they provide different types of information [see (31) for details] and considered together, they yield a more conservative and reliable evaluation of model fit. The model can be considered satisfactory when the CFI and TLI values are higher than or close to 0.95 and still acceptable if the values are above 0.90 (31). An RMSEA value of below 0.05 indicates excellent fit, a value around 0.08 indicates adequate fit, and a value above 0.10 indicates poor fit. The pclose value is a statistical test, which evaluates the statistical deviation of the RMSEA value from 0.05, therefore non-significant pclose values (p > 0.05) indicate good model fit (31). SRMR values below 0.10 indicate a good fit. Sample size adjusted Bayesian Information Criteria (SSABIC) was also calculated to compare the competing models; lower values indicate models that are more parsimonious. Because the models are based on the same set of observed variables, sample size adjusted Bayesian Information Criteria index was used in model selection beside the parsimony and interpretability of the models.

In order to quantify the degree of unidimensionality in bifactor models, the percentage of common variance attributable to the general factor through the use of explained common variance (ECV) index (32, 33) was applied. Additionally, omega and omega hierarchical indices were used to measure how precisely a self-reported symptom scale score assesses the combination of general and specific constructs and a certain target construct (34). In the evaluation of the specific factors, we also used an index of construct replicability, or H index (35), which provides the correlation between a factor and an optimally weighted item composite (36). When H is low, the latent variable is not well-defined by the indicators and, thus, is expected to change across studies, whereas when H is high (>0.70), the latent variable is well-defined by its indicators and shows stability across studies (36).

The present study also tested measurement invariance across languages using multiple-group CFAs with a convenience feature of Mplus in one model (omnibus test of invariance) (29). In the first step, models with increasingly constrained parameters were estimated: (i) factor loadings and intercepts were freely estimated across groups (configural invariance), (ii) factor loadings were set to be equal across groups (metric invariance), and (iii) factor loadings and intercepts were set to be equal across groups (scalar invariance). When comparing the increasingly constrained models, due to the oversensitivity of the chi-square difference test (37), relative change in fit indices (i.e., ΔCFI and ΔRMSEA) were also examined. A change of ≥ −0.01 in the CFI and a change of ≥0.015 in the RMSEA indicates non-invariance (38, 39).

Finally, CFA with covariates or multiple indicators multiple causes (MIMIC) confirmatory factor analysis was carried out in all language-based subsamples to test the construct validity of the best fitting structure (the bifactor model with two specific factors) by exploring the associations between the general and specific factors and three possible predictors (i.e., psychiatric symptoms, time spent online on weekdays, and time spent online during weekend days). CFA with covariates or MIMIC can estimate the effect of indicators on latent variables at the same time when direct effects of grouping variables or other continuous variables on the latent variables are also included. The “outcome” variables of the model were the latent constructs of the bifactor model (i.e., the general factor and the two specific factors). The model was then complemented with the structural part by including a set of exogenous variables, in this case psychiatric symptoms and time spent using the internet during weekdays and weekends to investigate the effects of these variables (“causes”) on the latent constructs. Gender and age were introduced in all models as control variables.

Results

Descriptive Statistics

Descriptive statistics of the total sample and the nine subsamples are detailed in Table 1 (i.e., mean age, gender, status, educational level, mean scores of PIU, time spent online and psychopathology). The mean age of the total sample was approximately 25.8 years and varied between 23.3 years (English sample) and 32.0 years (Greek sample) for the subsamples. Approximately two-fifths of the total sample was male (38.1%). However, the subsamples differed substantially regarding gender distribution. For instance, the Hungarian sample had one single male respondent, while only 3% of the Polish sample was female. The majority of the participants were university students (64.2% of the total sample), except in the Spanish and the Greek samples, where there were more employed participants than students (57.5 and 45.5%, respectively). Time spent online also varied considerably across the samples with Turkish- and English-speaking participants spending the most time online both during the week and the weekends.

Measurement Models for the PIUQ-9

We tested four measurement models across all languages. The fit indices are reported in Table 2. The χ² test was significant in all four models across all languages. However, this test is oversensitive to large sample size. Therefore, the other fit indices (i.e., CFI, TLI, RMSEA, and SRMR) were compared in all subsamples. The original three-factor model yielded acceptable fit indices in four of the nine subsamples (Italian, German, Turkish, and Greek). The two-factor model in which the neglect and control disorder factors were merged because of the high correlation between them produced acceptable fit in two of the nine subsamples (Turkish and Greek). The bifactor model with the three specific factors had acceptable fit indices in six out of nine languages (Italian, German, Spanish, Turkish, English, and Greek). Finally, the bifactor model with two specific factors yielded acceptable or good fit indices in all subsamples except the Hungarian one. In this latter subsample, the TLI and RMSEA indices were those that did not meet the suggested thresholds. However, this model had considerably better fit than the other three models in all subsamples including the Hungarian one, where it was close to an acceptable fit. Furthermore, this model was the most parsimonious according to the SSABIC values. Consequently, the bifactor model with two specific factors was chosen as the best fitting model across the nine languages.

TABLE 2

Table 2. Confirmatory factor analysis of four measurement models of PIUQ-9.

Factor loadings for the bifactor model with two specific factors across all languages are presented in Table 3. In this model, all items loaded significantly on the global PIU factor across all languages. The factor loadings ranged from 0.19 (Item 1 in the English subsample) to 0.77 (Item 4 in the Turkish subsample). Regarding the first specific factor (obsession), Item 9 had non-significant or low loadings on its specific factor, while having strong significant loading on the main factor (the general problem factor) across all subsamples. Regarding the second specific factor that merged two of the original factors (neglect and control disorder), Items 5, 7, and 8 had non-significant or low loadings on their specific factor, while also having strong significant loadings on the general problem factor across all languages.

TABLE 3

Table 3. Standardized factor loadings of the bifactor model with two specific factors of PIUQ-9.

The explained common variance (ECV) index was estimated in the bifactor model and it was found that regarding the PIUQ-9, the general problem factor explained from 57.0% (German subsample) to 76.5% (Turkish subsample) of common variance, which supports the presence of a strong global factor. The explained common variances of specific factors are also reported in Table 4. These varied from 6.8% (Turkish subsample) to 18.6% (French subsample) in the case of the obsession subscale and from 16.7% (Turkish subsample) to 30.3% (English subsample) in the case of the second specific factor (neglect + control disorder). We also calculated omega and omega hierarchical indices to denote how precisely a self-reported symptom scale score assesses the combination of general and specific constructs, and a specific target construct. To evaluate the measurement precision of each subscale in assessing the blend of global PIU and specific factors we calculated coefficient omega; and in assessing only specific problems or only global PIU we computed coefficient omega hierarchical [for details, see (34)]. All omega and omega hierarchical coefficients are reported in Table 3. Omega coefficients varied between 76.5% (German subsample) to 95.4% (Greek subsample) in the case of the obsession specific factor and between 79.7% (French subsample) and 92.0% (Greek subsample) for the merged specific factor. In the case of the omega hierarchical coefficients, coefficients varied between 14.1% (English subsample) and 36.7% (Greek subsample) for the obsession specific factor and between 11.4% (Turkish subsample) and 29.4% (English subsample) in the case of the merged specific factor. Although there is no clear cut-off for the omega hierarchical coefficients, the present authors argue that these coefficients are salient because the specific factors explained approximately 10-35% of the PIU score. Finally, the neglect + control disorder factor showed good construct replicability because its H value was above the suggested criterion (>0.70) in seven samples, and in two additional samples these values were close to 0.70. The obsession factor, on the other hand, was characterized by weak replicability, with its value being larger than 0.70 in only two samples.

TABLE 4

Table 4. Indicators of dimensionality and reliability of the bifactor model of PIUQ-9.

Measurement invariance of the bifactor model with two specific factors across all languages were also tested. The fit indices of the configural (equal factor structure), metric (equal factor loadings), and scalar (equal factor loadings and equal intercepts) invariance are reported in Table 5. Configural invariance was supported, however, metric and scalar invariance were not. Imposing the equality constrains (metric invariance) resulted in decreasing degree of fit (Δχ² = 284.3 Δdf = 120 p < 0.001; ΔRMSEA = 0.004, ΔCFI = 0.013, ΔSRMR = 0.024). Comparison of scalar against metric invariance also resulted in decreasing degree of fit (Δχ² = 1152.3 Δdf = 48, p < 0.001; ΔRMSEA = 0.028, ΔCFI = 0.065, ΔSRMR = 0.020).

TABLE 5

Table 5. Goodness-of-fit statistics and information criteria for the estimated models relating to the PIUQ-9.

Multiple Indicator Multiple Cause (MIMIC) Model

In the second step, the structural part of the model was added to the best fitting measurement model (i.e., the bifactor model with two specific factors) and the MIMIC model was estimated in all language-based subsamples. The analysis tested the construct validity of the bifactor model by exploring the associations between the general and specific factors and three possible predictors, namely, psychiatric symptoms and time spent using the internet during weekdays and weekends. Gender and age were introduced in all models as control variables (see Table 6).

TABLE 6

Table 6. MIMIC model with standardized coefficients (gender and age were both controlled for in the models).

The influence of psychiatric symptoms and time spent using the internet during weekdays and weekends on the general problem factor and the two specific factors (obsession and neglect + control disorder) was estimated simultaneously via standardized partial regression coefficients in all nine subsamples. Overall, psychiatric symptoms had a moderate-to-strong direct effect on the general problem factor in all subsamples, ranging from β = 0.28 (German subsample) to β = 0.52 (Greek subsample). Results regarding the effect of time spent online during weekdays and at the weekend were not entirely consistent across the subsamples. However, in six out of nine subsamples, results pointed in the same direction. More specifically, beta coefficients were significantly higher for time spent on weekend days than time spent on weekdays in three samples (Italian, French, and Hungarian), higher without reaching statistical significance in another three samples (Polish, English, and Greek), and fairly similar (German and Spanish) or having an opposite direction (Turkish) in the remaining samples, according to the Wald-test. Nevertheless, the degree of the association of time spent online with the general problem factor was low or moderate, or even negative, ranging from −0.14 to 0.32 across the samples. R-squares varied between 13% (German subsample) and 43% (Greek subsample) for the general problem factor.

The effect of psychiatric symptoms on the two specific factors was inconsistent across the nine subsamples. In the majority of the subsamples it was low, non-significant, or even negative. However, in a few subsamples (such as the German, French and Polish subsamples), it was significant with a low effect size. Furthermore, in the majority of the samples, the effect of psychiatric symptoms was fairly similar for the two specific factors. However, in a few subsamples with lower samples sizes (e.g., English and Greek subsamples) the effects were in the opposite direction. The effects of time spent online during weekdays and at the weekend on the two specific factors were low and non-significant in the majority of the subsamples (except for the Spanish and Turkish subsamples, where time spent online during weekdays had a low significant effect on the neglect + control disorder factor). R-squares varied between 0.01 and 0.08 for the obsession specific factor and from 0.03 to 0.16 for the merged specific factor.

Discussion

The present study tested the psychometric properties of the PIUQ-9 across nine different language-based subsamples and tested four alternative models. These were (i) the original three-factor structure (factors: obsession, neglect, and control disorder), (ii) a two-factor model [factors: obsession and neglect + control disorder (merged due to high correlation among them)], (iii) a bifactor model with a general factor and three specific factors (obsession, neglect and control disorder), and (iv) and another bifactor model with a general factor and two specific factors [obsession and neglect + control disorder (merged due to high correlation among them)]. The results showed that the bifactor model with one general factor and two-specific factors was superior to the other three models and it had an acceptable or good fit in eight out of nine subsamples.

The bifactor model suggests that an overall problem dimension (general problem) exists explaining from 57.0% (German subsample) to 76.5% (Turkish subsample) of the variance in PIUQ-9 across subsamples, while there are two specific factors capturing two qualitatively distinct facets of the problem: obsession and neglect + control disorder. However, according to the results, only the neglect + control disorder factor contained additional information to the general factor, because the proportion of explained variance of the scale score attributed to the obsession factor was low. The neglect + control disorder factor reflects the negative consequences of PIU for the individual's life, being unable to stop using the internet, and neglecting non-internet activities. This is congruent with the DSM-5 formulation of Internet Gaming Disorder and previous descriptions of PIU from a cognitive-behavioral perspective [e.g., (40)]. Furthermore, indicators of dimensionality and reliability, as well as factor loadings, were fairly similar across the nine language-based samples, suggesting some kind of cross-validity of the scale and the proposed bifactor model with two specific factors.

Psychiatric symptoms had moderate-to-strong direct effects on the general problem factor in all subsamples. This is in line with previous findings suggesting psychopathology as measured via the Global Severity Index predicts symptoms of internet addiction (17). Time spent online mostly had low or moderate direct effect on the general problem factor across the nine subsamples. This is also in line with previous findings suggesting that time spent on specific activities (e.g., online gaming, social media use, etc.) in itself is a weak predictor of problematic use (9, 41–43). Overall, looking across all nine samples, the relationship with time spent online appears to be significant in majority of the samples, but lower than in a previous validation study (21). Another interesting finding was that time spent online during the weekend had a significantly higher effect size on the general problem factor than time spent online during weekdays in three of the subsamples, and higher effect size (without reaching statistical significance) in another three subsamples.

Previous studies mostly assessed time spent online with a single question measuring the hours spent online for an entire week or an average day, instead of differentiating between weekdays and the weekends [e.g., (25, 44, 45)]. Nevertheless, it appears that spending time online in these periods of the week is differently related to PIU. Given that the present sample is a general population sample and not a clinical one, this appears reasonable. While treatment-seeking patients in clinical settings often spend most of their time online neglecting work-related and other duties during the week (46), individuals in the general population most likely spend great amounts of time online during the weekend when they have less pressing tasks to do (e.g., household chores rather than tasks related to their occupation and/or education). Therefore, in this population it is reasonable to assume that problems arise from neglecting tasks and hobbies in the individuals' spare time. In most of the subsamples in the present study, PIU was significantly related to internet use on weekend days, while less significant results were found for weekdays. This is in line with previous studies differentiating time spent online during The week and at weekends. Time spent online during both periods was predictive of PIU (47, 48), with higher results for time spent online during weekends and a stronger association with addiction [e.g., (48–50)].

Limitations

The present study has several limitations that should be noted when interpreting the findings. Causal relationships cannot be established given the cross-sectional design of the present study. The subsamples were self-selected and not representative of the general populations in those countries. Furthermore, sampling methods were slightly different across the nine samples, which may also explain the heterogeneity in terms of age, gender and/or other socio-demographic variables, and is a strong limitation for the replicability of their findings. The data were collected using self-report questionnaires, which are known for producing potential biases (e.g., social desirability biases, short-term recall biases, etc.). Furthermore, the PIUQ-9 was the only measure assessing problematic internet use administered in this study. We decided to limit the number of scales and items as much as possible to increase the likelihood of obtaining larger samples and better quality data. However, the lack of other PIU measures prevented the testing of the incremental validity of the scale. It is also worth noting that the present study used a general population sample, most probably with a low proportion of truly problematic cases concerning internet use. Studies using clinical samples have different results [e.g., (51)]. Finally, the present study was unable provide support for the measurement invariance across different languages in the cross-cultural samples used. This may be explained by the large heterogeneity of samples in terms of gender, age, and other socio-economic variables, as well as daily internet use patterns. Consequently, further studies should use comparable samples to test the measurement invariance of the PIUQ-9.

Implications

The findings of the present study are important in terms of prevention because they suggest that specific prevention programs should target population groups that show an increased degree of risk related to their sociodemographic, interpersonal (52), and cultural characteristics. The present study using the PIUQ-9 allowed for an understanding of the importance of psychopathological symptoms and time spent online particularly during the weekends in relation with PIU. The relationship between PIU and time spent online in the nine subsamples indicates that stimulus control strategies [e.g., (53)] that limit the person's access to the internet may be useful in dealing with PIU. Moreover, the present study supports that the PIUQ-9 is a short instrument with good psychometric properties across a number of different languages and cultures that can be used in educational and clinical settings as a screening tool to discriminate those at risk of PIU from those not at risk.

Conclusion

The main objective of the present study was to assess the psychometric properties of the PIUQ-9 across samples from nine different European countries. According to the results, the bifactor model with one general problem factor and two specific factors (i.e., obsession and neglect + control disorder) yielded acceptable or good fit indices in almost all subsamples. Furthermore, psychiatric symptoms had a moderate-to-strong direct effect on the general problem factor in all subsamples, supporting the construct validity of the scale. In a majority of the subsamples, time spent online at the weekend had considerably higher effect sizes on the general problem factor than time spent online during weekdays. The present study highlights the appropriate psychometric properties of the PIUQ-9 across a number of European languages and cultures. Given its brevity, it is especially useful in large-scale studies in which the length of the scale (i.e., survey fatigue) can be an important issue in reducing attrition and increasing the response rate.

Data Availability

The datasets generated for this study are available on request to the corresponding author.

Author Contributions

SL, KK-C, DK, AG, IS, AB, FJ, JG, MG-G, NO, KS, GF, and ZD conceived and designed the study. SL, KK-C, DK, AG, IS, AB, FJ, JG, MG-G, NO, KS, GF, ZD, and OK were responsible for data collection. Data analyses were performed by RU and OK with special assistance from SL. OK, RU, MG, and SL wrote the manuscript. All authors provided critical revision to its further development, read and approved the final manuscript.

Funding

The study was supported by the Hungarian National Research, Development and Innovation Office (Grant numbers: K111938, KKP126835) and the COST Action (grant number: CA16207) funded by the Horizon 2020 Framework Programme of the European Union. This work was completed in the ELTE Institutional Excellence Program (783-3/2018/FEKUTSRAT) supported by the Hungarian Ministry of Human Capacities (OK and RU).

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2019.00136/full#supplementary-material

References

1. Aboujaoude E. The Internet's effect on personality traits: an important casualty of the “Internet addiction” paradigm. J Behav Addictions. (2016) 6:1–4. doi: 10.1556/2006.6.2017.009

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Beard KW, Wolf EM. Modification in the proposed diagnostic criteria for Internet addiction. CyberPsychol Behav. (2001) 4:377–83. doi: 10.1089/109493101300210286

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Chou W-J, Huang M-F, Chang Y-P, Chen Y-M, Hu H-F, Yen C-F. Social skills deficits and their association with Internet addiction and activities in adolescents with attention-deficit/hyperactivity disorder. J Behav Addictions. (2016) 6:42–50. doi: 10.1556/2006.6.2017.005

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Kim D, Lee D, Lee J, Namkoong K, Jung Y-C. Association between childhood and adult attention deficit hyperactivity disorder symptoms in Korean young adults with Internet addiction. J Behav Addictions. (2017) 6:345–53. doi: 10.1556/2006.6.2017.044

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Koronczai B, Urban R, Kökönyei G, Paksi B, Papp K, Kun B, et al. Confirmation of the three-factor model of problematic internet use on off-line adolescent and adult samples. Cyberpsychol Behav Soc Network. (2011) 14:657–64. doi: 10.1089/cyber.2010.0345

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Stavropoulos V, Gomez R, Steen E, Beard C, Liew L, Griffiths MD. The longitudinal association between anxiety and Internet addiction in adolescence: the moderating effect of classroom extraversion. J Behav Addictions. (2017) 6:237–47. doi: 10.1556/2006.6.2017.026

PubMed Abstract | CrossRef Full Text | Google Scholar

7. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 5th ed. Washington, DC: American Psychiatric Association (2013).

8. World Health Organization. ICD-11 for Mortality and Morbidity Statistics. (2018). Available online at: https://icd.who.int/browse11/l-m/en (Accessed September 20, 2018).

9. Király O, Nagygyörgy K, Koronczai B, Griffiths MD, Demetrovics Z. Assessment of problematic internet use and online video gaming. In: Aboujaoude E, Starcevic V, editors. Mental Health in the Digital Age: Grave Dangers, Great Promise. Oxford: Oxford University Press (2015). p. 46–68.

Google Scholar

10. Kuss DJ, Griffiths MD, Karila L, Billieux J. Internet addiction: a systematic review of epidemiological research for the last decade. Curr Pharmaceut Des. (2014) 20:4026–52. doi: 10.2174/13816128113199990617

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Laconi S, Rodgers RF, Chabrol H. The measurement of Internet addiction: a critical review of existing scales and their psychometric properties. Computers Hum Behav. (2014) 41:190–202. doi: 10.1016/j.chb.2014.09.026

CrossRef Full Text | Google Scholar

12. Weinstein A, Lejoyeux M. Internet addiction or excessive Internet use. Am J Drug Alcohol Abuse. (2010) 36:277–83. doi: 10.3109/00952990.2010.491880

CrossRef Full Text | Google Scholar

13. Liang L, Zhou D, Yuan C, Shao A, Bian Y. Gender differences in the relationship between internet addiction and depression: a cross-lagged study in Chinese adolescents. Computers Hum Behav. (2016) 63:463–70. doi: 10.1016/j.chb.2016.04.043

CrossRef Full Text | Google Scholar

14. Gunes H, Tanidir C, Adaletli H, Kilicoglu AG, Mutlu C, Bahali MK, et al. Oppositional defiant disorder/conduct disorder co-occurrence increases the risk of Internet addiction in adolescents with attention-deficit hyperactivity disorder. J Behav Addictions. (2018) 7:284–91. doi: 10.1556/2006.7.2018.46

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Chamberlain SR, Ioannidis K, Grant JE. The impact of comorbid impulsive/compulsive disorders in problematic Internet use. J Behav Addictions. (2018) 7:269–75. doi: 10.1556/2006.7.2018.30

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Yen J-Y, Ko C-H, Yen C-F, Chen S-H, Chung W-L, Chen C-C. Psychiatric symptoms in adolescents with Internet addiction: comparison with substance use. Psychiatr Clin Neurosci. (2008) 62:9–16. doi: 10.1111/j.1440-1819.2007.01770.x

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Kuss DJ, Dunn TJ, Wölfling K, Müller KW, Hędzelek M, Marcinkowski J. Excessive Internet use and psychopathology: the role of coping. Clin Neuropsychiatr. (2017) 14:73–81. Retrieved from: http://irep.ntu.ac.uk/id/eprint/28364/1/PubSub5893A_Kuss.pdf

Google Scholar

18. Fineberg NA, Demetrovics Z, Stein DJ, Ioannidis K, Potenza MN, Grünblatt E, et al. Manifesto for a European Research Network into problematic usage of the internet. Eur Neuropsychopharmacol. (2018) 11:1232–46. doi: 10.1016/j.euroneuro.2018.08.004

CrossRef Full Text | Google Scholar

19. Griffiths MD. Internet addiction—Time to be taken seriously? Addiction Res. (2000) 8:413–8. doi: 10.3109/16066350009005587

CrossRef Full Text | Google Scholar

20. Starcevic V. Is internet addiction a useful concept? Aust N Z J Psychiatr. (2013) 47:16–9. doi: 10.1177/0004867412461693

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Demetrovics Z, Király O, Koronczai B, Griffiths MD, Nagygyörgy K, Elekes Z, et al. Psychometric properties of the Problematic Internet Use Questionnaire Short-Form (PIUQ-SF-6) in a nationally representative sample of adolescents. PLoS ONE. (2016) 11:e0159409. doi: 10.1371/journal.pone.0159409

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Koronczai B, Kökönyei G, Urbán R, Király O, Nagygyörgy K, Felvinczi K, et al. Confirmation of the Chinese version of the problematic internet use questionnaire short form (PIUQ-SF). Int J Ment Health Addiction. (2017) 15:191–7. doi: 10.1007/s11469-016-9664-4

CrossRef Full Text | Google Scholar

23. Demetrovics Z, Szeredi B, Rozsa S. The three-factor model of Internet addiction: the development of the Problematic Internet Use Questionnaire. Behav Res Methods. (2008) 40:563–74. doi: 10.3758/BRM.40.2.563

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Kelley KJ, Gruber EM. Psychometric properties of the Problematic Internet Use Questionnaire. Computers Hum Behav. (2010) 26:1838–45. doi: 10.1016/j.chb.2010.07.018

CrossRef Full Text | Google Scholar

25. Kern L, Acier D. Adaptation française de l'échelle Problematic Internet Use Questionnaire. Evol Psychiatr. (2013) 78:357–71. doi: 10.1016/j.evopsy.2012.07.002

CrossRef Full Text | Google Scholar

26. Laconi S, Kaliszewska-Czeremska K, Gnisci A, Sergi I, Barke A, Jeromin F, et al. Cross-cultural study of problematic Internet use in nine European countries. Computers Hum Behav. (2018) 84:430–40. doi: 10.1016/j.chb.2018.03.020

CrossRef Full Text | Google Scholar

27. Derogatis LR. BSI Brief Symptom Inventory. Administration, Scoring, and Procedures Manual. 4th ed. Minneapolis, MN: National Computer Systems (1993).

Google Scholar

28. Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine. (2000) 25:3186–91. doi: 10.1097/00007632-200012150-00014

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Muthén LK, Muthén BO. Mplus User's Guide. 8th ed. Los Angeles, CA: Muthén and Muthén (1998–2017).

30. Brown TA, Moore MT. Confirmatory factor analysis. In: Hoyle RH, editor. Handbook of Structural Equation Modeling. New York: NY: Guilford Press (2012). p. 361–379.

Google Scholar

31. Brown TA. Confirmatory Factor Analysis for Applied Research. New York, NY: Guilford (2006).

Google Scholar

32. Ten Berge JMF, Socan G. The greatest lower bound to the reliability of a test and the hypothesis of unidimensionality. Psychometrika. (2004) 69:613–25. doi: 10.1007/bf02289858

CrossRef Full Text | Google Scholar

33. Bentler PM. Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika. (2009) 74:137–43. doi: 10.1007/s11336-008-9100-1

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Brunner M, Nagy G, Wilhelm O. A tutorial on hierarchically structured constructs. J Personality. (2012) 80:796–846. doi: 10.1111/j.1467-6494.2011.00749.x

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Hancock GR, Mueller RO. Rethinking construct reliability within latent variable systems. In: Cudeck R, du Toit S, Sorbom D, editors. Structural Equation Modeling: Present and Future—A Festschrift in Honor of Karl Joreskog. Lincolnwood, IL: Scientific Software International (2001). p. 195–216.

Google Scholar

36. Rodriguez A, Reise SP, Haviland MG. Applying bifactor statistical indices in the evaluation of psychological measures. J Personality Assessment. (2016) 98:223–37. doi: 10.1080/00223891.2015.1089249

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Marsh HW, Hau K-T, Grayson D. Goodness of fit evaluation. In: Maydeu-Olivares A, McArdle J, editors. Contemporary Psychometrics. Mahwah NJ: Erlbaum (2005). p. 275–340.

Google Scholar

38. Chen FF. Sensitivity of goodness of fit indexes to lack of measurement invariance. Struct Equation Model. (2007) 14:464–504. doi: 10.1080/10705510701301834

CrossRef Full Text | Google Scholar

39. Cheung GW, Rensvold RB. Evaluating goodness-of-fit indexes for testing measurement invariance. Struct Equation Model. (2002) 9:233–55. doi: 10.1207/S15328007SEM0902_5

CrossRef Full Text | Google Scholar

40. Gámez-Guadix M, Calvete E, Orue I, Las Hayas C. Problematic Internet use and problematic alcohol use from the cognitive–behavioral model: a longitudinal study among adolescents. Addictive Behav. (2015) 40:109–14. doi: 10.1016/j.addbeh.2014.09.009

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Griffiths MD. The role of context in online gaming excess and addiction: some case study evidence. Int J Ment Health Addiction. (2010) 8:119–25. doi: 10.1007/s11469-009-9229-x

CrossRef Full Text | Google Scholar

42. Kuss DJ, Louws J, Wiers RWW. Online gaming addiction? Motives predict addictive play behavior in Massively Multiplayer Online Role-Playing Games. Cyberpsychol Behav Soc Network. (2012) 15:480–5. doi: 10.1089/cyber.2012.0034

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Laconi S, Andréoletti A, Chauchard E, Rodgers RF, Chabrol H. Problematic Internet use, time spent online and personality traits. L'Encéphale. (2016) 42:214–8. doi: 10.1016/j.encep.2015.12.017

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Durkee T, Kaess M, Carli V, Parzer P, Wasserman C, Floderus B, et al. Prevalence of pathological internet use among adolescents in Europe: demographic and social factors. Addiction. (2012) 107:2210–22. doi: 10.1111/j.1360-0443.2012.03946.x

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Laconi S, Tricard N, Chabrol H. Differences between specific and generalized problematic Internet uses according to gender, age, time spent online and psychopathological symptoms. Computers Hum Behav. (2015) 48:236–44. doi: 10.1016/j.chb.2015.02.006

CrossRef Full Text | Google Scholar

46. Kuss DJ, Griffiths MD. Internet Addiction in Psychotherapy. London: Palgrave (2015).

Google Scholar

47. Macur M, Pontes HM. Individual differences and the development of internet addiction: a nationally representative study. In: Bozoglan B, editor. Psychological, Social, and Cultural Aspects of Internet Addiction (Hershey, PA: IGI Global) (2018), p. 221–235.

Google Scholar

48. Xu J, Shen L-X, Yan C-H, Hu H, Yang F, Wang L, et al. Personal characteristics related to the risk of adolescent internet addiction: a survey in Shanghai, China. BMC Public Health. (2012) 12:1106. doi: 10.1186/1471-2458-12-1106

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Kuss DJ, Griffiths MD, Binder JF. Internet addiction in students: prevalence and risk factors. Computers Hum Behav. (2013) 29:959–66. doi: 10.1016/j.chb.2012.12.024

CrossRef Full Text | Google Scholar

50. Li Q, Guo X, Zhao X, Bai X. Weekdays or weekends: exploring the relationships between microblog posting patterns and addiction. In: Thirty Fourth International Conference on Information Systems (Milan) (2013).

PubMed Abstract | Google Scholar

51. Kuss DJ, Lopez-Fernandez O. Internet addiction and problematic use: a systematic review of clinical research. World J Psychiatr. (2016) 6:143–76. doi: 10.5498/wjp.v6.i1.143

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Vondrackova P, Gabrhelik R. Prevention of Internet addiction: a systematic review. J Behav Addictions. (2016) 5:568–79. doi: 10.1556/2006.5.2016.085

PubMed Abstract | CrossRef Full Text | Google Scholar

53. Miltenberger RG. Behavior Modification: Principles and Procedures. Wadsworth: Cengage Learning (2011).

Google Scholar