Apr 24, 2026
Psychometric Validation of the Training Needs Assessment Tool (TNAT-38): Comprehensive COSMIN-Aligned Structural Validity, Measurement Invariance, and Differential Item Functioning Analysis
- Eman Zaid Almaazmi1,
- Bassam Al Khameri1,
- Lama AlKhuja2,
- Khalifa Baqer3,
- Ahd Shahin1,
- Kabir Girohtra2,
- Srishti Mehrotra2,
- Mohamed Alali2,
- Nabil Zary4
- 1Community Care Department, Dubai Health;
- 2Institute of Learning, Mohammed Bin Rashid University of Medicine and Health Sciences, Dubai Health;
- 3Operations Sector, Dubai Health;
- 4Mohammed Bin Rashid University of Medicine and Health Sciences
- NeuroInk
Protocol Citation: Eman Zaid Almaazmi, Bassam Al Khameri, Lama AlKhuja, Khalifa Baqer, Ahd Shahin, Kabir Girohtra, Srishti Mehrotra, Mohamed Alali, Nabil Zary 2026. Psychometric Validation of the Training Needs Assessment Tool (TNAT-38): Comprehensive COSMIN-Aligned Structural Validity, Measurement Invariance, and Differential Item Functioning Analysis. protocols.io https://dx.doi.org/10.17504/protocols.io.x54v9qnozl3e/v1
License: This is an open access protocol distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Protocol status: Working
We use this protocol and it's working
Created: April 21, 2026
Last Modified: April 24, 2026
Protocol Integer ID: 315446
Keywords: psychometric validation, training needs assessment, confirmatory factor analysis, measurement invariance, differential item functioning, COSMIN, caregivers, psychometric validation of the training needs assessment tool, training needs across diverse caregiver type, training needs assessment tool, competency gaps critical for curriculum design, model cfa comparison, competency gap, psychometric validation, retest reliability, competency, initial cfa, structural validity, training need, aligned structural validity, comprehensive psychometric evidence, validated instrument, independent holdout cfa, burden scale, existing tools measure burden, diverse caregiver type, prospective test, tools measure burden, retest design, efa parallel analysis
Abstract
Background: No validated instrument systematically assesses training needs across diverse caregiver types. Existing tools measure burden or general needs, not competency gaps critical for curriculum design.
Objectives: Establish comprehensive psychometric evidence for the TNAT-38 (38-item dual-scale measuring importance and competency across seven domains) through COSMIN-aligned methods. Determine whether planned downstream randomized controlled trials can proceed with TNAT-38 as the primary outcome.
Methods: Cross-sectional validation with a prospective test-retest design recruiting 412 caregivers (stratified T1–T8) with n=79 completing a 14-day test-retest. Split-half design: EFA (n=206), initial CFA (n=206). Independent holdout CFA (n=100–150, Phase 2). Will assess structural validity (EFA parallel analysis, three-model CFA comparison), internal consistency (Cronbach’s α, McDonald’s ω), test-retest reliability (ICC(2,1)), multi-level measurement invariance (language, type, context), convergent/discriminant validity (objective measures, burden scales), and differential item functioning (logistic regression with Benjamini-Hochberg FDR correction, q<0.05).
Expected Outcomes: The seven-factor model demonstrates a good fit (CFI ≥ 0.95, RMSEA < 0.06). Scalar invariance achieved across language/type. DIF analysis identifies no items with substantial bias by literacy. The Go/No-Go Decision Committee determines RCT approval per pre-specified thresholds.
Guidelines
1. Primary outcomes: (a)
Seven-factor CFA structure confirmed on independent holdout sample (CFI ≥0.90, RMSEA <0.08); (b) Scalar or metric measurement invariance established across language/type/context with ΔCFI thresholds achieved; (c) DIF analysis identifies items suitable for pooled vs. stratified interpretation
2. Secondary outcomes: (a)
Internal consistency (Cronbach’s α ≥0.80, McDonald’s ω consistent); (b) Test-retest stability (ICC ≥0.75); (c) Convergent/discriminant validity evidence
3. Decision outcome: Go/No-Go determination per pre-specified scenarios (GREEN/AMBER/RED/RED-ALT); RCT approval contingent on Committee sign-off
Materials
- AF1: TNAT-38 Full Instrument (38 items, dual-scale, seven domains)
- AF2: COSMIN Risk of Bias Compliance Plan (prospective design assessment)
- AF3: CFA Model Specifications (all technical details, lavaan R code)
- AF4: IRT Analysis Protocol (GRM model specification and planned analyses)
- AF5: ISPOR 12-Step Translation and Cultural Adaptation Protocol
- AF6: Measurement Invariance Protocol (multi-group CFA)
- AF7: Pre-Specified Convergent and Discriminant Hypotheses
- AF8: Sample Size Justification (power and precision)
- AF9: Missing Data Protocol (MCAR/MAR testing and imputation)
- AF10: Statistical Code Template (9 complete R scripts)
Before start
Institutional Review Board: Data collection will not commence until formal ethics approval is received.
Informed Consent: Written informed consent in Arabic or English; explicit agreement for data use in publication; confidentiality via de-identification; GDPR-compliant data handling for any non-UAE participants.
Data Management: De-identified data; encrypted storage (AES-256 at rest, TLS 1.3 in transit); access restricted to the core research team; 5-year retention post-publication. No identifiable information in publications or supplementary materials.
Pre-Registration: All fit thresholds, decision rules, analysis procedures, and Go/No-Go criteria are pre-registered before CFA Phase 2 analysis to prevent post-hoc threshold adjustment.
Risk Level: Minimal risk — psychometric validation through survey administration; no medical intervention.
Reporting Standards: COSMIN checklist for psychometric validation; STROBE-extended checklist for observational studies.
Study Design and Sample Specification
Objective: Define study design, participant recruitment strategy, and sample composition to support structural validity and measurement invariance testing.
Procedure:
- Overall design: Cross-sectional psychometric validation study (baseline n=412) with prospective test-retest subsample (n=79 at 14±2 days)
- Setting: Six site types in UAE (long-term care facilities, private home care agencies, government healthcare, NGOs, caregiver training institutes, hospital geriatric units) in Dubai, Abu Dhabi, Sharjah
- Participant eligibility: Adults ≥18 years providing care to persons ≥60 years with ≥3 months experience; fluent in Arabic or English; excludes severe cognitive impairment
- Stratified sampling across eight caregiver types (T1–T8): Home health aide, in-home personal care attendant, hospital patient care assistant, long-term care facility nursing assistant, dementia care specialist, care coordinator/supervisor, palliative care assistant, community health worker/volunteer; target n=50–65 per type
- Target sample: N=412 (10.8:1 participant-to-item ratio for EFA/CFA adequacy)
- Test-retest subsample: N=79 (20% systematic recruitment at Month 1; 14-day interval with caregiving situation stability verification; exclude if role/employment/care recipient status changes; expected 96% retention)
Sample Size Justification:
- 10.8:1 ratio exceeds recommendations for EFA/CFA split design (minimum 5:1 per split sample)
- Split-half (n=206 per phase) enables >5:1 participant-to-estimated-CFA-parameter ratio
- Multigroup CFA across eight types: 50–65 per type provides power to detect ΔCFI ≤0.010
- Test-retest ICC 95% CI width ±0.10 with n=79
Success Criteria: N=412 recruited; ≥94% response rate; <5% missing data per item
TNAT-38 Instrument Specification
Objective: Specify TNAT-38 structure, item content, and scoring prior to analysis.
Procedure:
- Instrument design: 38 items across seven competency domains; dual-scale (Importance + Competency)
- Domains (CD1–CD7):
– CD1: Personal Care & Safety (6 items)
– CD2: Health Monitoring & Medication Management (6 items)
– CD3: Communication & Interpersonal Skills (6 items)
– CD4: Care Coordination & Navigation (5 items)
– CD5: Emotional Resilience & Self-Care (5 items)
– CD6: Cultural Competence & Ethical Awareness (4 items)
– CD7: Digital Literacy & Technology Use (6 items)
- Response scales: Importance scale (1–5: Not Important to Extremely Important); Competency scale (1–5: Not Competent to Highly Competent); Gap score (Importance − Competency, range −4 to +4)
- Prior validation: Content validity established through Delphi (S-CVI/Ave=0.94, I-CVI ≥0.78 for 36/38 items); cross-cultural adaptation planned per ISPOR 12-step guidelines (forward/back-translation, expert panel review, cognitive debriefing, pilot testing); available in Arabic and English; additional translations (Tagalog, Nepali, Somali) in progress
- Administration: Bilingual coordinators; participant-chosen language; paper or tablet-based via REDCap; ~30 minutes total (demographics 5 min, TNAT-38 15 min, CSES-10 3 min, ZBI-12 3 min)
Success Criteria: TNAT-38 cross-culturally validated; administration protocols standardized; all translations complete before holdout CFA Phase 2
Exploratory Factor Analysis
Objective: Determine whether the seven-factor structure emerges naturally from exploratory analysis.
Procedure:
- Method: Principal axis factoring with oblimin rotation (polychoric correlation matrices for ordinal Likert data)
- Factor retention: Parallel analysis comparing observed eigenvalues to 95th percentile random eigenvalues (1,000 permutations)
- Parallel analysis prediction: Expected 4-factor retention by statistical criterion; scree plot shows inflection after factor 7; 7-factor model pre-specified from expert consensus, will be tested alongside 4-factor solution
- Output: (a) Scree plot with parallel analysis threshold; (b) eigenvalues and cumulative variance explained; (c) unrotated and rotated factor matrices; (d) factor correlations; (e) factor interpretability assessment
- Data quality checks: Verify polychoric correlation matrix condition (target condition number \<20, no singularity); report Heywood cases if present
- Software: R `psych` and `EFAtools` packages
Success Criteria: Four-factor and seven-factor models both tested; factors substantively interpretable; ready for CFA confirmation in Phase 1 holdout sample
Confirmatory Factor Analysis
Objective: Confirm seven-factor structure through CFA on independent samples using pre-specified fit thresholds.
Procedure / Phase 1 (n=206, independent from EFA sample):
- Model specification: Seven correlated factors (CD1–CD7); 38 observed items; WLSMV estimation with polychoric correlations; delta parameterization; first item per domain loading = 1.0 for identification
- Three candidate models tested:
- Model 1 (Primary): Correlated 7-factor
- Model 2: Higher-order factor (7 domains load on single competency factor)
- Model 3: Bifactor (7 specific factors + general competency factor)
- Fit evaluation (pre-specified thresholds):
- RMSEA \<0.06 (good); 0.06–0.08 (acceptable); ≥0.08 (poor)
- CFI ≥0.95 (good); 0.90–0.95 (acceptable); \<0.90 (poor)
- TLI ≥0.95 (good); 0.90–0.95 (acceptable); \<0.90 (poor)
- SRMR \<0.05 (good); 0.05–0.08 (acceptable); ≥0.08 (poor)
- Model comparison: Compare 7-factor to 4-factor alternative via ΔCFI; success if 7-factor ΔCFI ≤0.05 and CFI superiority demonstrated
- Factor loadings: All items target λ ≥0.40; items \<0.40 flagged for review; cross-loadings \>0.30 documented
- Modification indices: If poor fit, examine top 5 MIs; allow correlated residuals only within domains and if theoretically justified; no cross-loadings freed without theory-informed rationale
- Sensitivity analysis: WLSMV vs. robust MLR comparison; result should show ΔCFI \<0.003
- Software: lavaan (R), Mplus (secondary replication)
Procedure — Phase 2 (n=100–150, prospective holdout sample):
- Sample recruitment: Independent consecutive recruitment from operational units NOT included in Phase 1; stratification by type (≥12–15 per T1–T8); recruitment blind to Phase 1 results
- CFA specification (identical to Phase 1 primary model):** Seven correlated factors; 38 items; WLSMV; no modifications permitted without independent replication (n≥50)
- Fit evaluation: Same pre-specified thresholds as Phase 1
- Factor loadings: Report all λ values; items \<0.40 flagged with recommendation (retain with note vs. revision vs. removal per content validity assessment)
- Alternative models (if primary model poor fit): Test 6-factor, higher-order, bifactor structures; report fit comparison table; recommend best-fit model to Decision Committee with rationale
- Cross-validation with Mplus: Conduct identical CFA in Mplus; compare lavaan output; report any ΔCFI \>0.002
Success Criteria: Phase 1 CFI ≥0.90, RMSEA \<0.08; Phase 2 independent CFA confirms fit thresholds; all factor loadings λ ≥0.40
Multi-Level Measurement Invariance Testing
Objective: Ensure TNAT-38 measures equivalent constructs across language groups, caregiver types, and healthcare contexts.
Procedure / Level 1: Language Invariance
- Stratification: Arabic vs. English vs. additional languages if translations complete (Tagalog, Nepali, Somali); minimum n=30 per language
- Multi-group CFA procedure:
- Configural invariance: Fit 7-factor model to each language group; all parameters free
- Metric invariance: Constrain factor loadings to equality across groups; compare to configural (ΔCFI ≤0.01 = invariance supported)
- Scalar invariance: Constrain loadings + intercepts to equality; compare to metric (ΔCFI ≤0.01 = invariance supported)
- Decision criterion: ΔCFI \>0.01 triggers partial invariance protocol (backward deletion of non-invariant parameters); identify specific items requiring language-specific interpretation guidance
- Reporting: Factor loadings by language; item intercepts; scalar invariance decision with 95% CIs for ΔCFI
Procedure / Level 2: Caregiver Type Invariance
- Stratification: All eight T1–T8 types (minimum n=25 per type) OR collapsed Formal vs. Informal if cell sizes insufficient; relaxed threshold ΔCFI ≤0.015 due to smaller per-group N
- Multi-group CFA: Configural → metric → scalar invariance tests; focus on whether formal-training-pathway caregivers (T1–T4, T6) interpret items differently than informal caregivers (T5, T7, T8)
- Pairwise comparisons: If eight-group model unstable, conduct key pairwise tests (e.g., T1 vs. T5, Formal vs. Informal) with Bonferroni-adjusted thresholds
- Reporting: Type × mean domain scores; non-invariant items identified; recommendation for downstream trial analysis strategy (pooled vs. stratified)
Procedure / Level 3: Healthcare Context Invariance
- Stratification: Institutional (hospital, aged care facility) vs. Community (home-based, community health center, volunteer role); minimum n=50 per group
- MGCFA: Configural and metric invariance testing (ΔCFI ≤0.01); scalar invariance secondary
- Rationale: Care setting fundamentally shapes competency practice and self-evaluation; testing ensures TNAT-38 does not systematically bias scores across contexts
- Reporting: Fit indices; factor loadings by context; interpretation guidance for downstream trials
Partial Invariance Protocol (Applied if Any Level Fails):
- Trigger: ΔCFI exceeds pre-specified threshold
- Procedure: Backward deletion of non-invariant constraints (release highest modification index first); iterate until ΔCFI ≤ threshold
- Decision rules: (a) 1–2 non-invariant items per domain (\<25%) = acceptable partial invariance; note in Results; (b) 3–4 items (25–50%) = concerning; recommend stratified analysis; (c) \>4 items (\>50%) = domain fundamentally non-invariant; recommend alternative scoring or domain removal
Success Criteria: Scalar invariance achieved for Language Level 1; Metric invariance for Type Level 2 and Context Level 3; \<10% total items non-invariant across all levels
Differential Item Functioning (DIF) Analysis
Objective: Identify items that function differently across subgroups due to measurement bias rather than true competency differences.
Procedure:
- Co-primary stratification variables:
- Literacy/Education (D1): Low (Secondary Education) vs High (Secondary+); rationale: reading complexity may disadvantage low-literacy caregivers
- Caregiver Type (D2): Formal vs. Informal (collapsed) OR all T1–T8 pairwise; rationale: formal training pathways vs. experiential learning may create different item interpretation
- DIF method — Logistic Regression (Lord's approach):
logit(P(Yij = 1)) = β₀ + β₁(Total Score) + β₂(Group) + β₃(Total Score × Group)
- Yij = binary response (high vs. low endorsement per category)
- Total Score = sum of remaining 37 items (ability control)
- Group = reference (0) vs. focal group (1)
- Interaction term detects non-uniform DIF
- Software: R `lordif` package (primary); `difR` package (secondary methods validation)
- Effect size: Report (a) Uniform DIF magnitude (β₂); (b) Non-uniform DIF (β₃); (c) Nagelkerke ΔR² between groups (effect size); (d) Mantel-Haenszel-scaled DIF (0–1.0 scale for interpretability)
- Multiple testing correction — Benjamini-Hochberg FDR:
- Total tests: 38 items × 2 primary dimensions = 76 tests (plus pairwise Type comparisons)
- FDR control at q \<0.05 (more powerful than Bonferroni; controls expected false discovery rate at 5%)
- Critical value: Largest p-value where p(i) ≤ (i/m) × q, m=total tests
- Flag all items with p ≤ critical value
- DIF classification:
- A-DIF (Artifact DIF): Item favors one group due to ability differences, not measurement bias (captured by controlling for ability); do not flag as problematic
- Meaningful DIF: Item interpretation fundamentally differs across groups (uniform or non-uniform); flag for interpretation guidance or revision
- Large DIF: Nagelkerke ΔR² \>0.13 (large effect); warrants discussion and possible revision
- Reporting: DIF results table (Item, Group, β₂, β₃, p-values, FDR-adjusted q, effect size); interpretation per DIF classification; recommendation for each flagged item (interpret with caution vs. revise vs. remove)
Success Criteria: No items with statistically significant DIF by literacy after FDR correction; ≤2 items with meaningful DIF by caregiver type (documented for context-specific interpretation)
Internal Consistency and Reliability
Objective: Establish internal consistency (Cronbach's α, McDonald's ω) and test-retest stability (ICC).
Procedure:
- Internal consistency (full sample, n=412):
- Cronbach's α per domain and total TNAT-38 (target α ≥0.80 for clinical use)
- McDonald's ω (omega-total) and omega-hierarchical (if bifactor model viable)
- Omega-subscale per domain (target ≥0.70)
- Report 95% CIs for all estimates
- Item-total correlations: Verify each item correlates with the domain score (r ≥0.30); identify items \<0.30 as potential redundancy
- Test-retest reliability (subsample, n=79 at 14 days):
- ICC(2,1) model: two-way mixed effects, absolute agreement, single rater
- Total TNAT-38 target: ICC ≥0.75
- Per-domain targets: ICC ≥0.70
- Report 95% CIs; values crossing 0.70 or 0.75 thresholds trigger careful interpretation
- Caregiving situation stability verification: Document that test-retest participants reported no major changes in role, employment, care recipient status during 14-day interval; participants reporting significant change will be excluded from test-retest analysis (expected frequency \<5% based on feasibility estimates)
- Software: R `psych`, `irr`, `semTools` packages
Success Criteria: Cronbach's α ≥0.80 total and ≥0.70 per domain; ICC(2,1) ≥0.75 total, ≥0.70 per domain; no item-domain correlations \<0.30
Convergent and Discriminant Validity
Objective: Establish relationships with objective competency measures (convergent) and distinctness from burden measures (discriminant).
Procedure:
- Convergent validity (concurrent validation):
- Administered at baseline: CSES-10 (Caregiver Self-Efficacy Scale, 10 items, range 10–40)
- Theoretical relationship: Self-efficacy (confidence) vs. Competency (capability); literature reports r=0.35–0.65 (median≈0.45)
- Hypothesis: TNAT-38 competency scores correlate with CSES-10 at r ≥0.40
- Rationale: Related but distinct constructs; moderate correlation expected
- Discriminant validity:
- Hypothesis: TNAT-38 gap scores weakly correlate with caregiver burden (ZBI-12; r \<0.30)
- Administered at baseline: ZBI-12 (12-item burden measure, range 0–48)
- Rationale: Competency gaps ≠ burden; literature shows competency-burden r=0.10–0.35
- Interpretation: If r \>0.30, investigate whether domains capturing both competency and emotional burden
- Subgroup analysis: Report convergent/discriminant validity separately by caregiver type and language group (if n sufficient); assess whether validity evidence is consistent across populations
Success Criteria: TNAT-38 competency correlates with CSES-10 at r ≥0.40 (convergent); TNAT-38 gap scores correlate with ZBI-12 at r \<0.30 (discriminant)
Go/No-Go Decision Protocol
Objective: Determine whether TNAT-38 can proceed as the primary outcome for planned downstream randomized controlled trials evaluating caregiver training interventions, based on pre-specified validation criteria.
Procedure:
- Decision Committee composition: E3 (Psychometrics Lead, Mei-Ling Chen); E6 (Trial Methodology Lead, Anders Bjork-Eriksson); Principal Investigator; Lead Statistician; 3 of 4 quorum required; 2/3 majority for approval
- SCENARIO A / GREEN (Proceed Immediately):
- CFA fit: RMSEA \<0.06, CFI ≥0.95, TLI ≥0.95, SRMR \<0.05
- Factor loadings: All λ ≥0.40; <2 items with cross-loadings >0.30
- Measurement invariance: Scalar for Language (ΔCFI ≤0.01); Scalar for Type (ΔCFI ≤0.015); Metric for Context (ΔCFI ≤0.01); \<2 items total non-invariant
- DIF: No items statistically significant by literacy (FDR q ≥0.05); ≤2 items meaningful DIF by type
- Convergent validity: TNAT-38 competency–CSES-10 r ≥0.40
- Test-retest: ICC ≥0.75 total, ≥0.70 per domain
- Decision: Approve TNAT-38 as primary outcome; proceed immediately to RCT recruitment
- SCENARIO B / AMBER (Conditional Proceed with Revision):
- CFA fit: 0.06 ≤ RMSEA ≤0.08, 0.90 ≤ CFI \<0.95, SRMR \<0.08; acceptable range but not optimal
- OR: Good fit on some indices, acceptable on 1–2 others
- Factor loadings: 1–3 items \<0.40; ≤3 items cross-loadings \>0.30
- Measurement invariance: Partial invariance requiring context-specific interpretation (3–4 items non-invariant per domain)
- Decision: Require minor item revision (wording clarification, threshold adjustment); conduct targeted re-validation (n=50–75), confirming revision efficacy before RCT launch; conditional approval with 3-month timeline extension
- SCENARIO C / RED (No-Go; Require Major Revision):
- CFA fit: RMSEA \>0.08, CFI \<0.90, or multiple indices poor fit
- Factor loadings: \>3 items \<0.40; substantial cross-loadings
- Measurement invariance: Fundamental non-invariance (\>4 items per domain non-invariant); differential interpretation across types suggesting domain restructuring needed
- DIF: Multiple items (\>3 per domain) with substantial meaningful DIF
- Decision: Do not approve as primary outcome; recommend either: (a) restructure instrument (merge/remove domains); (b) revert to secondary outcome (TNAT-38 as exploratory); (c) employ alternative primary outcome (supervisor rating, skills checklist) with RCT timeline extension for selection/validation
- SCENARIO D / RED-ALT (Alternative Pathway):
- Primary CFA acceptable, but invariance/DIF concerns suggest domain-specific rather than pooled scoring
- Decision: Approve TNAT-38 for RCTs BUT with mandatory stratified analysis by caregiver type; domain-specific scoring; sensitivity analyses comparing pooled vs. stratified results; downstream trial methods pre-register stratified approach
- Timeline for decision: All validation analyses complete; Decision Committee convenes; decision within 2 weeks; notification to downstream trial leads within 1 week
Success Criteria: GREEN or AMBER status achieved; formal sign-off from all 4 voting committee members; decision documented in Research Notes
Statistical Software and Code Sharing
Objective: Ensure reproducibility through transparent software specification and open-access code.
Procedure:
- Primary software platforms:
- R (lavaan ≥0.6-10 for CFA; psych, EFAtools for EFA; lordif, difR for DIF; irr, semTools for reliability)
- Mplus (version 8.3+ for sensitivity/alternative model testing)
- Data analysis code:
- Fully annotated R code for all steps (EFA, CFA, MGCFA, DIF)
- Reproducibility: Code runs identically on any machine with proper packages installed
- Test cases: Synthetic dataset for code validation
- Code availability:
- Upload to Open Science Framework ([osf.io/xwyk3](https://osf.io/xwyk3)) and/or Zenodo community ([zenodo.org/communities/lhs](https://zenodo.org/communities/lhs))
- Published as supplementary material with companion manuscript
- DOI and citation provided for code attribution
Success Criteria: All analysis code published with the manuscript; reproducibility verified on an independent system