M²Preserve: A Multidimensional Framework for Evaluating Meaning Preservation in Text Simplification

Evaluating meaning preservation in text simplification remains an open challenge: human evaluations rely on coarse-grained holistic judgments, while automatic metrics frequently misalign with human assessments and fail to capture simplification-specific edits. We introduce M²Preserve, a multidimensional fine-grained framework that decomposes meaning preservation into two complementary dimensions, Completeness and Faithfulness, defined over key facts aligned between original and simplified texts. To support evaluation, we construct M²PreserveEval, a paragraph-level dataset with 8,773 human-annotated key-fact instances and high inter-annotator agreement. With this dataset, we benchmark a wide range of automatic metrics and evaluate M²Preserve-LLM as an automated evaluator. Results show that conventional metrics are weak indicators of Completeness, whereas M²Preserve-LLM produces more structured and interpretable assessments. Further analysis reveals that simplification systems vary more in how much source content they retain than in how faithfully they render the facts they include, pointing to content selection and fact retention as central challenges in simplification.

INLG 2026