EPIC Guide

• Generic QE Models: A QE model with broad coverage and low specificity, trained on diverse multilingual data to perform well across general content types. This is the default model used for the EPIC free trial and standard plans.

• Specialized QE Models: A QE model with moderate specificity, fine-tuned for particular domains or language varieties to improve accuracy where generic models fall short. TAUS develops and distributes this kind of model together with a partner providing the necessary training data, under a partnership agreement granting shared revenues. We currently have the EN -> CA Fr specialized model.

• Custom QE Models: A QE model with high specificity, trained with customer-provided data to reflect unique terminology, tone, and style preferences. TAUS develops a custom model for any interested party providing the necessary training data. The custom model is owned by the party requesting it.

Contact us for more information on specialized or custom models.

To train a custom model, we need reviewed and approved translations for at least one language pair and one domain (e.g. a specific customer or product line with consistent style and terminology). Ideally, these include both the original MT output and the final post-edited versions to show how translations are corrected.

We also welcome glossaries or term lists (if they’ve been consistently followed in the data) and a short description of your translation workflow (MT usage, post-editing, review process, and any guidelines used).

RECOMMENDED MINIMUM

A minimum of 25,000 segments is recommended, but more (up to hundreds or thousands) is of course always better.

ACCEPTED FORMATS

We accept TMX, XLIFF, CSV, or other structured formats.

To properly assess the performance of your customized QE model, follow these steps:

1. Create a Gold Standard Test Set

Select at least 500 real-world translation segments from your target domain and language pair. Make sure the test set includes a diverse mix of segments, from accurate translations to those with clear errors. Each segment should be manually labeled with a quantifiable human quality score, such as a scale from 0 to 5, where 5 indicates a perfect translation and 0 indicates a completely unusable one. These human labels serve as the reference (gold standard) for evaluating the QE models.

2. Score the Test Set with Both (Generic and Customized) QE Models

Run the test set through both:

• The Generic QE model (as a baseline)

• Your Customized QE model

The goal is to highlight where the custom model improves scoring accuracy for your specific content.

3. Compare with Human Judgments

Evaluate how well each model's scores align with the human-assigned labels. Use standard correlation metrics like Pearson or Spearman to quantify this alignment. A higher correlation indicates that the model is accurately reflecting human expectations of quality.

4. Review Disagreements

Identify segments where the model score differs significantly from the human label. Have a human reviewer reassess these cases to:

• Correct possible labeling inconsistencies

• Understand limitations or blind spots in the model

This helps refine the model and builds confidence in its reliability. By following this process, you ensure that model evaluation is:

• Grounded in real data

• Repeatable across updates

• Validated by human insight where needed

TAUS NLP team carefully checks and benchmarks any update on the QE models. The procedures for benchmarking are described in the 'TAUS Quality Estimation Benchmarking Report', available for download here.