30 March, Amsterdam, hosted by Booking.com

Machine Learning, Quality Management and MT Evaluation


Exclusive for buyers and vendors of translation services



Enterprises today face the challenge to translate evermore content in an increasing number of languages. In the last couple of years, the translation industry has seen a lot of development with the adoption of new technologies and innovative solutions and service models. More and more companies start to enhance their existing translation processes with machine translation technology and new community- and data-driven platforms. The opportunities offered by automation and innovation in translation are great, but the challenges are equally impressive. How do we assess and manage translation quality and deliver according to the user's needs? How do we evaluate the output of MT engines? And how do we harvest the data so we can provide meaningful reports on productivity and quality?


The TAUS QE Roundtable is meant to bring translation buyers and vendors together to learn from each other and to find common ground and ways to collaborate. The objective of the one-day event is to provide an open platform to the participating companies to discuss effective enterprise language methods and to pursue future trends.

Focus and Format of the Roundtable

The focus of the roundtable is on the challenges and opportunities of quality management and quality evaluation in the translation industry. It'll be a highly interactive meeting, with short presentations of use cases and experiences, followed by discussions with the participants. The discussions will be moderated by Attila Görög and Anne-Maj van der Meer (TAUS). The aim of the meeting is to deliver concrete and useful results. We plan to draft a Best Practices Guideline Report based on the topics discussed in the afternoon session.

GALA Conference 2017

The TAUS QE Roundtable takes place one day after the GALA Conference 2017 in Amsterdam. TAUS members who are not GALA members receive a $100 discount on the conference fee. To get your discount code, please contact This email address is being protected from spambots. You need JavaScript enabled to view it..

Topics and Preparation

Topics for Discussion

The preliminary overview of the agenda contains the following topics for discussion:

  1. TAUS and Quality - a brief history. Attila Görög will share the TAUS vision on translation quality and the journey that lead from DQF to the Quality Dashboard.
  2. Introductory presentations from a selection of the participating companies, outlining use cases and opportunities and challenges in automation and measuring translation quality.
  3. Presentation on how to automate quality evaluation and other manual processes in the translation industry by making use of machine learning.
  4. Two presentations on how to measure translation quality of both human and machine translation. How can we correlate edit distance and productivity? How do we go from a static quality review metric to a more dynamic model?


Participants are requested to prepare a short presentation of maximum five slides consisting of an overview of their translation organization and process and an outline of the challenges and opportunities their organization is facing in the coming year(s) and possibly a use case of machine translation or quality management. At the roundtable, each of the participating companies will give a short presentation (10 minutes).



9:00 / Welcome with coffee

9:15 / Welcome, by Booking.com

9:20 / Introductions and Agenda Overview

9:30 / TAUS and translation quality - a brief history, by Attila Görög (TAUS)

9:40 / Introductory Presentations

Each participating company makes a 10-minute presentation (including a short Q&A), profiling their translation organization and sharing their opportunities and challenges in automation and quality measurement.

10:30 / Coffee Break

11:00 / Introductory Presentations (continued)

12:30 / Lunch

14:00 / Machine Learning, by Jaap van der Meer (TAUS)

Algorithmic management is the secret behind innovative translation platforms that automate and track every task and tell us what to do. Datafication is the biggest trend of translation in the coming year(s). Everyone will have his or her own dashboard and we are dialing ourselves into the age of robots and singularity. “How long will it take before my customers find out?” In this presentation Jaap van der Meer, director of TAUS, will sketch the evolution of the translation industry and give an insight into how machines not only get better in translation but also in managing translations by constantly learning from the data we feed into the systems through our cloud connections.

Questions: Are there any use cases in the industry currently where machine learning is applied? Can we use ML for automatic error flagging? Once we have annotated data, what can we do with it and what kind of ML algorithms can we train? Should we also look outside the "translation arena" and learn from other ML applications such as Word-Sense Disambiguation, Named Entity tagging, Semantic analysis, etc?

14:30 /  Quality Management, by Dace Dzeguze (TAUS)

How a common and dynamic approach to translation quality evaluation can help to objectively measure and benchmark translation quality and productivity, reduce costs, increase efficiency and resolve friction in the supply chain.

Questions: How do you know that the quality of translation that you buy/sell is good enough, fit for purpose?How can industry benchmark contribute to better quality management at your organization?

15:00 / Evaluating Machine Translationby Maxim Khalilov (Booking.com)

For many years, evaluation of machine translation was primarily focused on the post-editing scenario, in which MT serves as a productivity increase element of a professional translation pipeline. However, in e-commerce the most desirable application of MT is direct publishing of machine translated content that dictates different requirements to the MT quality evaluation model. In this talk, Maxim Khalilov will present an enhanced quality evaluation model, called a Business Sensitivity Framework for MT, that serves to decrease the probability of critical translation errors committed by an MT system.

Questions: What are the main problems in productionalizing the business sensitivity framework  for machine translation?Which MT quality evaluation metrics are still useful given the rise of neural MT?What are best practices in evaluating a business impact of translation quality?

15:30 / Refreshment Break

16:00 / Breakout sessions

During the three sessions in the afternoon, attendees can write on a flip chart the topics or questions they would like to discuss in the breakout sessions. After the last session, they decide which breakout session they will join. Discussion leaders will coach the groups to review and analyze the questions and steer towards conclusions and recommendations.

17:00 / Report back to plenary

Each of the three groups will report back to all attendees what their conclusions are and will also make recommendations for possible collaborative industry actions.

17:15 / Closing remarks and Adjourn

17:30 / Happy Hour @ Café Schiller – closing drinks


The TAUS QE Roundtable will be hosted at Booking.com in Amsterdam. The address of the venue is:

Vijzelstraat 68
1017 HL Amsterdam

The meeting room is located on the 5th floor and is called 'Destination'. Information on how to get to this location will be made available shortly.


From Amsterdam Central Station, the easiest and fastest way will be to take tram number 16 and exit the tram at 'Keizersgracht'. From there it's just a matter of crossing the canal in front of you and walking into number 68 on your right hand. The total trip will take about 15 minutes. You can also walk from Central Station to Booking.com. You'll follow the same route as the tram and it'll take you about 25 minutes.


The registration fee is € 400. TAUS subscribers get 50% discount and pay € 200. Click here to go to the registration form.

Presentation Slides

Presentation Slides


Presentation slides are available to TAUS members only. To download the slides, please log in first.