The Quality Dashboard is the location from where to get information on your projects in DQF. It provides the user with business intelligence with benchmarking reports and trends in his or her translation projects.

The goals of the Quality Dashboard are: creating a dynamic way of finding information and Business Intelligence you’re looking for (Benchmark and Trend reports), ensuring a quick overview of the to you most relevant statistics (project reports), and providing users with detailed information (downloadable reports).

Overview

Planning Overview

  1. Improving and extending project-based reports:
    1. Reviewing calculations on the current translation, correction and error annotation reports;
    2. Extending functionality of the charts on the current translation, correction and error annotation reports with drop-downs for language and vendor filters;
    3. Extend the list of projects based on user role.
  2. Adding downloadable spreadsheets to the project-based reports:
    1. Spreadsheet that summarizes the charts of the report;
    2. Spreadsheet with detailed information on segment level.
  3. Other
    1. Benchmark reports.
    2. Trend reports.
    3. Internal benchmarking by project.
    4. Data connector.

NOTE: the numbers above correspond with the "Ad" numbers in the tabs.

Ad 1.1

Reviewing calculations on the current translation, correction and error annotation reports 

Several reviews of the calculations are planned for the current reports. These reviews concern more accurate numbers as well as improvements on the projects’ workflow modeling.

Ad 1.2

Extending functionality of the charts on the current translation, correction, and error annotation reports with drop-downs for language and vendor filters

The charts in the project-based reports will have drop-down menus to filter on language or vendor. Default view is the full set of data, not taking into account language or vendor. With the drop-down filter, you can narrow down the data in the chart to that of a certain language and/or vendor.

Figure 1 - Quality Dashboard Filters

Ad 1.3

Extend the list of projects based on user role

Projects are usually not tied to one user or company only. Different users or companies play their own role in a translation workflow. Roles can consist of setting up a project, outsourcing a project, translating a project, reviewing translation or project, or being reviewed. All these actions can generate data and be turned into a report for the user/company. Users might also perform different actions in the same project. For example, a user can translate in a project and might want to see a report on his role in the translation; while on the other hand, his translation can be reviewed, and he would like to see the results of this review.

In order to see the desired report, the user should be able to choose the correct project based on the project name, but also to choose the correct report based on his or her role in this project. The roles can be divided into two categories: active and passive roles.

The active roles are outsourcing projects, translating and reviewing. The projects in which the user is involved in an active way will be presented in one list. For each task type there is a column and a report is viewable once the user had an active role in this task type. For example, a company can have outsourced a translation or have done a translation. The users of this company can select a report on the translation of this project and see how they or their vendors performed in this task. On the other hand, a company might want to see a report that tells them about the results of a review of their translation. Projects in which companies play a passive role will appear in a different list.

Figure 2 - Project List

Ad 2.1

Spreadsheet that to summarizes the charts of the report

For each of the project based reports there will be two downloadable documents. The first document contains the same information as the report’s charts, but in a spreadsheet. This means a table with all the metadata of the project, and tables that repeat the content of the charts, with a breakdown in languages and vendors.

Ad 2.2

Spreadsheet with for detailed information on segment level

The other document contains information of the project on segment level.

These are the columns in the downloadable report regarding translation:

  • CAT tool segment ID
  • DQF segment ID
  • File name
  • Source language
  • Source segment
  • Target language
  • Segment origin
  • MT/TM name
  • MT/TM version
  • Match rate
  • Original target segment
  • Translated target segment
  • Edit distance
  • Normalized edit distance
  • Editing time
  • Segment length in words
  • Vendor

Each row below these column headers contain a segment and shows the values on each of these fields. 

The same for the columns in the downloadable report regarding correction:

  • CAT tool segment ID
  • DQF segment ID
  • File name
  • Source language
  • Source segment
  • Target language
  • Segment Origin
  • MT/TM name
  • MT/TM version
  • Match rate
  • Original target segment
  • Corrected target segment
  • Edit distance
  • Normalized edit distance
  • Editing time
  • Segment length in words
  • Vendor

And for the downloadable report regarding error annotation, where each row contains an error:

  • CAT tool segment ID
  • DQF segment ID
  • File name
  • Source language
  • Source segment
  • Target language
  • Segment Origin
  • MT/TM name
  • MT/TM version
  • Match rate
  • Target segment
  • Subsegment
  • Error position start
  • Error position end
  • Error category
  • Error severity
  • Penalty points
  • Reviewer
  • Translator

The goal of these reports is to offer the user have easy access to the statistics of the reports and to store them. The segment level spreadsheet can be used to make own customized calculations.

Ad 3.1

Benchmark Reports

The benchmark reports show statistics regarding a number of metrics and show these metrics for almost any combination of filters and groupings. Benchmark reports show average values on these metrics (either or not filtered or grouped) for both the total industry, as collected in the DQF database, and the user’s own projects.

The data can be filtered or grouped according to the project’s metadata (sector, content type, customer, vendor, etc) or some of the segment’s properties (target language, segment origin, etc.).

The benchmark project gives the user a powerful tool to analyze the available data from different perspectives. By combining filters the user can find the relative contributions of factors for improvement of quality and productivity and compare their own numbers to those of the outside world.The metrics which will be included in the benchmark reports are:

  1. productivity (number of words per hour);
  2. number of (unweighted) errors per 1000 words;
  3. number of weighted errors per 1000 words and;
  4. the correction density (edit distance per 100 characters).

Below are some examples of benchmark reports. The user has the option to see the data ungrouped and unfiltered. Or he selects any arbitrary set of filters to apply to the data. The user can also select a grouping, which means that groups of filtered data will be put side by side in order to compare the impact of the filter.

Unfiltered and ungrouped data on productivity of the translations (words per hour)

 

Figure 4 - Data filtered on English-US source language (among other filters) and grouped according to a selection of segment origins

 

Figure 5 - Benchmark report on number of (unweighted) errors per 1000 words

 

Figure 6 - Benchmark report on weighted errors

Some further features:

  • Export data in the form of spreadsheets
  • Tooltips on the filter names
  • Filter and group on filetype (in addition to the filters and groupings on the screenshots)

Ad 3.2

Trend Reports 

Trend reports are a form of internal benchmarking: sets of the users’ own project are compared to other sets, over a stretch of time. Similar to the Benchmark report, this report gives the user the possibility to filter and group data. The main grouping, however, is across time.

Trend reports will follow the benchmark report when it comes to metrics. So this report, too, will use: productivity, number of annotated errors, weighted number of annotated errors, and correction density. Some extra features for the Trend reports:

  • Users can switch between column and line charts;
  • Filter and group on filetype (in addition to the filters and groupings on the screenshots).

 Figure 7 - Example Trend Report

Ad 3.3

Internal benchmarking by project

For internal benchmarking across single projects the Quality Dashboard provides  the possibility to set a single project against one or two other single projects from the same user/company or against the average of the user’s company’s projects with the same basic properties.More details for this kind or reporting are expected later.

Ad 3.4

Data connector

The ‘data connector’ is the working title for the feature that gives users the possibility to connect their own dashboard to the DQF database.  This means TAUS will provide API endpoints that return raw numbers from the database (through the API) upon request with a URL.