Non-live subtitles personalization for VoD over HbbTV devices

by Rafael Bermúdez and Jordi Fabregat

This article describes the approach taken by a broadcaster to achieve the commitment of enable non-live subtitles for video on demand on HbbTV devices, and carry out an advanced implementation of subtitling EBU-TT standard (ref. EBU Tech 3350). The key concept is to map EBU-TT XML elements into HTML layer and CSS styles. The logic is controlled from Javascript language. Regrettably, we can’t reach 100% of available market due low performance experienced on 16% of devices. Inside, reader could find explanied several challenges and solutions for structure information and map to HTML and CSS the normative attributes, and reach the necessary accuracy to synchronizing subtitles with video. The approach taken is a time-based instead of frame-based. Last, once reached functional version, the article concludes describing subtitle personalization functionalities. The customizable parameters are change the background color, customize the font size or modify a top or bottom subtitle position.

Non-live subtitles personalization for VoD over hbbtv devices 0

Timeline

At this stage, the objective was be able to load the subtitles and render them. All associated with the video at the precise viewing moment, respecting the begin and end times interpreted from EBU-TT source file.

Non-live subtitles personalization for VoD over hbbtv devices 1

Load and parse

The following steps are carried out when user decides to enable subtitling:

  • First, load EBU-TT subtitles source file.
  • Then, parse the xml file. Through this, all nodes are extracted.
  • Last, alignment by time. The system prepares a virtual timeline containing a necessary data ordered from begin-end time references.

Video Synchronization

This topic is the most difficult part, due to limitations of some HbbTV devices. Theoretically, to synchronize subtitles with the video, the software in charge should check in each encoded video frame if a subtitle is scheduled and show or hide texts as required.

Non-live subtitles personalization for VoD over hbbtv devices 2

Performance

Seeing this, the approach requires a high-processing capacity at hardware system level, forcing to render and remove subtitles to the nearest  begin-end times, in less than 40 ms per video frame. Conversely, it could cause undesired collisions as it’s shown in the following picture.

Non-live subtitles personalization for VoD over hbbtv devices 3

In order to reduce collisions (coloured in yellow), we had to rearrange strategy. A time-based approach instead of frame-based one.

Given a sample rate, the system must check if there is any subtitle candidate before next check point. For example, considering a 500ms sample rate, check if in next 500ms window exists any candidate point. As result, devices have more time to perform the necessary operations, while minimizing temporal display error.

Non-live subtitles personalization for VoD over hbbtv devices 4

Accuracy

HbbTV video object did not have video-frames or time-seconds change events to subscribe. Therefore, the way to implement video sampling rate is use the W3C APIs, by means of window.setTimeout or window.setInterval methods, which are highly inaccurate, especially working at high frequency rates. The lack of accuracy may cause errors on subtitles rendering because in the virtual Timeline there are opaque slots, not evaluated.

Non-live subtitles personalization for VoD over hbbtv devices 5

At this point, the implemented solution cannot guarantee a full timeline analysis due to asynchronous nature of javascript. As result, some subtitles might never be showed or hidden.

Here a new criteria was added to the system, to enlarge the analysis window at double of sample frequency rate, to avoid the loss of information.

 

Non-live subtitles personalization for VoD over hbbtv devices 6

Minimum requirements

The developed approach could be seen as conservative, but still exists a percentage of HbbTV devices on market without the enough capability. Regrettably on those devices subtitles service is not available to prevent a unusual user experience.

The threshold to determine the minimum necessary requirements is set at 50 milliseconds. The system evaluates the process-time required by a JavaScript engine to carry out a complete “stack iteration”, and devices that exceed threshold are discarded.

Non-live subtitles personalization for VoD over hbbtv devices 7

According to our own statistics around 16% of devices HbbTV enabled have a low performance profile.

Styling and layout

This phase implies upgrade previous parse methods, and take into account the ‘region’ and ‘style’ nodes during subtitle text rendering.

Non-live subtitles personalization for VoD over hbbtv devices 8

Regions

A tt:region element defines a space or an area in which subtitle content is to be placed. It specifies a set of layout information through attributes.
Non-live subtitles personalization for VoD over hbbtv devices 9

HTML Layer structure

To achieve an advanced implementation of EBU-TT, all possible scenarios must be covered. The solution is designed based on the following HTML div’s layer structure, the starting point is ‘region’ layer and their attributes.
Non-live subtitles personalization for VoD over hbbtv devices 10
All normative attributes defined by region has been translated into CSS properties. The table below shows the agreed relationship.

EBU-TT property CSS property HTML layer
tts:origin top and left Subtitle line
tts:extent width and height Subtitle line
tts:displayAlign Display Align Display Align
tts:padding padding Region
tts:showBackground background-color Region
tts:overflow overflow Subtitle line

Styles

As with regions, EBU-TT-D style attributes has been mapped into CSS properties in order to fit the desired appearance.

EBU-TT property CSS property HTML layer
tts:fontFamily font-family Subtitle line
tts:fontSize font-size Subtitle line
tts:lineHeight line-height Subtitle line
tts:textAlign text-align Subtitle line
tts:color color Subtitle line
tts:backgroundColor background-color Subtitle line
tts:fontStyle font-style Subtitle line
tts:fontWeight font-weight Subtitle line
tts:textDecoration text-decoration Subtitle line
tts:wrapOption white-space Subtitle line
ebutts:multiRowAlign text-align Multi-row Align
ebutts:linePadding padding Subtitle line

Personalization

The third phase of the project was enable subtitle personalization. The customizable parameters are:

  • Background: Add or remove the background color of subtitles inside boundary.
  • Font-size: Change the font size
  • Position: Align top or bottom subtitle position.

Non-live subtitles personalization for VoD over hbbtv devices 13

Workload

To solve customization features, there are two possible options a client-side or server-side solution.
A server-side solution consist on: the authoring station is responsible to generate several files covering all feasible personalization scenarios, and make it available to clients. This solution is powerful and allows complex calculations. Server performance capability is higher than HbbTV devices. Besides, these pre-calculations can solve possible collisions between regions or avoid confusions among font color and the underlying video displayed.
The disadvantages are the cost and availability to modify the production chain to include that service.

Alternatively, a client-side solution is based on real-time renderization according to user choice, so the calculations are done by client. The advantages of this solution are: not modify existing production chain and speed up the application of selected changes. In contrast to previous solution, the source file is only downloaded and parsed once at the beginning. Contrarily, this implementation can not make complex calculations, or avoid collisions between regions, due to device performance.
After carrying out an assessment on pros/cons, this one is the chosen approach by the project.

Background

This option allows the user to choose whether the background color of subtitles have been activated or not . Technically, this parameter overrides background-color CSS property at “Subtitle line” layer.

Non-live subtitles personalization for VoD over hbbtv devices 14

The problem identified is: due to the low contrast among subtitle font and the background video can cause reading difficulties. In order to reduce this problem the system applies the CSS text-shadow property.

Font-Size

Using this option the user can increase or decrease the subtitles font-size. Concerning HTML, this configuration overwrites font-size CSS property at “Region Layer”.

Non-live subtitles personalization for VoD over hbbtv devices 15

On identified issues chapter mention, that an increase of the font-size does not implicitly change the size of the regions or positions. Thereby, under next assumptions may cause collisions among regions:

  • two regions are spatially close on the same timeframe
  • user have a black coloured background
  • and, large font-size selected

Non-live subtitles personalization for VoD over hbbtv devices 16

Therefore, the application offer a larger or smaller font size option, referenced to default-case. The largest font-size was decided by trial-and-error learning, reducing as much as possible the collisions unto small overlaps. The smaller font-size was determined by process of elimination keeping proportion with larger size, because a small font do not provoke collisions.

Position

This option allow to customize the subtitles position at the top or bottom part of the video area.

Non-live subtitles personalization for VoD over hbbtv devices 17

The first attempt is Region’s top property fixed policy. Thus, the system controls regions vertical position to show a subtitle. This condition is necessary but not sufficient because we identified  problems if  coexist a couple of regions on the same time, as result regions are overlapped.
Then, a less invasive technique was put in practice: mirroring. Consists in positioning of subtitle regions Y axis reversed. Technically speaking, modifies CSS layer top property, as follows:

Non-live subtitles personalization for VoD over hbbtv devices 18

If a couple of regions coexist in time, as result, the order of subtitles are reversed, causing difficulties for follow some dialogues, nevertheless it happens rarely.

Non-live subtitles personalization for VoD over hbbtv devices 19

Non-live subtitles personalization for VoD over hbbtv devices 19bis

On the other hand, also editorial decisions adopted by subtitler, for example when the video contain credits, subtitles are displaced from default position. As result the mirror technique can’t position subtitles on top and are rendered over relevant parts of the video.

Non-live subtitles personalization for VoD over hbbtv devices 20

Conclusions and further steps

In order to validate the development we passed the EBU-TT-D tests defined by IRT. It  represents a evidence of success in developing a fully functional EBU-TT-D subtitles application over HbbTV  technology. A couple of normative properties were omitted, tts:writingMode and tts:direction, because are rare on western writing, and initially out of scope.

Despite the solution is fully HbbTV compatible based on lower computational complexity design, we must apply a performance threshold because a 16% of the market is composed by low performance devices.

Moreover, the project have achieved the subtitle customization milestone, allowing changes at runtime by the client device on following properties like font-size, background color and position of subtitle text.

At this point, it is proposed the following steps:

  • Expand personalization possibilities, such as scale video and paint subtitles outside without overlaps but reducing
  • Implement server-side subtitles personalization solution, solving collisions and contrast issues
  • If required, complete the solution adding pending attributes to be fully-compatible with other languages out of west-europe such as oriental, arabic, and so forth.
Posted in Developement Tagged with: ,