At this stage, the objective was be able to load the subtitles and render them. All associated with the video at the precise viewing moment, respecting the begin and end times interpreted from EBU-TT source file.
Load and parse
The following steps are carried out when user decides to enable subtitling:
- First, load EBU-TT subtitles source file.
- Then, parse the xml file. Through this, all nodes are extracted.
- Last, alignment by time. The system prepares a virtual timeline containing a necessary data ordered from begin-end time references.
This topic is the most difficult part, due to limitations of some HbbTV devices. Theoretically, to synchronize subtitles with the video, the software in charge should check in each encoded video frame if a subtitle is scheduled and show or hide texts as required.
Seeing this, the approach requires a high-processing capacity at hardware system level, forcing to render and remove subtitles to the nearest begin-end times, in less than 40 ms per video frame. Conversely, it could cause undesired collisions as it’s shown in the following picture.
In order to reduce collisions (coloured in yellow), we had to rearrange strategy. A time-based approach instead of frame-based one.
Given a sample rate, the system must check if there is any subtitle candidate before next check point. For example, considering a 500ms sample rate, check if in next 500ms window exists any candidate point. As result, devices have more time to perform the necessary operations, while minimizing temporal display error.
HbbTV video object did not have video-frames or time-seconds change events to subscribe. Therefore, the way to implement video sampling rate is use the W3C APIs, by means of window.setTimeout or window.setInterval methods, which are highly inaccurate, especially working at high frequency rates. The lack of accuracy may cause errors on subtitles rendering because in the virtual Timeline there are opaque slots, not evaluated.
Here a new criteria was added to the system, to enlarge the analysis window at double of sample frequency rate, to avoid the loss of information.
The developed approach could be seen as conservative, but still exists a percentage of HbbTV devices on market without the enough capability. Regrettably on those devices subtitles service is not available to prevent a unusual user experience.
According to our own statistics around 16% of devices HbbTV enabled have a low performance profile.
Styling and layout
This phase implies upgrade previous parse methods, and take into account the ‘region’ and ‘style’ nodes during subtitle text rendering.
A tt:region element defines a space or an area in which subtitle content is to be placed. It specifies a set of layout information through attributes.
HTML Layer structure
To achieve an advanced implementation of EBU-TT, all possible scenarios must be covered. The solution is designed based on the following HTML div’s layer structure, the starting point is ‘region’ layer and their attributes.
All normative attributes defined by region has been translated into CSS properties. The table below shows the agreed relationship.
|EBU-TT property||CSS property||HTML layer|
|tts:origin||top and left||Subtitle line|
|tts:extent||width and height||Subtitle line|
|tts:displayAlign||Display Align||Display Align|
As with regions, EBU-TT-D style attributes has been mapped into CSS properties in order to fit the desired appearance.
|EBU-TT property||CSS property||HTML layer|
The third phase of the project was enable subtitle personalization. The customizable parameters are:
- Background: Add or remove the background color of subtitles inside boundary.
- Font-size: Change the font size
- Position: Align top or bottom subtitle position.
To solve customization features, there are two possible options a client-side or server-side solution.
A server-side solution consist on: the authoring station is responsible to generate several files covering all feasible personalization scenarios, and make it available to clients. This solution is powerful and allows complex calculations. Server performance capability is higher than HbbTV devices. Besides, these pre-calculations can solve possible collisions between regions or avoid confusions among font color and the underlying video displayed.
The disadvantages are the cost and availability to modify the production chain to include that service.
Alternatively, a client-side solution is based on real-time renderization according to user choice, so the calculations are done by client. The advantages of this solution are: not modify existing production chain and speed up the application of selected changes. In contrast to previous solution, the source file is only downloaded and parsed once at the beginning. Contrarily, this implementation can not make complex calculations, or avoid collisions between regions, due to device performance.
After carrying out an assessment on pros/cons, this one is the chosen approach by the project.
This option allows the user to choose whether the background color of subtitles have been activated or not . Technically, this parameter overrides background-color CSS property at “Subtitle line” layer.
The problem identified is: due to the low contrast among subtitle font and the background video can cause reading difficulties. In order to reduce this problem the system applies the CSS text-shadow property.
Using this option the user can increase or decrease the subtitles font-size. Concerning HTML, this configuration overwrites font-size CSS property at “Region Layer”.
On identified issues chapter mention, that an increase of the font-size does not implicitly change the size of the regions or positions. Thereby, under next assumptions may cause collisions among regions:
- two regions are spatially close on the same timeframe
- user have a black coloured background
- and, large font-size selected
Therefore, the application offer a larger or smaller font size option, referenced to default-case. The largest font-size was decided by trial-and-error learning, reducing as much as possible the collisions unto small overlaps. The smaller font-size was determined by process of elimination keeping proportion with larger size, because a small font do not provoke collisions.
This option allow to customize the subtitles position at the top or bottom part of the video area.
The first attempt is Region’s top property fixed policy. Thus, the system controls regions vertical position to show a subtitle. This condition is necessary but not sufficient because we identified problems if coexist a couple of regions on the same time, as result regions are overlapped.
Then, a less invasive technique was put in practice: mirroring. Consists in positioning of subtitle regions Y axis reversed. Technically speaking, modifies CSS layer top property, as follows:
If a couple of regions coexist in time, as result, the order of subtitles are reversed, causing difficulties for follow some dialogues, nevertheless it happens rarely.
On the other hand, also editorial decisions adopted by subtitler, for example when the video contain credits, subtitles are displaced from default position. As result the mirror technique can’t position subtitles on top and are rendered over relevant parts of the video.
Conclusions and further steps
In order to validate the development we passed the EBU-TT-D tests defined by IRT. It represents a evidence of success in developing a fully functional EBU-TT-D subtitles application over HbbTV technology. A couple of normative properties were omitted, tts:writingMode and tts:direction, because are rare on western writing, and initially out of scope.
Despite the solution is fully HbbTV compatible based on lower computational complexity design, we must apply a performance threshold because a 16% of the market is composed by low performance devices.
Moreover, the project have achieved the subtitle customization milestone, allowing changes at runtime by the client device on following properties like font-size, background color and position of subtitle text.
At this point, it is proposed the following steps:
- Expand personalization possibilities, such as scale video and paint subtitles outside without overlaps but reducing
- Implement server-side subtitles personalization solution, solving collisions and contrast issues
- If required, complete the solution adding pending attributes to be fully-compatible with other languages out of west-europe such as oriental, arabic, and so forth.