In this evaluation we consider the actual listening conditions of listeners. Based upon the tools applied by EBU-R 128 and corresponding technical documents we want to investigate the interdependence between loudness harmonisation, loudness range characteristics and listening conditions. In the context of NoTube, the user’s listening environment can typically be characterised as a computer based listening environment with, among others, built-in laptop speakers or small external PC speakers. But taking into account the proceeding fusion of internet and TV (e.g. HbbTV etc.) we consider also studio-quality stereo systems and home cinema speaker systems.

The evaluation is based upon a sufficient number of short video clips covering different genres like “Movie”, “Commentary”, “Concert”, “Sport”, “Show”, “Commercial” and “News”. The audio part of the clips selected for evaluation was varied with respect to Programme Loudness and Loudness Range. In each case three variants are presented to the user. The task of the participants is simply to indicate the preferred variant.

Although EBU-R 128 defines Target Programme Loudness it does not enclose any relation to the reproduced sound pressure level. This relation depends on individual taste, personal preferences, user habits and the listening environment. This fact implies that such loudness evaluations are strongly influenced by the “individual volume” which is set by the listener. In order to include this important individual parameter a corresponding procedure was prepared to adjust and register the “individual volume” (see below). It defines the individual reference volume and shall be constant and not changed during the complete evaluation procedure, so the test persons are asked to not change the volume level finally adjusted in this procedure. In contrast to other audio evaluations where a constant listening level is given each participant is able to choose his preferred listening level individually.

Loudness Adaptation

For the evaluation of the loudness adaptation short extracts were selected from ten video clips representing different genres. The compilation of three variants of audio adaptation under test was carried out by measurement and adaptation of the Programme Loudness using tools referring to EBU-R 128.

Table 1: Audio Adaptation Types

Loudness Range Adaptation

For the evaluation of the Loudness Range adaptation five extracts from video clips were selected from different genres. The Loudness Range adaptation used for the test is strictly based on the measurement of the loudness descriptor “Loudness Range (LRA)” as specified in EBU-TECH 3342. The following figure shows the resulting LRA values in LU (Loudness Units as defined in EBU-R 128) under test of the five loudness range items uncompressed and after LRA compression.

Figure 1: Loudness Range of items under test (each uncompressed and adjusted compression c1_15 dB/c1_25 dB)

Listening level

In order to enable the test person to adjust and to indicate his individual listening level (sound pressure) a short extract from a news broadcast was selected as test item. Our ears are especially familiar with the sound of human speech, thus, news anchors are ideally qualified to help adjusting to a convenient (or even the correct respectively original) listening level. The individual listening level that is finally adjusted by the user is considered to be the reference volume and should not be changed throughout the complete test procedure. The individually adjusted listening levels are obtained using a special test signal with announcements in different (loudness) listening levels where the announced listening level meets the corresponding loudness level.

These level announcements are presented after adjustment of the individual reference listening level. The participants are asked to indicate the first level announcement which they clearly can understand by clicking the corresponding button. This method is well-known as “Hearing Threshold Method” and presumed to be notably precise. The differences between the announced levels are 7 LU. From the psychoacoustic point of view this difference can be indicated as “clearly distinguishable“. An estimation of the relationship to the corresponding sound pressure level can be achieved by measuring the resulting reproduced average sound pressure level of the anchorman in an individual listening condition. The relationship between announcement and loudness level is presented in the following table.

Table 2: Listening level announcements – Relationship between reference loudness and sound pressure listening level

Introduction and evaluation

The   adjustment  of the listening level  is part of the introduction to the evaluation. The introduction contains an additional video sequence which is composed of nine short clips with different loudness adaptations to make the test person familiar with the loudness adaptation under test. The introduction comprises also a questionnaire to collect the age of the test person and the following characteristics of the individual listening condition:

• age
• type of speaker
• size of speaker
• distance to speaker
• background noise

The arrangement of the loudness evaluation is like follows, each clip under test is presented in those 3 variants which have been described above. The participant is asked to listen to each of the 3 variants of the actual clip and thereafter to indicate which loudness/loudness range variant he personally prefers considering his taste/custom respectively his individual listening environment. After indicating the preferred variant by clicking the corresponding button the next clip with its three versions is presented for evaluation. The order of the presentation of clip number respectively variant is done randomly.

Preliminary Results

The preliminary results presented here are based on the evaluation period from September 19th 2011. Besides partners from the NoTube consortium we invited participants from other communities, e.g. the EBU audio expert group “FAR-PLOUD” and the “Surround-sound-Forum” within VDT (Verband Deutscher Tomeister). However, the results presented below are only a preliminary subset based on 48 valid test runs which were collected so far. In general, the presented data at this stage of the analysis has to be considered as purely descriptive. Here, only the the calculated percentage collected for each attribute collected is presented. In this preliminary presentation of the results we want to focus only on the aspects listening level, listening situation, background noise and the main results concerning loudness and loudness range evaluation.

The distribution of the listening level in figure 2 shows that the majority of participants chose a rather high listening level. The preferred level was Level 3 which was selected by almost 40% of the test persons.

Figure 2: Distribution of Listening level

The distribution of the speaker type in figure 3 shows a nearly equal representation of headphones (both in-ear and on-ear), built-in speakers and external speakers (proportion of subwoofers related to PC and stereo speakers only) with a predominance of headphones.

Figure 3: Distribution of Speaker Type

The distribution of the indicated background noise as presented in figure 4 shows a clear dominance of weak background noises. Only one user declared to have strong background noise.

Figure 4: Distribution of Background Noise

The results of the evaluation of both loudness and loudness range adaptation are presented in the following figures.

Figure 5: Preferences of Loudness Adaptation

Figure 6: Preferences of Loudness Range Adaptations

Conclusions

First conclusions with respect to the evaluation of loudness and loudness range adaptation which can be drawn from the descriptive data presented above approve the excellent performance of EBU-R 128 with respect to Target Programme Loudness. This answers the open question whether the loudness harmonisation following EBU-R 128 which was clearly preferred in the previous evaluation is depending on individual listening conditions. Considering the listening conditions covered in this loudness web evaluation there is clearly no influence observable.

With respect to the evaluation of the different loudness range adaptations there is a tendency identifiable. The participants seem to prefer rather medium or even strong loudness range compression (compression characteristics c1_15 dB/c1_25 dB in figure 1) than uncompressed audio with high loudness range. A first analysis of the data with respect to correlations between loudness range adaptation aspects and the type of speaker or background noise showed no noticeable interrelation. On the other hand this result could be an indication that loudness ranges of 25 LU and more do not meet home listening environments respectively the expectance of the majority of home listeners. But in order to answer these questions an analysis of a larger dataset is necessary.

Call for participation

We thus invite you to participate in the web evaluation which will be open for another two weeks. The test can be carried out online. Simply go to http://survey.irt.de/notube to take part and to learn more about Loudness Normalisation on the Web! The evaluation will be open until Friday, December 16th 2011.

(This post was contributed by Gerhard Spikofski, IRT) 

How can you help us with your feedback?

In the Beta Version of BMF 2.0 we make available the BMF-W3C XML Schema with documentation. We would be glad to receive comments and suggestions on following topics:

• Structure of the XSD and applicability for implementations
• Semantic of the model (Are there any open questions? Is there an error in the description of the context? Do you miss any important metadata?)
• Comprehensibility of the documentation in the XML Schema

We would be very grateful if you could find some time for feedback. It is the aim of BMF 2.0 to comply with real world requirements from users as well as from developers. In this way you can influence the further development of BMF so it will be better customized for your future applications.

We would appreciate to receive feedback until the 15th of December 2011. If you have any further questions please do not hesitate to contact us at bmf@irt.de.
For details please check http://bmf.irt.de/en.

The importance of getting these things right is reflected in the growing interest outside of the NoTube project in data mining of activity and social data for the provision of social recommendations.

To re-cap, NoTube’s Beancounter technology supports the automatic generation of an implicit user interests profile. This is based on re-use of an individual’s activity on social media services, such as the content of their tweets and Facebook ‘likes’, to determine their interests. The idea is to re-use the scattered and disparate activity data and make it useful by combining it, looking for patterns, and using it to suggest things to watch.

Beancounter is currently being re-built in the backend. Alongside this we have been working with visual designers from the design agency Fabrique in Amsterdam to develop a new user interface for the Web front-end. This blog post outlines some of the design challenges we have encountered along the way.

Challenge 1: Displaying an overview of your interests

To create your Beancounter profile you need to link one or more social media accounts (Facebook, Twitter, LastFM etc) to your Beancounter account. Beancounter retrieves activity data from these sources, interprets the information contained in the activities and matches it to concepts from the Linked Open Data cloud (DBpedia concepts) to give you an overview of your interests. These interests include programmes, movies, people, locations and genres. Each interest is assigned a weight in the profile; the more instances of an interest in your activities, the higher the weight of the interest – and the more influence it has on your recommendations. These weightings change over time as the Beancounter continually adjusts to your activities. For example, if you watch or listen to lots of coverage of The Olympic Games but aren’t generally interested in sport the rest of the time, the weightings for sport-related interests will temporarily increase during The Olympics.

This interests profile is useful because it can be automatically used as input for personalised TV recommendations (via NoTube’s N-Screen prototype for example) to help you decide what to watch. And because the profile is portable, it could also be used as input for other applications.

During implementation of the first version of the Beancounter it became clear that there are potentially a very large number of interests to display for any given user, many of which have very low weightings because they have only appeared once or twice in the user’s activities and therefore have very little significance for recommendations.

We wanted a simple way to clearly present the spread of interests, which also immediately conveys the influence (i.e. weight) of each interest for recommendations, and takes account of the fact that the number of interests could quickly become quite large. As a solution, the designers adapted the tag cloud metaphor into a scaleable grid, with variable font and cell sizes to indicate relative weightings and hierarchy, so that the most influential concepts are prioritised at the top.

Design mock-up for displaying Beancounter interests

Draft Beancounter design mock-up showing the activities that contributed to a particular interest in the user's Profile

Challenge 2: Showing how each activity affects your interests

Another of the UI design challenges that emerged from implementation of the first version of the Beancounter is how to show the contribution of an activity to the ‘evolution’ of your interests. For example, as described in a video of an earlier demo, clicking on the activity “You watched Timecop…” in the user’s activity stream opens a pop-up (shown in the screenshot below) using bar charts to show the ‘before’ and ‘after’ status of the user’s interests based on watching the film Timecop. The two existing concepts ’1990s science fiction films’ and ‘Films shot anamorphically’ now have a greater weighting than they did before. The other concepts (where only the blue bar is displayed) are new interests associated with Timecop, which didn’t already exist in the profile.

Early Beancounter UI showing the effects on a user's interests of watching the film Timecop

We wanted to simplify the presentation of this information, and to make it easier to understand what is happening and why it might be interesting. We’re getting there with the new design, although we’re still thinking about the best way to convey the idea that the length of the bar really represents ‘the influence of this concept on your recommendations’, and of providing a relative scale to measure this level of influence against.

Draft design mock-up for Beancounter showing the effects of an activity on the user's interests

This design also allows the possibility for the functionality to be extended so that you could manually adjust the weighting of a particular concept (for example, by sliding the bar to the left or the right) to give it more or less influence over recommendations.

Challenge 3: Displaying on-the-fly data analytics

In addition to displaying your interests, it has always been our intention to offer some analysis of the data that the Beancounter has collected about you. This is based on the premise that people are usually interested in information about themselves, and the initial inspiration came from the Dopplr annual report and the BBC’s RadioPop prototype for social radio listening.

Beancounter offers the potential for a range of detailed analytics, including what you’ve watched and listened to most often and when, the things you are most interested in now and at previous points in time, people who have similar watching and listening habits, and those who are least similar. For more design inspiration we looked at many examples of beautiful data visualisations. We particularly liked the infographics from Hunch.com and The Feltron Annual Reports. However, many of these were hand-crafted, and our requirement is for attractive design modules that can be adapted for automated on-the-fly presentation.

Draft Beancounter design mock-up displaying an analysis of the user's data

Challenge 4: Interacting with multiple layers of information

The way that data is stored in the new Beancounter allows for any activity (e.g. listening to a Tom Waits track on Last.fm), interest (e.g. Tom Waits), type of interest (e.g. all people) or type of activity (e.g. all the things you’ve listened to) in the UI to be linked to the relevant analytics relating to that object; providing timelines, comparative views, explanations and statistics. Whilst this enables the user to delve deeper and gain extra insight to their data should they wish to, we want to make sure that the UI doesn’t become cluttered and confused with all these additional overlays. We’re therefore working with the visual designers to determine the most elegant model for interacting with these multiple layers of information without being overwhelmed by them.

Next steps

We’re still finalising the design work. Over the next few months we will be integrating these design mock-ups into a new Beancounter UI so that you will be able to try it out, get personalised TV recommendations in NoTube’s N-Screen, and perhaps discover some interesting new things about yourself…

• (partial) Redux
• TED talks
• iPlayer

They all have the same basic design with small tweaks for image size. They interoperate – you can drag and drop between them. The main differences lie in the collection of data for the backends and in the calculation of similarity between videos. For similarity calculation we use three different techniques for the three different datasets, as we had three different sets of data available.

The Redux version was our first experiment in this area. BBC Redux is a BBC research video on-demand testbed. We were lucky enough to be able to obtain anonymised watching data for programmes in a five-month subset of the period it covers. So our first experiment, led by Dan Brickley, was to take that watching data – around 1.2 million observations over 12,000 programmes – and use open source tools to generate similarity indexes. We were able to use a standard function in Mahout, Apache’s machine learning and data mining software, to generate similarity indexes using a Tanimoto Coefficient model. This function essentially uses people as links between programmes (“Bob watched both ‘Cash in the Attic’ and ‘Bargain Hunt’”), and sorts programme pairs according to how many people watched them both. With this dataset, this technique produced some nice examples of clearly related clusters (for example what you might call ‘daytime home-related factual’, see picture below).

A cluster of 'daytime home-related factual'

It is quite rare to have access to this kind of data about what people have watched. It’s both valuable and private, and may not be readily available. It may not exist, if no-one has watched anything yet. For the TED dataset we therefore took a different approach. TED talks are a diverse set of talks by people prominent in their field, licensed under the Creative Commons BY-NC-ND license. From our point of view, the advantage of using this dataset was that transcripts were available for all talks. To calculate similarity between the talks for N-screen we were therefore able to use a tf-idf algorithm. This technique treats each programme as a document, and finds the most characteristic words for each document within the total corpus of documents, and can be used to match the documents based on the words selected. We were lucky enough to be able to use some Ruby software open sourced by a colleague at the BBC to do this.

This technique produced clearly similar clusters within the 600 video dataset, for example, in this selection, you can clearly see items relating to women and also to drawing and art:

Our third example is an iPlayer version of N-screen. On any given day, there are about 1000 TV and radio programmes available to UK viewers on iPlayer, the BBC’s on-demand service. This is an interesting dataset for us because of its high-quality metadata, including categories, formats, channel and people featured. We were curious as to whether we could generate interestingly similar programme connections using only metadata. Our first approach was to try a Tanimoto similarity over the structured metadata, but the results were not particularly satisfactory – many programmes had no similar items. We then tried tf-idf over the metadata descriptions. This seemed to pick up characteristics of the text rather than of the programmes (for example repeated quirks in phrasing of the descriptions). The best approach we have tried (evaluated only informally) is tf-idf over a combination of metadata and the results of an entity-recognition technique.

We used the existing metadata from /programmes json format (for example http://www.bbc.co.uk/programmes/b00k7pvx.json or http://www.bbc.co.uk/programmes/b015ms3r.json). As you can see from those examples, some have descriptions of people who are in the programme, with mappings to dbpedia where available. We can get more of these by using a service to extract entities from the description text. We used Lupedia, which was developed in the NoTube project by Ontotext for this. We took this data coupled with the channel and the categories to produce a list of keywords, and then ran tf-idf over the top of that. The result can be variable:

Example of a not very good similarity match

but in many cases, reasonably good:

Example of a good selection of similar material

and occasionally throws up an interesting surprise:

Unexpected link between programmes

The next stage is to evaluate these results formally.

N-Screen related content screenshot

We had a number of very specific requirements:

• Real time communication
• Different types of receivers (people, TV/video players, others)
• Structured data transfer
• Anonymous usage

We also needed good, open tools and libraries available because of the limited amount of time we had to implement.

Like several other groups, we’ve been using XMPP (Jabber) for the backend because it works in real time and has plenty of tools and libraries. Others have been using the PubSub framework to broadcast synchronised content to connected devices, but integral to our plan was to enable any people watching to also be able to share. I had a surprising amount of success with using a central negotiator that allowed ad-hoc groups to be formed from anonymous users, populating each user’s roster with other people it knew about. However, a much less error-prone approach has been to use ad-hoc XMPP group chats, and this has enabled us to make a pure HTML/Javascript implementation with no backend dependencies apart from an XMPP server and some simple APIs to the database of content.

I’ll talk a little about the requirements in more detail, mentioning some implementation issues as we go.

Requirements

Real-time communication

This is essential for drag and drop between devices to be ‘realistic’ – i.e. for a good user experience. Network issues can always be tricky here, particularly under demonstration (rather than real-life) conditions.

Different types of receivers

A ‘TV’ listens for ‘play’ and ‘pause’ messages and does something with them. A ‘person’ listens for ‘drop’ messages and displays them appropriately. There might also be other kinds of listeners – loggers perhaps, or bots that enhance or modify content dropped to them. All types need to take account of who is joining the group and the kind of thing that they are so that they can do the right thing and display them appropriately.

Structured data transfer

For user experience reasons a fair bit of data needs to be send on most interactions. A shared item needs to have basic metadata (identifier, URL, title, description, image) and also who shared it. Other kinds of message include announcements about the kind of thing you are. We chose Json as the body of the XML XMPP message, though XML would also have been fine or better. One issue is that ‘IQ’ (hidden data) messages cannot be sent to group chats, so that all group messages are visible in a standard chat room if connected to with, say, PSI.

Anonymous usage

Although there is plenty of potential for connecting N-screen with Twitter and / or Facebook, we didn’t want to require it. In N-Screen you need to give a name so that other people using the application can refer to you, but that’s the limit of the requirement for identification. For scalability and maintainability reasons we didn’t want to create a lot of named users on the XMPP server. Fortunately, XMPP allows you to create group chats with anonymous users, which is perfect for our needs.

The setup

We’ve been through many iterations to get here but I’m now pretty happy with the setup we have.

Ejabberd server with Bosh and group chat enabled

Ejabberd is not particularly simple to set up, but once it is up, seems pretty stable. I’ve put some tips on troubleshooting here (scroll to the end). PSI is a great tool for debugging as you can set it to log the XML messages going past.

PSI view of a groupchat created behind the scness of N-Screen

PSI XML view of a groupchat in N-Screen

One thing to note is that for ejabberd at least the group chat URL is

[room_name]@conference.[server]/[nick]

e.g.

default@conference.localhost/libby


APIs to the content

I used a simple ruby server and mysql backend to generate Json search and random APIs. For content-to-content recommendations for TED we have used TF-IDF analysis of the transcripts using this code by my BBC colleague Chris Lowis.

The workflow is as follows:

• The user goes to a webpage, and gets an alert requesting their name
• Based on the window hash (the bit after ‘#’), the Javascript chooses what group chat to join / create, using Strophe over Bosh to make the connection and announces itself to the room using a presence message with the name provided by the user
• The eJabberd server then automatically tells the user about the other partipants in the room, and the Javascript renders them either as users or as TV
• The ‘TV’ is also a piece of Javascript / Strophe that additionally announces itself to all joiners of the room as a particular type of thing (a ‘TV’). Multiple TVs are allowed in the room.
• All user pages keep a list of all TVs, and on dropping to the TV sends a programme onto all of them
• On leaving the page the user is disconnected from the eJabberd server – this can take a few seconds to percolate to the user interface.

The rest is client-side, which I’ll talk about further in another post. Feel free to try out N-Screen here.

As we’ve seen, the problem with on-demand video is too much choice is exhausting and demotivating and leads to satisficing behaviour and sometimes no choice at all, particularly in group-choice situations.

It looks as if watching together apart (watching the same thing in different physical locations) is going to be a big deal in the future. Both google and facebook are putting in place tools that allow people to hang out while watching videos together.

Lets think about a group choosing what to watch from scatch. What sorts of things do they say?

• who is s/he? (who is that actor / participant?)
• who directed it? (who made it?)
• what’s it about?

but also:

• what do you want to watch?

The first set of questions are the kinds of questions metadata can answer. Who is in it, who created it, what sort of thing you can expect from it, what it is similar to.

The second type of question is much harder. We have preferences about each others’ future mental states, or to put it another way, we would usually like everyone to enjoy the content we will watch together, without fully knowing the other participants’ preferences or state of mind. It’s a hard decision problem, and it’s no wonder people give up quickly.

N-Screen is a second screen HTML / Javascript web application that allows people to express their preferences to each other directly, by dragging and dropping content to each other individually or as a group, directly answering the second kind of question. When someone receives some content like this in N-Screen, they can click on it to see more information about it, answering the first kind of question.

The system is designed to be used in conjunction with an out-of-band communications channel (e.g. face to face chat, Skype, or IRC) for the direct negotiations, as much depends on the subtleties of communication – understanding how people are feeling – and this is best done using some familiar channel. It’s called ‘N-screen’ because it might be the primary screen, or one of a bunch of equals; it could play video locally or remotely (in theory).

It’s primarily for tablets and laptops, but runs on anything with a modern Web browser; from smartphones to touch-tables and desktop PCs. It works very nicely on a desktop PC with a touch screen; whereas it serves only as proof-of-concept on an iphone or android phone right now. Similarly, it can run on a touch table, but doesn’t make the most of its potential.

Once people have found something interesting to watch together one of them can drag and drop it to the TV and it will play.

One of the design aspirations behind this work, was to explore practical ‘hands on’ notions of collective intelligence, particularly from groups (perhaps professionals with a common goal; perhaps school children) who are intensely exploring some collection or topic together.

We have a demonstration which you are welcome to try, that uses some of the wonderful TED Talks videos. It’s a work in progress and we’ll be adding features over the next few weeks (if you want to try playing it, use this URL in another window (uses flash).

Do let us know if you have any comments.

Following the results of this test, we decided to perform another user evaluation to investigate the variation of both the Programme Loudness and Loudness Range (LRA) considering different listening situations as they can occur in the context of NoTube, i.e. using a computer, a mobile device or a Hybrid TV. We prepared different versions of a number of video clips to evaluate the application of loudness and LRA adaptation for typical listening situations of users.

The test can be carried out via the Web and to acquire the largest possible number of test cases for this evaluation we invite everyone to participate! Go to http://survey.irt.de/notube or scan the QR code with your smart phone to take part and to learn more about Loudness Normalisation on the Web! The evaluation will be open until Wednesday, October 12th 2011.

Find us at Connected World, Hall 13, Stand 202 – opposite the Internet Cafe!

The video illustrates the role of metadata in the convergence of TV and the Web and explains the approach of NoTube. Enjoy!

Please go to TV Metadata Interoperability for more information about the role of TV metadata in the convergence of Web and TV and in NoTube .