Research on Universal Broadcasting at NHK STRL

All Rights Reserved, Copyright © NHK

Research on Universal

Broadcasting at NHK STRL

Masahiro Shibata

NHK Science & Technology Research Labs.


Background

2


Terminology

3

Universal Broadcasting

Broadcasting which can deliver information to

everyone including visually or auditorily impaired

people, elder people, and foreigners.

Accessible Technology

Technology to realize universal broadcasting or

human-friendly information presentation.


Objectives of Universal Broadcasting

4

1. To deliver information “extensively in Japan”

2. To deliver information to “everyone equally”

regardless of impairment/age or language

difference

Two dimensional

“extensiveness”

Three dimensional

“extensiveness”

= Universal Broadcasting

Eve

ryo

ne

-equ

ally


History of closed caption (CC)

and audio description (AD) services

5

Analog broadcasting Oct. 1983 Character multiplex broadcast (test service)

Oct. 1984 Closed caption (CC) started. (morning drama “Oshin”)

1982 Audio multiplex broadcast

Apr. 1991 Audio description (AD) started.

(morning drama “Rinrin-to”)

Digital broadcasting ISDB-T standard

Any digital receiver has decoders for CC and AD

⇒ Hardware issues solved


Accessible-service guideline

6

Service guideline set by Japanese government

Guideline for CC

（set in 1997） Guideline for CC and AD

(set in 2007)

Target year 2007 2017

Scope Recorded

programs

All programs

from 7AM to 12PM

CC 100% 100%

AD － 10% (15% for NHK 2 (Ed.)) Status in 2010 NHK 1 NHK 2 (Ed.)

CC 62.2% 52.5%

AD 7.6% 11.2%

Completed

（percentage per 7AM to 12PM)


Research on accessible technology

7

CG

sign language

Accessible technology for the impaired

CC by speech

recognition

AD script

editor

Accessible technology for the elderly, foreigners

答え：後に同じことが書いてあるので省略しましょう

夏に電力が足りなくなるかもしれない問題で、人々の節電への関心が高くなったため、扇風機が良

く売れています。質問：「夏に．．．高くなったため」

の部分はいりますか？

現在の難しさ 2500

“Easy Japanese ” for foreigners

Multi-modal

display

Speech processing for the elderly


Research on accessible

technology for the impaired

[Closed Caption (CC) Service]

8

News show, general news etc.

Relayed keyboards method Stenographic keyboard method

Direct speech recognition

http://www.speed-wp.co.jp/laboratory/

Re-spoken speech recognition

Music programs etc.

Sports, information programs etc. Short news, American baseball, etc.

Methods of producing live CC in NHK


10

Non-live CC Drama, Documentary, etc

With

CC

Keyboard Part of general news

Music show etc.

Current status

of NHK 1

Re-spoken/Direct

Speech rec. Sumo, baseball etc

With

ou

t C

C

Hybrid-type

Speech rec. Short news

Talk shows, etc.

Recognition of

Spontaneous speech

R

&

D

Speech recognition for CC


text

Speech rec. Error correct CC on-air

voice

Speech of announcer

or reporter

Direct

Interview

Re-speak 1-2 operators

Male-female recognition, Auto. dictionary update

Hybrid speech recognition for CC

11

Recognize speech of announcer and reporter directly, and re-spoken speech for interview Recognition accuracy of over 95%

Higher recognition rate and lower operation cost

R&D issue: Spontaneous speech in news show programs Need to overcome unclear or rapid speech and phrasing in speech

Recognition accuracy of less than 90% for spontaneous conversations

Speech recognition

Operators for error correction

Output

display

Latest CC system by speech recognition

for 4 o’clock News Mar. 14, 2012〜




[Audio Description (AD)

Service]

13


AD production

Current AD should be considered as a kind of “special radio drama” and its

production is labor intensive.

Script writers and narrators have to be highly trained.

Finished

program Writing

AD script

Recording

AD voice

(in studio)

Mixing down

to sound track

Script writer

Script writer

Program director

Narrator

On Air

Program director

Narrator

Engineer

14

Time-consuming!

AD script editor

Detected periods

with stage

direction

Input script

Script exceeding

the estimation is

indicated by red

Automatically

estimated syllable

counts

Supplemental

direction for narrator

Automatically

generated

script draft

Current script

window

Wave form of

program sound

Automatically

detected periods of

voice and silence


Evaluation of the system

Average of three trials by MS Office Operators

0

20

40

60

80

100

120

140

160

180

手作業

（1人の結果）

電子台本なし電子台本アリ

システム利用

165

101

77

Manual input Without

data of

original script

With

data of original

script

Using AD script editor system

Work time of

Experienced

AD Writers

Tim

e to

write

scrip

t(m

in)

16

(Duration of the

program was 15 min)


Evaluation of the system

17

Opinion of Novice Writers • The system is useful.

• Indicating silent portion is convenient.

• Estimation of syllable count is a good help.

Opinion of Experienced Writers • The system does not help much.

• AD should be inserted to right positions regardless of the presence of main program voices. This should be done by writers’ skill.

Conclusion

The system can extend the variation of AD programs without intensively training new writers while AD can be supplied only for very popular programs with the current production style.




[Sign Language (SL) Service]

18


SL service

19

For deaf people by nature or those who lost hearing in early childhood, sign language is their first language. Sign language service (rather than Closed Caption (CC) ) is requested.

Grammar of Japanese Sign Language (JSL) is different from that of the aural Japanese.

Difficulty to keep sign language interpreters for regular service Especially for emergency news

TV

program broadcasting

internet SL CG

Station Home Translate CC to SL

CG sign language service by automatic translation


Overview of automatic translation system

Japanese - JSL

dictionary

Japanese - JSL

corpus

Input Japanese sentence: 私の名前は金子です。 (“My name is Kaneko.”)

Convert Japanese sentence into sequence of

symbols representing JSL word

Generate signing motion for each symbol

(including hand and finger motions, facial

expressions, and so on).

Generate seamless motion transitions between

each symbol by using motion interpolation.

Output computer-animated JSL

[ 私, 名前, 金子, 言う, N ]

( I, name, Kaneko, say, nodding )

20


Japanese – JSL CG dictionary system

Found Japanese entry

Confidence measure

Thumbnail image of

CG animation

Click entry to display

CG animation of JSL word

Entries of synonyms

21

Input Japanese word


Evaluation of

Japanese-JSL CG dictionary system

22

■Natural

■Some part is unnatural but omittable

■Some part is unnatural but not

bothering

■Strange

■Very unnatural

Naturalness

■Easy to read

■Some part is unreadable but omittable

■Some part is unreadable but not

bothering

■Bothering

■Unreadable

Readability

Based on questionnaire survey at Tokyo Federation of Deaf meeting on May 5-6, 2011


Japanese-JSL CG translation

prototype for weather information

23

Screen shot of Japanese-JSL example-based translation prototype.

C:/Users/shibata.m-mg/Documents/My Documents/NHK-Document/24年度/人間・情報科学/発表/ITU-NHK国際ワークショップ/Work/映像/翻訳.wmv




[Multi-modal Display]

24


Digital TV receiver for visually impaired

25

Visual information and graphical user interface (GUI) are everywhere! PC, TV, smart-phone, remote controller, ticketing machine, ATM, etc

But they can be big obstacle for visually impaired people

Example of data broadcasting screen

DTV receiver

for visually

impaired people

•Reading-out menu and other

visual information

•Magnifying characters on screen

•(Braille expression of texts)


Also applicable for guide terminals in public places such as

railway stations, ATMs in the bank.

Interactive tactile display

26

Two dimensional tactile display + touch position sensor

As a GUI device for visually impaired people

Hierarchical expression Menu expression by Braille


Information-barrier-free for visually

impaired people

27

Voice and Braille

Tactile and force feedback

3D objects

Interactive tactile display

2D graphics (+ GUI)

texts

Universality

for

impaired/non-impaired



technology for the elderly

28


Aging of the society

29

Aging rate and its speed of Japan is highest in the world.

Compensation to auditory and visual ability is social requirement.

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

日本

イタリア

ドイツ

アメリカ合衆国

韓国

シンガポール

Japan

Italy

Germany

U.S.A.

Korea

Singapore

Redrawn from “Annual

Report on the Aging

Society: 2009”

(by Cabinet Office Japan)

Agin

g r

ate

(≧

65

)


Problems in information perception of

the elderly

30

Visual information

Degraded accommodation function (presbyopia)

Low vision

Lower sensitivity of bluish color

Auditory information

Less sensitive to higher tone

Difficult to understand speech with high speed

Difficult to understand speech under background noise




[Speech Rate Conversion]

31


Evaluation experiment TV and radio set with “Slow button”

32

Started research by responding a claim from an elderly person; “Recently, speech on broadcasting is too fast for me to understand”

Slow down speech without degrading sound quality

Speech rate conversion for the elderly


Speech rate conversion without

changing voice pitch

33

Temporal duration of waveform is changed to keep fundamental period

×

Slower time

① ② ③ ③ ④ ⑤ ⑥ ⑦ ⑧ ⑧ ⑨ ⑩

Faster time

① ② ④ ⑤ ⑥ ⑦ ⑨ ⑩

Original time

time

① ② ③ ④ ⑤ ⑥ ⑦ ⑧ ⑨ ⑩

stop

Fundamental period is extended and pitch becomes lower

Conventional

×

① ② ③ ④ ⑤ ⑥ ⑦ ⑧ ⑨ ⑩


Speech rate conversion without

changing length

34

Start is coincided at

blue line positions

Start is not coincided but… Again it coincides

Original

Converted


Original

（n times） BGM speech silent

Important part（speech）

Speech rate conversion for visually

impaired people

35

Understandable high speed speech

Visually impaired people usually understand PC screen

by using screen reader ⇒ they want to hear like quick

reading of sighted people

Make this part easier to understand

Converted （same length)

Make slower Make slower

time


Applications of speech rate conversion

36

Mobile terminal

iPhone app.

For language learning and audio book

For PC

Plug-in for Windows media player

Variable speed with rip synchronization

Variable speech for audio and visual content

on the internet

DAISY Player for visually impaired

people

Applicable both to CD player and

software

※DAISY：Digital Accessible Information SYstem




[Speech Processing]

37



38

Home

Speech

enhance

Deriving

speech

output

Loudness

control

Developing as receiver’s

functionalities

Estimated

speech

Estimated

background sounds

input

Speech (narration etc.)

Background

sounds

Mixed

sounds

Mixing balance meter

Studio

Calculating

loudness &

Estimating

favorability of

mixing

balance



technology for foreigners

[“Easy Japanese”]

39


Research on “Easy Japanese” for

foreigners living in Japan

40

Registered foreigners : 1.7% of pop. in 2009.

Increase of foreigners population

Difficulty of enriching multi-lingual services

Developing Web news services by

“Easy Japanese”:understandable

for foreigners in Japan

Research issues:

・Establish methods to re-write “Easy

Japanese” from news manuscripts in

usual Japanese.

・Develop authoring tools for “Easy

Japanese”

(Press-release from the ministry of justice)


An experimental service started on

2nd

April 2012

41

http://www3.nhk.or.jp/news/easy/

Uses short sentences and

easy words

All Kanji characters have

Furigana (phonetic characters)

Underlined words have

dictionary explanation


Authoring tool for “Easy Japanese”

42

Rewriting is done by a team of a

Japanese language teacher and

a reporter.

The tool assists the team work.

Authoring tool can measure the

text difficulty and show it in

index figures and color.


Other aspects of accessibility

for broadcasting

[Emergency Warning

Broadcasting System (EWBS)]

43


What is EWBS ?

44

Remote activation of Radio and TV receivers ready for EWBS

AM, FM Radio and TV : Control signal and Alert Sound

Digital TV Broadcasting “ISDB-T” :

EWBS Control (Activation Flag)

In Japan, EWBS has been operated since September 1985.

EWBS on ISDB-T has been operated since December 2003.

EWBS test signals are monthly broadcast in Japan.

Broadcasting Station

Transmitter Broadcasting Service Area

Automatic Activation

Meteorological Agency


Mechanism of EWBS for transmitting

EWBS signal sent by on air signal

start signal ; switches on receivers

end signal ; turns off receivers

Analog system for Radio &TV ; 640Hz & 1024Hz frequency

Digital system TV ; digital data as flag on ISDB-T

45


Functions of EWBS in disaster relief

1. Gathering/receiving disaster information from

authorized organizations

2. Delivering disaster information to the public general

Advantages of EWBS

1. Always connected to everybody, send signal ASAP

There are no congestions like telephone lines

2. Most cost efficient system

One way “Delivery” system is enough.

(No two-way radio communication system is required.)

46


Towards/beyond

Universal Broadcasting

47


Towards Universal Broadcasting

48

media

se

rvic

e

/fu

nctio

na

lity

Closed caption

Read-out

….

Sign language

Audio description

Speech rate conversion

Broadcasting

To support variety of services/functionalities:

Hybridcast ®

(a platform of collaboration of broadcasting

and communication)


Enriched services

Broadcasting media (Simultaneous, High quality, Trustworthy)

Program

49

Hybridcast ® receiver

Program-related inf.

Internet service （Cooperation with

broadcasting）

Communication media (Applicable to personal request)

DRM technology

Synchronization of

various information

Coop. with mobile term.

®

Demand, SNS input

Concept of Hybridcast ®

Broadcaster

Service provider



50

Technology for viewer adaptation

Context

Who?

Demand?

Interest?

Situation?

Broadcaster

Content provider

Context-aware

•Content provision

•AV signal adaptation

CG

sign language

Accessible technology for the impaired

CC by speech

recognition

AD script

editor

Accessible technology for the elderly, foreigners

答え：後に同じことが書いてあるので省略しましょう

夏に電力が足りなくなるかもしれない問題で、人々の節電への関心が高くなったため、扇風機が良

く売れています。質問：「夏に．．．高くなったため」

の部分はいりますか？

現在の難しさ 2500

“Easy Japanese ” for foreigners

Multi-modal

display



Beyond Universal Broadcasting

51

media

se

rvic

e

/fu

nctio

na

lity

Closed caption

Read-out

…

Sign language

Speech rate conversion

Broadcasting Web, VOD, BB DVD

For media-independent data sharing： Standardization (ITU, SMPTE, W3C, etc.)

Universal media services

⇒Society free from information barrier

Mobile

Audio description


Closing remarks

52

Introduced researches on accessible technologies

Technologies for impaired people

Technologies for the elderly, foreigners

Other aspects of accessibility for broadcasting

In order to realize the society free from information

barrier


Based on accessible technologies

Hybridcast ® as a platform

Viewer adaptation

Beyond Universal Broadcasting

Expansion to other media


53

Thank you very much!!

Date post:	22-Dec-2021
Category:	Documents
Upload:	others
View:	4 times
Download:	0 times