The SP- and SI-Frames Design for H.264/AVC

Marta Karczewicz and Ragip Kurceren

IEEE Trans. on Circuit and System for Video Technology, Vol.13, No. 7, July 2003IEEE Trans. on Circuit and System for Video Technology, Vol.13, No. 7, July 2003

(Nokia Research Center)

Outline

What’s SP- and SI-Frames Motivation Decoding and Encoding Processes for SP-

and SI-Frames Results

What’s SP- and SI-Frames (1/2)

SP-frames make use of motion compensated predictive coding to exploit temporal redundancy in the sequence similar to P-frames.

The difference between SP- and P-frames is that SP-frames allow identical frames to be reconstructed even when they are predicted using different reference frames.

What’s SP- and SI-Frames (2/2)

SP-frames can be used instead of I-frames in such applications as bitstream switchingbitstream switching, splicingsplicing, random arandom accessccess, fast forwardfast forward, fast backwardfast backward, and error resiliencerror resilience/recoverye/recovery.

SP-frames require significantly fewer bits than I-frames to achieve similar quality.

In some of the mentioned applications, SI-frames are used in conjunction with SP-frames. An SI-frame uses only spatial prediction as an I-frame and still reconstructs identically the corresponding SP-frame, which uses motion-compensated prediction.

Frame types in the existing standards

I- , P-, and B-Frames

I P P P P P I P …

I B P I B B P …B

Applications for SP- and SI-Frames(1)

Bitstream switching

P2, n-2 P2, n-1 S2, n P2, n+1 P2, n+2

S 12, n

P1, n-2 P1, n-1 S1, n P1, n+1 P1, n+2

Bitstream 1

Bitstream 2

Secondary SP-frame

PrimarySP-frame

Applications for SP- and SI-Frames(2)

Splicing and Random Access

P2, n-2 P2, n-1 S2, n P2, n+1 P2, n+2

SI 2, n

P1, n-2 P1, n-1 S1, n P1, n+1 P1, n+2

Bitstream 1

Bitstream 2

Applications for SP- and SI-Frames(3)

Error Recovery/Resiliency

P1, n-2 P1, n-1 S1, n P1, n+1

S 21, n

SI1, n

P1, n-3

Applications for SP- and SI-Frames(4)

Video Redundancy Coding Sync frame

1 2 3 4 5 6 7 8 9 10 11 12 13

3 4 5 6 7 8 9 10 11 12 13 14 15

5 6 7 8 9 10 11 12 13 14 15 16 17

Decoding and Encoding Processes for SP- and SI-Frames

Dem

ultip

lexi

ng

InverseQuantization

InverseTransform

FrameMemory

MCPrediction

IntraPredition

Motion InformationIntra Prediction Mode

Generic block diagram of decoding process.

+PQP

Generic block diagram of decoding process for secondary SP- and SI-frames

Dem

ultip

lexi

ng

+Inverse

QuantizationInverse

Transform

FrameMemory

MCPrediction

IntraPredition

Transform

Quantization

SPQP

SPQP

Motion InformationIntra Prediction Mode

dreclrec

lpred

lerr

lc

RP

Generic block diagram of decoding process for primary SP-frames

Dem

ultip

lexi

ng

+Inverse

QuantizationQuantization

FrameMemory

MCPrediction

Transform

PQP

Motion Information

crec

cpred

derr

lc

RP

InverseTransform

InverseQuantization

SPQPdrec

lrec

SPQPlerr

Generic block diagram of encoding process for nonintra blocks in SP-frames

Multiplexing

InverseQuantization

Transform

MotionEstimation

FrameMemory

+

SPQP

corig Quantizationcerr

dpredPQP

Quantization

Transform

lpred

P

Motion Information

R

Inverse Quantization &Inverse Transform

Quantization

InverseQuantization

+

SPQPlrec

cpred

derr PQP

lerr

-

To generate secondary representation of the primary SP-frame lc

Dem

ultip

lexi

ng

+Inverse

QuantizationQuantization

FrameMemory

MCPrediction

Transform

PQP

Motion Information

crec

cpred

derr

lcR

P

InverseTransform

InverseQuantization

SPQPdrec

lrec

SPQPlerr

F1,n => F2,n

lerr,2 = lrec – lpred,2

Generic block diagram of encoding process for nonintra blocks in SP-frames

Multiplexing

InverseQuantization

Transform

MotionEstimation

FrameMemory

+

SPQP

corig Quantizationcerr

dpredPQP

Quantization

Transform

lpred

P

Motion Information

R

Inverse Quantization &Inverse Transform

Quantization

InverseQuantization

+

SPQPlrec

cpred

derr PQP

lerr

-

lerr,2 = lrec – lpred,2

lpred,2

lerr,2

lc

Generic block diagram of decoding process for secondary SP- and SI-frames

Dem

ultip

lexi

ng

+Inverse

QuantizationInverse

Transform

FrameMemory

MCPrediction

IntraPredition

Transform

Quantization

SPQP

SPQP

Motion InformationIntra Prediction Mode

dreclreclpred

lerr

lc

RP

lpred,2

F2,n

lerr,2

lerr,2 + lpred,2 = (lrec – lpred,2)+ lpred,2 = lrec

lerr,2 = lrec – lpred,2

Results – coding efficiencies

Results – coding efficiencies when inserted periodically

Results – Comparison with S-Frame (Drifting

effect)

Results – Comparison with S-Frame (Size)

The SP-frame is 3.4times smaller than the S-frame in average.

6.2 times

Results – PSNR and Total bits over 100 frames (4 switches)

SP or I Frame rate = 1fs

Results – PSNR and Total bits over 100 frames (no switches)

SP or I Frame rate = 1fs

~ The End ~

Thank you!!