Fast Inter Mode Decision Based on Textural Segmentation and Correlations for Multiview Video Coding
Wei Zhu, Xiang Tian, Fan Zhou and Yaowu ChenIEEE TCE, 2010
2
Outline Introduction Observation and Analysis Proposed Fast Inter Mode Decision Experimental Results Conclusion
3
Introduction Joint Video Team(JVT) has developed a Joint Multiview
Video Model(JMVM) which is based on H.264/AVC to compress multiple video data efficiently.
Fig.1 shows an illustration of prediction structure employed in JMVM(Hierarchical B prediction structure).
Primary view Auxiliary view
ME DE
ME +
4
Introduction MVC uses Rate-Distortion Optimization(RDO) technique to
select the optimal coding mode which is time consuming.
The coding structure in MVC are more complex than H.264/AVC, especially for auxiliary views employing inter-view prediction.
The prediction directions could be decided to reduce the computational complexity of auxiliary views.
The correlations between views can be exploited to further speed up the mode decision of MVC.
5
Observation and Analysis Use JMVM 4.0 with the full mode decision to analyze modes
features.
Analysis
Skip mode occupies the largest proportion among all MB modes
a large proportion background no inter estimation, faster determine to be Skip mode as fast as possible
Inter8x8 mode occupies the smallest proportion among Inter modes
consume 50% encoding time reduce the estimation of Inter8x8
6
Observation and Analysis Inter_DE mode only occupy a small proportion, but the
computation time consumed by disparity estimation for inter-view prediction occupies more than 50% of encoding time.
reduce disparity estimation as much as possible
Intra mode occupies a small proportion among all MB modes, and computation is also small.
implement it before Inter mode decision, the results can be used for the optimization of Inter mode decisions
7
Observation and Analysis Target: exploit the correlation between Intra mode and Inter
mode, and the correlation between textural complexities and RD costs.
Analysis
Both Intra mode and Inter mode have higher RD costs in higher texture region
Both Intra mode and Inter mode have lower RD costs in lower texture region
Inter/Intra mode RD costs are sensitive to textural regions
Intra mode RD costs can be used for the segmentation of textural region※ Blocks with higher values of RD costs are
darker and lower values are lighter.
Intra Inter
8
Proposed Fast Inter Mode Decision① Reference views’ Intra/Skip RD
costs info are transferred to the current view.
② Current frame is segmented into three textural regions.
③ Do Intra, Skip, and Inter16x16 modes.
Analysis
Intra : get its RD cost to obtain the textural region type of MB
Skip : get its RD cost or the early decision of Skip mode
Inter16x16 : get results of ME and DE, and use these results to decide if to do DE/Inter8x8 or not
9
Proposed Fast Inter Mode DecisionA. Segmentation of Textural Regions Reference views’ Intra mode RD costs are used to segment
textural regions. High textural area Intra4x4 mode Low textural area Intra16x16 mode
Intra mode is classified into INTRA4 and INTRA16 INTRA4 : Intra4x4, Intra8x8 and PCM modes high texture INTRA16 : Intra16x16 low texture
10
Proposed Fast Inter Mode Decision Sort RD costs of INTRA4 and INTRA16 from low to high
INTRA16 mode RD cost are in a relative
low level
INTRA4 mode RD cost are in a relative
high level
The variation of RD costs with the same Intra mode among views are consistent
The distribution of all Intra mode RD costs can be divided into three segments the low, the high, and the in-between values
11
Proposed Fast Inter Mode Decision Calculate thresholds of the textural segmentation:
Use neighboring views Intra RD costs
○ k : index of neighboring reference views
Obtain segmentation threshold
For each MB in current frame, obtain its textural region type
○ n : index of MB
12
Proposed Fast Inter Mode Decision Distribution of textural regions in Race1 sequence is shown
in Fig.6
High textural region
Median textural region
Low textural region
Distributions of textural regions are varying with different QP, it is more suitable for Inter mode decision
13
Proposed Fast Inter Mode Decision Idea : Considering that RD costs have different sensitivities to
textural complexities, so the RD cost threshold should be calculated for different textural regions.
Numbers of Skip mode selected by full mode decision:
The numbers are similar between frames with the same QP
B. Early decision of Skip Mode
14
Proposed Fast Inter Mode Decision
15
Proposed Fast Inter Mode Decision Average Skip mode RD cost curves:
low texture
always above high texture
Values are similar at same time instance
16
Proposed Fast Inter Mode Decision Skip mode RD cost includes only distortion part. It is more relevant
to textural complexity of MB. The relationship between and :
Early decision of Skip mode:
① if Skip mode has minimum RD cost among Intra, Skip, Inter16x16.
② Its RD cost is less than its corresponding threshold ()according to textural region type.
Not used for anchor frames
17
Proposed Fast Inter Mode DecisionC. Selection of Disparity Estimation The correlation of selected inter-view prediction between Inter16x16
mode and all Inter modes was analyzed:
Prediction results of INTER16x16 mode can be utilized to guide the selection of DE for other Inter modes
※ and
Inter16x16 All Inter mode
Choose Inter-view prediction Choose temporal prediction
Enabling of disparity estimation for other Inter modes:
1.01.11.2
18
Proposed Fast Inter Mode Decision
A distribution of Inter8x8 mode selected by the full mode decision:
The proportions of Inter8x8 mode in different textural regions:
Most of Inter8x8 mode appear in regions with more textural complexity
There are few Inter8x8 mode in backgrounds with no motion activity
D. Reduction of Inter8x8 Mode Estimation
Most of Inter8x8 mode are in the high textural region, very few in low textural region
19
Proposed Fast Inter Mode Decision Enabling of Inter8x8 mode estimation:
※ is the motion vector of MB
20
Experiment Results
Software version JMVM4.0
Search mode Fast search
Search range 96
QP 22, 27, 32, 37
Test sequence Exit, Ballroom, Race1, Flamenco2, Rena
Number of view 2 or 3
Compare with Full mode decision in JMVC, Shen’s algorithm [1]
[1] L. Q. Shen, Z. Liu, S. Liu, Z. Y. Zhang, and P. An, “Selective disparity estimation and variable size motion estimation based on motion homogeneity for multi-view coding,” IEEE Trans. Broadcasting, vol. 55, no. 4, pp. 761-766, Dec. 2009.
21
Experiment Results For the early decision of Skip mode, parameter was determined by
experimental analysis:
※
22
Experiment Results Compare to the full mode decision:
23
Experiment Results Compare to Shen’s algorithm
24
Experiment Results Compare to Shen’s algorithm
25
Conclusion A fast Inter mode decision is proposed to reduce the
computational complexity of MVC.
Skip mode is early decided, the disparity estimation is selected, and the Inter8×8 mode estimation is reduced.
As compared to the full mode decision about 55~70% time saving with negligible loss of coding efficiency.
The proposed algorithm could be combined with fast motion search and disparity search algorithms to further save the computation time.