Graphics and Media Lab

MSU Graphics & Media Lab (Video Group)

The algorithm allows to increase video frame rate (fps) by the factor equal to any integer number. The main feature of this method is high robustness with regard to motion complexity.

Task

Frame rate conversion is done as shown on the picture below:

Frame rate conversion scheme

The main advantages of this method are:

• high accuracy of motion tracking and compensation;
• low level of artifacts even in case of complex motion;
• occlusions detection and processing;
• arbitrary whole number of interpolated frames between reference frames.

First is achieved by construction of compensated frames at quarter pixel accuracy. Segmentation of original frames with detection of areas with complex motion allows to avoid visual degradation of interpolated frames even on scenes with hard motion. Explicit scheme of occlusions detection improves visual quality of interpolated frames in areas near the frame edges.

Examples

Pictures series below illustrates method performance for sequence "susi". In the first row there are previous and next reference frames of original sequence. Second row contains interpolated frame located between above frames in temporal domain. Focus your attention at hears area in the right half of the frame: there are no visible artifacts in spite of fast motion:

Previous reference frame	Next reference frame

Interpolated frame

Visual comparison results of several FRC methods and developed one are presented below. Sequence "stefan" was used for comparison. Fast movements of the player are the significant obstacle for FRC algorithm so artifacts will probably appear near edges of the player silhouette. The additional problem is the head of the player. Because it is the small detail it is hard to calculate correct motion parameters for it. These are the reasons why many artifacts are produced on the showed below interpolated frames. As can be clearly seen proposed method produces the lowest level of artifacts comparing to other methods (this is especially evident on the player face):

Previous reference frame	Next reference frame
MotionPerfect result	Retimer result
Twixtor result	Proposed method result

Objective comparison results are presented on the graphs below. Y-PSNR metric was used for comparison. In metric calculation only interpolated frames have been used. First two diagrams shows results for 'flower' and 'stefan' test sequences. In order to calculate Y-PSNR metric these sequences was first decimated and then decimated frames was recovered using FRC. After that interpolated frames was compared with collocated frames in original sequences.

Y-PSNR results for sequence 'flower'

Y-PSNR results for sequence 'stefan'

The diagram below shows the average results of processed sequences for various methods. Developed algorithm outperforms the others in most cases:

Objective Comparison

Competitors

• MotionPerfect, DynaPel Systems Inc.
• Retimer, REALVIZ S.A.
• Twixtor, RE:Vision Effects Inc.

See also

Video Frame Rate Double Up-Conversion

Team

• Dmitriy S. Vatolin, Lomonosov Moscow State University