HopperRender

An optical flow frame interpolator with DirectShow integration that allows you to watch any local media file at the native refresh rate of your monitor. This is my first project using DirectShow and CUDA, which has now transitioned to OpenCL for better cross-platform compatibility. The goal is to achieve pretty decent frame interpolation with a variety of user customizable settings. The filter can be added to a DirectShow media player like MPC-HC or MPC-BE.

Please note that this project is no longer in development and is superseded by my mpv frame interpolator. However, I will occasionally port significant improvements over to this project.

Features

• Realtime frame interpolation of any source framerate to the native refresh rate of your monitor
• Compatible with HDR video (usage of madVR recommended)
• All resolutions (even DVDs and 4K Blu-rays) are supported
• No installation or internet connection required
• Cross-platform compatible with most NVIDIA and AMD cards
• Warps frames in both directions and blends them for the smoothest experience
• User customizable settings to adjust the quality and presentation of the interpolation
• HSV Flow visualization lets you see the calculated movements of objects in a scene
• Automatically adjusts internal settings to match the PC's performance
• Compatible with madVR, Enhanced Video Renderer, MPC-Video Renderer, and more
• Automatically detects the source frame rate (as well as playback speed) and disables interpolation if not needed

How to get started?

To use this filter, you need to use a DirectShow player like MPC-HC or MPC-BE.

If you decide to use MPC-BE, please install LAVFilters and add the LAV Video Filter to your filters in the player settings. This will ensure proper compatibility with HopperRender.

The usage of madVR is recommended, but not necessary.

MPC Video Renderer and madVR are officially supported. Using the Enhanced Video Renderer is not advised, as it doesn't support HDR and tends to be incompatible with HopperRender.

Installation

1. Open MPC-HC or MPC-BE's settings.
2. Select the External Filters submenu.
3. Click on Add Filter.

4. Click on Browse.

5. Select the HopperRender.dll file.

6. Select Prefer.

7. Close the settings and the media player.

That's it! You can now play a video with MPC-HC/BE and HopperRender will interpolate it to 60fps or more.

Note: Do not move or delete the folder containing the HopperRender.dll file, otherwise the media player won't find it.

Settings

You can access the settings when playing back a video with HopperRender by right clicking on the video in MPC-HC/BE, selecting the Filters menu and then HopperRender.

• You can activate and deactivate the interpolation

• You can select which type of frame output you want to see:

  • Warped Frame 1 -> 2: Shows just the warping from the previous to the current frame
  • Warped Frame 2 -> 1: Shows just the warping from the current to the previous frame
  • Blended Frame: Blends both warp directions together
  • HSV Flow: Visualizes the optical flow as a color representation, where the color indicates the direction of movement
  
  • Grey Flow: Visualizes the optical flow as a black and white representation, where the brightness indicates the magnitude of movement
  • Side-by-side 1: Shows the difference between no interpolation on the left, and interpolation on the right (split in the middle)
  • Side-by-side 2: Shows the difference between no interpolation on the left, and interpolation on the right (scaled down side by side)

• You can set a custom target fps that should be interpolated to. Setting this to 0 will automatically use the display refresh rate.

• You can set the Delta Scalar (controls how much the filter will try to move the frames to simulate the motion on screen)

• You can set the Neighbor Scalar (controls how much the filter will be biased by the surrounding motion, higher values will lead to more uniform interpolation but might miss small motions)

• You can set the Black and White levels which allows for level correction if the input has limited range (i.e. the blacks look grey, and the whites are not full brightness)

• In the status section, you can see the current state of HopperRender, the source and target framerate, the frame and calculation resolutions, as well as the calculation times of the optical flow calulcation and frame warping pipeline (both times combined should always be lower than 1/source framerate!)

• The settings will be automatically saved to the registry HKEY_CURRENT_USER\Software\HopperRender so next time the filter is used, it loads the settings automatically

• It is also possible to manually configure the resolution used in the internal optical flow calculation and thus change the load on the GPU. Although this isn't recommended, as higher resolutions actually produce a worse result, and lower resolutions don't provide much benifit as the calculation is already limited to 270p. If you still want to experiment with this, open the registry editor, navigate to HKEY_CURRENT_USER\Software\HopperRender and create a new DWORD value named MaxCalcRes. When setting a value, please make sure to use the Decimal type. The resolution of the input video will be repeatedly halved until the frame height is less or equal to the configured value. This means you can't specify the exact resolution used, but a limit that ensures the used resolution isn't too high. To use the default value of 270, simply delete the DWORD value.

How it works

Note: The following is a very brief overview of how the filter works. It is not a complete, or 100% accurate description of the interpolation process. Refer to the source code for more details.

• To prevent the algorithm from focusing on pixel level details or compression artifacts, we first (depending on the user setting) blur the frames internally to use for the optical flow calculation
• HopperRender uses an offset array that shifts the frame according to the values contained in it
• The offset array has 5 layers that contain different shifts that can be 'tried out' at the same time to find the best one
• The first step involves setting the 5 layers to a horizontal shift of -2, -1, 0, 1, and 2
• Then, the first frame is shifted accordingly and we subtract the y-channel difference of the shifted frame to the unshifted next frame
• We then reduce all the absolute pixel deltas to one value and find out which layer (i.e. which horizontal shift) contains the lowest value and therefore difference
• Depending on the resulting layer index, we can either move on to the same procedure for the vertical movement, or continue moving in the negative or positive x direction
• We repeat this process until we are certain we found the best offset, or are out of calculation steps
• After having found the best offset for the entire frame, we decrease our window size to a quarter the size and continue the search again for every individual window starting at the previous position
• Depending on the user setting, we do this until we get to the individual pixel level
• Finally, we flip the offset array to give us not just the ideal shift to warp frame 1 to frame 2, but also to warp frame 2 to frame 1
• We then blur both offset arrays depending on the user settings to get a more smooth warp
• Then we use these offset arrays to generate intermediate frames by multiplying the offset values by certain scalars
• We add a bit of artifact removal for the pixels that weren't ideally moved and blend the warped frames from both directions together

Acknowledgements

This project is based on the EZRGB24 Filter Sample and the DirectShow core parts were inspired by the LAV Video Decoder.

The Dolby Vision Metadata passthrough code elements were adapted from the MPC Video Renderer.