Basic Usage

Vectorly's Background Filter takes as an input any MediaStream or MediaStreamTrack element, so for a WebRTC application, all you need to do is to instantiate the filter object with the MediaStream or MediaStreamTrack element you want to filter. The output is a MediaStreamTrack, which can be sent via WebRTC or loaded locally into a video element

The basic API for loading the background filter is:

   const stream = await navigator.mediaDevices.getUserMedia({video:true, audio:true});
   const filter = new BackgroundFilter(stream, {token: 'vectorly-token', background: 'blur'});
   const outputStream =  await filter.getOutput();
					

The above example is for a Background Blur filter. For virtual backgrounds, where you replace the background with an image, the API is:

   const filter = new BackgroundFilter(stream, {token: 'vectorly-token', background: 'my-image-url.png'});
					

You can find a full set of methods and parameters on the API reference page

Browser support

Our background filters are supported on all major browsers, except for internet explorer. See a table of browser support below

	Chrome	Safari	Firefox	Edge	Opera	IE
Supported	Yes	Yes	Yes	Yes	Yes	No
SIMD acceleration	Yes, since 91	No	Yes, since 89	Yes, since 84	Yes, since 77	No
Offscreen Support	Yes	No	No	Yes	Yes	No

Offscreen Support
Offscreen support uses OffscreenCanvas to run video-processing workloads in a worker. This both has some performance benefits and is also necessary to continue running the AI filter when the current tab is hidden or minimized.

Recent versions of Chrome, Firefox, Edge and Opera support SIMD acceleration since June 2021. SIMD acceleration enables much higher framerates and low CPU overhead. SIMD acceleration is enabled automatically by default, if it is supported by the browser.

Integration

Integrating the filter with any specific Video Conferencing API or service just requires finding the MediaStream element associated with video stream you want to filter. The following sub-sections discuss how to integrate the filter with various conferencing services.

Vanilla WebRTC

As shown above, the API for basic/general WebRTC is:

   const videoTrack = await navigator.mediaDevices.getUserMedia({video:true, audio:true}).getVideoTracks()[0];
   const filter = new BackgroundFilter(videoTrack, { background: 'image.jpg'});
   const virtualBackgroundTrack = await filter.getOutputTrack();

You can find a demo repository for vanilla WebRTC background filtering here

Jitsi

You can enable filters on any VideoTrack object by feeding into the Filter's Jitsi Plugin (see reference).

   const room = connection.initJitsiConference('conference', confOptions);

   JitsiMeetJS.createLocalTracks({devices: ['video']}).then(tracks => {
      const filter = new BackgroundFilter(tracks[0], {token: 'insert-vectorly-token', background: 'blur'});

	filter.getOutput().then(function(filteredStream){
            room.addTrack(filteredStream);
        });

   });

Agora

For Web deployments using Agora (specifically the 4.x API), you can just feed the video track to the Background filter, which will return a filtered video track which you can publish.

    let videoTrack = AgoraRTC.createCameraVideoTrack();
    let audioTrack = AgoraRTC.createMicrophoneAudioTrack();


    const filter = new BackgroundFilter(videoTrack._mediaStreamTrack, {token: 'insert-vectorly-token-here', background: 'blur'});
    filter.getOutputTracj().then(function(filteredTrack ){

       const filteredAgoraTrack = AgoraRTC.createCustomVideoTrack({
         mediaStreamTrack: filteredTrack
       });
      client.publish([filteredAgoraTrack, audioTrack]);
    });

						

You can find a working demo repo here

Twilio

You can enable filters on any VideoTrack object by extracting the raw MediaStreamTrack, running the filter on that, and creating a new LocalVideoTrack.

   const Twilio = require('twilio-video');

   const localVideoTrack = await Twilio.createLocalVideoTrack();
   const filter = new BackgroundFilter(videoTrack.mediaStreamTrack, {token: 'insert-vectorly-token', background: 'blur'});
   const outputTrack = await filter.getOutputTrack();
   const filteredTrack = new Twilio.LocalVideoTrack(outputTrack);

   room.localParticipant.publishTrack(filteredTrack);

							

Daily.co

If you're building a custom UI with Daily.co, you can use the Daily.co callObject's setInputDevices method to set the filtered video track as the upload stream.

   const sourceVideoTrack = callObject._participants.local.videoTrack;

   const filter = new vectorly.BackgroundFilter(sourceVideoTrack, {token: 'your-vectorly-token', background: 'https://demo.vectorly.io/virtual-backgrounds/1.jpg'});

   filter.getOutputTrack().then(function(filteredTrack ){

      callObject.setInputDevicesAsync({
         videoSource: filteredTrack
      });

   });
							

You can find a working example repo here

Vonage / OpenTok

When you create a Publisher object, just feed the filtered video Track as the video Source

   const stream = await navigator.mediaDevices.getUserMedia({video:true, audio:true});
   const filter = new BackgroundFilter(stream, {token: 'vectorly-token', background: 'blur'});
   const outputTrack =  await filter.getOutpuTrack();

   var publisher = OT.initPublisher('publisher', {
      insertMode: 'append',
      width: '100%',
      height: '100%',
      videoSource: outputTrack
   }, handleError);
							

You can find a working demo repo here

Browser support

Our upscaling filters are supported on all major browsers, except for internet explorer. See a table of browser support below

	Chrome	Safari	Firefox	Edge	Opera	IE
Supported	Yes	Yes	Yes	Yes	Yes	No

Integration

General WebRTC

The UpscaleFilter works with any video tag, so for a WebRTC application, all you need to do is to instantiate the upscaler object with the video element you want to upscale.

   const upscaler  = new UpscaleFilter(document.getElementById("remoteVideo"), {token: 'insert-vectorly-token-here'});

We have an example repository, showing how Vectorly can be integrated with WebRTC, as well as a full working general WebRTC demo here.

Integrating the upscaler with any specific Video Conferencing API or service just requires finding the video element associated with video stream you want to upscale.

Jitsi

You can enable upscaling on any VideoTrack object by intercepting the corresponding video element you attach it to (see reference).

   const room = connection.initJitsiConference('conference', confOptions);
   room.on(JitsiMeetJS.events.conference.TRACK_ADDED, function(track){

      const videoElement = document.createElement('video');
      document.body.appendChild(videoElement);
      track.attach(videoElement);

      const upscaler = new UpscaleFilter(videoElement.current, {token: 'insert-vectorly-token'});

   });

Agora

For Web deployments using Agora, you can find the video element of the stream you want to upscale by using the stream's ID.

    let stream = AgoraRTC.createStream({
        streamID: uid,
        audio: true,
        video: true,
        screen: false
    });

    stream.init(function() {

        stream.play('target-div');
        const video = document.getElementById("video" + stream.getId());
        const upscaler = new UpscaleFilter(video, {token: 'insert-vectorly-token-here'});

        client.publish(stream);

    });
						

Amazon Chime SDK

For the Amazon Chime SDK, you'll need to use the low level controls library, in conjunction with the Amazon Chime Video Processor API

    if(!this.upscaler){

        this.upscaler = new UpscaleCoreFilter();

        const config = {
          w: frameWidth,
          h: frameHeight,
          renderSize: {w: frameWidth*3, h: frameHeight*3},
          canvas: this.targetCanvas,
          networkParams: {name: 'residual_5k_3x', tag: 'general', version: '0'},
          token: ""
        }

       this.upscaler.load(config);

        this.upscaler.on('load', function(){
          this.upscalerReady = true;
        }.bind(this));

    }

    if(this.upscalerReady){
      this.upscaler.setInput(canvas) // Sets input element
      this.upscaler.render() // Renders to canvas
    }

						

The Amazon Chime Video Processor API provides an canvas as your input, and provdes a destination canvas as your output. All you need to do is use the vectorly core library, configure it to render to the output canvas, and feed it the input canvas and render on each render cycle

The above is a code snipped taken from our full working demo repository, which you can find here

Twilio

You can enable upscaling on any VideoTrack object by intercepting the corresponding video element you attach it to (see reference).

If you use the track.attach() method to create a video element:

   const Video = require('twilio-video');

   Video.createLocalVideoTrack().then(function(videoTrack) {
     const videoElement = videoTrack.attach();
     document.body.appendChild(videoElement);
     const upscaler = new UpscaleFilter(videoElement.current, {token: 'insert-vectorly-token'});
   });
							

If you specify your own video element:

   const Video = require('twilio-video');

   const videoElement = document.createElement('video');
   document.body.appendChild(videoElement);

   Video.createLocalVideoTrack().then(function(videoTrack) {
     videoTrack.attach(videoElement);
     const upscaler = new UpscaleFilter(videoEl.current, {token: 'insert-vectorly-token'});
   });
							

OpenTok / Vonage

When a Subscriber or Publisher creates a video element you can intercept it and feed that video to the Vectorly Upscaler.

You can use the subscriber.element property to intercept the video element

    session.on('streamCreated', function(event) {

        const subscriber = session.subscribe(event.stream, 'subscriber', {
            insertMode: 'append',
            width: '100%',
            height: '100%'
        }, handleError);

        subscriber.on('videoElementCreated', function (){
            const video = subscriber.element.querySelector('video');
            const upscaler  = new UpscaleFilter(video, {token: '..your-token....'});

        });

    });
							

This also works with a publisher object. Refer to the Vonage documentation for styling - Vectorly's upscaler will fit within the styling defined by OpenTok.

See our example repistory for a working code example

Daily.co

You can integrate Vectorly's AI upscaler with Daily.co if you're building a custom custom video chat interface . Using the default React code sample from Daily, we've built a full working demo reference

   useEffect(() => {
	videoEl.current &&
	(videoEl.current.srcObject = new MediaStream([videoTrack]));
	if (videoEl.current && props.isLarge) {
	window.upscalers = window.upscalers || {}
	window.upscalers[videoTrack.id] = new UpscaleFilter(videoEl.current, {token: 'insert-vectorly-token'});
	}
   }, [videoTrack]);
							

You just need to make sure you intercept the video element associated with the video track you want to upscale.

Vectorly's AI upscaler is not compatible with the pre-built UI from Daily.co, as the pre-built UI is loaded via iframe, making it impossible to access the video element through a third party application.

Electron

If you're building an electron app, the Vectorly library is fairly plug and play, and will work with either CDN or NPM installation.

You can see a demo electron app repostory here

Background Segmentation

Our background filter uses the Google Selfie model by default. We also have an experimental WebGL based background-segmentation model which has significantly less CPU usage than any existing alternatives (see here for more info).

You can verify performance for yourself with a profiling tool like Google Chrome's performance profiler

Performance Monitoring

For checking the performance on a given device, you can use the static checkPerformance method without actually instantiating it on a screen.

							 
     const performanceResults  = await BackgroundFilter.checkPerformance({token: '...'});
							 
						 

This cycle through a 720p input for 2 seconds and return an object with the following fields

							 
	{
	  "inputSize": "1280x720",
	  "frameCount": Number,
	  "time": Number,
	  "fps": Number
	}
							 
						 

You can also measure the live fps on any given stream using the filter.processer.metrics.fps propery as shown below:

							 

   const filter = new BackgroundFilter(video, config);

   console.log(`The current fps is ${filter.processor.metrics.fps}`);
							 
						 

You can see a working code example in our Background Demo code, the same code used for our public demo

Performance Considerations

We recommend only running the Web Background filters library for desktop users. The performance is considerably worse on Mobile because because the overhead of communicating between the browser and GPU is much higher on mobile devices.

We expect the performance to rival or exceed desktop clients with native mobile SDKs, as is the case for our AI Upscaler Android SDK. We are planning to develop a Background Filters Android and iOS SDK in Q3 2021.

AI Upscaling

The primary "cost" to doing super-resolution is computational complexity. While we have put a lot of work into making super resolution feasible on client devices, it is still something which needs to be managed. Here, we provide some initial performance benchmarks for the same demos shown above, in the demos sections.

Performance Considerations

AI Upscaling does require some computational effort, however it is mostly on the graphics card, so AI Upscaling's impact on CPU is limited. The amount of computation (and therefore the framerate / performance) depends on the size of input video you are upscaling

The following table should give a rough idea performance for different input video resolutions. These results are only for Web environments. Our mobile SDKs will have access to more powerful native libraries, enabling significantly better performance.

You can measure the fps on any given device by adding the analyticsEnabled flag as true in the configuration parameters.

							 
  const config = {
	   token: '...',
	   analyticsEnabled: true
   };

   const upscaler = new UpscaleFilter(video, config);
							 
						 

You can then measure the fps at any time with upscaler.metrics.fps property. The fps number provided by upscaler.metrics.fps will not exceed the source video's frame rate because we only render when a video frame changes.

It's recommended to stick to 240p or 360p inputs, as mid-range devices tend to struggle with larger inputs. You can also disable upscaling if the fps gets too low.

Quality

The primary benefit of Super resolution is to increase video quality. Using the original high-resolution video as a reference, we can use traditional video quality metrics like VMAF to quantify the quality improvement of Super Resolution, when compared to normal bicubic upscaling of the downsampled / low-resolution video content.

Our general AI upscaler filter generally achieves a 10 to 15 point VMAF improvement compared to bicubic scaling. With content-specific AI models, or heavier models, we will likely be able to achieve further quality gains. We are currently working on releasing quality comparisons for content specific models.

Quality visualization

For reference, below are side by side comparisons of bicubic upscaling of the low-resolution original / Super resolution of the low-resolution / High resolution original

Jellyfish

Ducks

Tractor