Couple suggestions for better fallback behavior (HLS vs MP4)

[rafaelrruiz]

I have HLS and MP4 versions of my streams. I'd like to show HLS when its supported and fallback to MP4 when its not. (See below for a fuller description of what I think is a pretty common scenario.) I think there are two changes needed in video.js that would make this possible.

1. The first is to distinguish between HLS content and regular MP4 content by adding "application/x-mpegURL" to the supported MIME types for the Flash tech. MIME-type is the fundamental way we check whether a tech supports a source so it seems logical that we should add this MIME type to the vjs.Flash.formats set.
2. The second is changing the way selectSource works to prioritize source over tech. In the spirit of following the

What do you all think?

Scenario: I have an HLS version of a video with multiple bitrates (cause that often provides a better user experience on a variety of devices and networks) but I also have a vanilla version in MP4 format for devices/platforms that simply can't display HLS.

For example, on Mac Safari or iOS browsers, I'd like to display HLS without the use of Flash (using the built-in

To accomplish this (with the changes above) I could use a techOrder of [ 'html5', 'flash' ] cause the native implementation is always preferable if it will play my content. And a sources array like this:

                sources = [                       
                    { src: content.url, type: "application/x-mpegurl" },
                    { src: content.httpUrl, type: "video/mp4" },                        
                ];

With the changes I describe above, each tech would be checked to see if the first source will play. If none suffice, then the techs are checked for the next source. On systems with native support for HLS or no-native support but Flash enabled, HLS will be chosen. On systems with neither HLS or Flash support, the MP4 is chosen. AND, on old browsers that might not have MP4 support built in, I still have the final choice option of Flash playing the MP4.

[gkatsev]

videojs's swf doesn't support HLS natively, it only supports HLS via the videojs-contrib-hls project. That tech adds itself to the front of the tech list so it gets used before other techs, i.e., HLS source is available and comes before an mp4 source, the HLS will play using the HLS tech. Moreover, if the browser supports HLS natively (desktop Safari, mobile devices), the HLS tech will not do anything and the html5 tech will be able to pick up and play HLS via the native video element of the platform. I think the above workflow falls in line with what you want to do.

Option 2 might be worth considering, maybe as an option to videojs, though. @heff thoughts?

[rafaelrruiz]

I'm currently using the swf file from videojs-swf project instead of
videojs-contrib-hls. My team did a side-by-side comparison last week and
videojs-contrib-hls had a noticeable flicker that was roughly timed with
each chunk that videojs-swf didn't display. [We used some test files we
have that show a clock and a geometric animation. With videojs-contrib-hls
we could see the clock freeze for a fraction of a second on a regular
interval roughly equal to the chunk size. With videojs-swf the clock was
much smoother, having a fractional freeze perhaps once a minute.) We
prefer the architecture of videojs-contrib-hls and are interested in
eventually actually contributing to the project to improve it. But right
now we're on a very tight deadline and needed to choose the best solution
we could get in the very short term.

Regarding the design you describe below - it seems that the HLS tech now
has the responsibility of knowing when a browser supports HLS natively, so
it can step out of the way. Sure it can call into the browser to do that
but that's already done by the HTML5 tech. Seems a little messy and a
by-pass of the selectSource algorithm by overloading the meaning of
canPlaySoruce (assuming that's how you're doing it).

I still think at an architectural design level - its more consistent with

On Wed, Jul 30, 2014 at 8:47 AM, Gary Katsevman notifications@github.com
wrote:

videojs's swf doesn't support HLS natively, it only supports HLS via the
videojs-contrib-hls http://videojs/videojs-contrib-hls project. That
tech adds itself to the front of the tech list so it gets used before other
techs, i.e., HLS source is available and comes before an mp4 source, the
HLS will play using the HLS tech. Moreover, if the browser supports HLS
natively (desktop Safari, mobile devices), the HLS tech will not do
anything and the html5 tech will be able to pick up and play HLS via the
native video element of the platform. I think the above workflow falls in
line with what you want to do.

Option 2 might be worth considering, maybe as an option to videojs,
though. @heff https://github.com/heff thoughts?

—
Reply to this email directly or view it on GitHub
#1384 (comment).

[gkatsev]

the HLS tech asks the browser whether it supports HLS and if it does, the HLS tech will claim it cannot play HLS so that the html5 tech will play it.
The videojs-swf that supports HLS isn't officially supported anymore, since we've switched over to videojs-contrib-hls. However, you should be able to monkeypatch the supported types for your version by adding application/vnd.apple.mpegurl and application/x-mpegurl to

video.js/src/js/media/flash.js

Lines 253 to 263 in 14c64f1

	vjs.Flash.formats = {
	'video/flv': 'FLV',
	'video/x-flv': 'FLV',
	'video/mp4': 'MP4',
	'video/m4v': 'MP4'
	};
	vjs.Flash.streamingFormats = {
	'rtmp/mp4': 'MP4',
	'rtmp/flv': 'FLV'
	};

[gkatsev]

Also, can you please open an issue on videojs-contrib-hls with your findings regarding the flicker? This probably has to do with when we shuttle the chunks over the flash over our Media Source Extension fallback.

[rafaelrruiz]

I will definitely do that tonight when I get home - would like to provide
you with a sample file.

I really love this stuff and want to get more involved. It would be
awesome if you could host a test page like
http://www.flashls.org/videojs/flash_demo.html to try various content with
the player on different browsers.

On Wed, Jul 30, 2014 at 12:36 PM, Gary Katsevman notifications@github.com
wrote:

Also, can you please open an issue on videojs-contrib-hls with your
findings regarding the flicker? This probably has to do with when we
shuttle the chunks over the flash over our Media Source Extension fallback.

—
Reply to this email directly or view it on GitHub
#1384 (comment).

[rafaelrruiz]

That's how I assumed it was implemented. Just pointing out that it seems
hacky that a tech would say it doesn't support something because it knows
some "internal" information about another tech. Making the change to
selectSource I suggest would mean you don't have to insert a tech at the
start of the techOrder (which could come as a surprise to the developer
that specified a specific techOrder) nor would you have to check what the
browser could support. The underlying algorithm would take care of that
for you. In the end you have a working solution so that's cool.

On Wed, Jul 30, 2014 at 12:35 PM, Gary Katsevman notifications@github.com
wrote:

the HLS tech asks the browser whether it supports HLS and if it does, the
HLS tech will claim it cannot play HLS so that the html5 tech will play it.
The videojs-swf that supports HLS isn't officially supported anymore,
since we've switched over to videojs-contrib-hls. However, you should be
able to monkeypatch the supported types for your version by adding
application/vnd.apple.mpegurl and application/x-mpegurl to

video.js/src/js/media/flash.js

Lines 253 to 263 in 14c64f1

	vjs.Flash.formats = {
	'video/flv': 'FLV',
	'video/x-flv': 'FLV',
	'video/mp4': 'MP4',
	'video/m4v': 'MP4'
	};
	vjs.Flash.streamingFormats = {
	'rtmp/mp4': 'MP4',
	'rtmp/flv': 'FLV'
	};

—
Reply to this email directly or view it on GitHub
#1384 (comment).

[dmlap]

@rafaelrruiz: good point about the test page. We should definitely add that to gh-pages for the contrib-hls repo.

I agree that the way the HLS tech interacts with tech selection is a little funky. Our goal is to have it operate like a polyfill which is a bit different from the way techs normally work. We chose this route because we wanted to keep the logic for tech/source selection centralized in video.js. If it turns out that other projects are trying to achieve similar things with their own custom techs, we might have to revisit the problem. Our theory has been that it's an unusual use-case and not worth over-thinking.

Your second proposal does seem more in-line with the spec behavior for the video element. We probably should spend some time making sure it doesn't break any weird scenarios but at first blush I like it. 👍

[heff]

The way video.js does source/tech selection is actually directly influenced by how the video element works. The closest equivalent would be if you embed a flash player inside a video tag, after the source tags. If the browser supports HTML5, it will check if it can play any of the provided sources (via html5). If HTML5 isn't supported it will load the flash player and play the source you provided Flash. Hence the term "Flash Fallback".

This puts more priority on the tech over the source, which isn't the best for every use case, but there's a lot of reasons why you would want to go with HTML5 video over Flash whenever possible, namely performance and stability. The most common HTML5 video examples, at least up to this point, have an MP4, a Webm, and (optionally) an Ogg file. That's a case where you definitely want to test all sources against the HTML5 tech before moving to Flash.

Even if you provide only an HLS source, the mechanism still works, but as soon as you introduce an MP4 to fallback to the priority shifts. And as @rafaelrruiz mentions, that's not an uncommon setup.

But that's at least the reason behind how it's setup and the other side of the argument. I agree we should provide some way to change priorities. The spec doesn't really help us with an API for that, at least that I'm aware of, so we're on our own to come up with it.

If we were to go with an option, I would try to make it a boolean, like prioritizeSources: true. That specifically isn't the most descriptive, but maybe it'd work. Another potential approach is allowing you to place priority on specific sources over others.

We have the HLS+MP4 use case, are there any other use cases that we can pull in here? Preferably more complicated ones.

• DASH + HLS? (a potential future setup for using DASH everywhere but iOS)
• What about when media queries come into the mix? (Support the media attribute on source elements #624)

[ericberdahl]

I concur with @rafaelrruiz that VideoJS feels like it presents an impedance mismatch with respect to html5 video and source selection.

Consider a (very) simple example:

<video>
   <source src=“best” type=“application/x-mpegurl">
   <source src=“better” type=“rtmp/mp4">
   <source src=“good” type=“video/mp4">
</video>

The clear intention of the developer is that the best stream be used if it can be played, better should be used if best is not playable, and good should be used if none of the others are playable. This is nothing more than the basic semantic of the w3c video tag.

VideoJS subtly breaks this semantic when multiple techs are in play. As currently implemented, vjs.Player.prototype.selectSource loops through each tech, in order, asking that tech if it can play any of the sources, in order. Thus, in a situation where the first tech cannot play the best source but can play the better source, and where the second tech can play the best source, the first tech will be selected to play the better source. Even though the best source is playable in this scenario, the better source is what the user will view. Thus, the developers’ intent has been lost by giving tech’s priority over sources.

@heff makes an interesting observation, that VideoJS' selection algorithm makes sense if one considers how flash fallback works. That is, if the (very) simple example above included an embedded flash element after the sources, the html video engine would select any of the sources before falling back to flash. However, I suggest that @heff's argument is invalid because it presupposes that the example were differently formed. Looking at example of a video element with multiple sources, we see that the behavior of pure html and VideoJS is radically different.

As it exists today, VideoJS implicitly purports to be an improved implementation of the html5 video tag, expanding on the built-in implementation in a given browser to provide an improved playback experience (and improved developer experience). Thus, VideoJS's addition of a flash engine is not equivalent to the flash-fallback that an html developer would code, but is an additional playback engine available to the implementation of the developer's video element. Therefore, VideoJS should follow html semantics and respect source order across the supported techs in a browser.

@rafaelrruiz also correctly identifies that changing this semantic, while arguably correct, poses a problem for existing pages, as it could lead to significant changes in source selection and potentially thereby inject bugs into those pages. I suggest that adding an option, much as suggested by @heff and @rafaelrruiz, is the best solution to that problem.

I also concur with @rafaelrruiz's suggestion that a developer be allowed to expand the set of formats supported by the Flash tech. The use case here is a developer who is supplying their own swf, built with support for additional formats. Much as @rafaelrruiz implies, a developer who changes the video-js.swf file used by a given page (using the supported option to point to a custom swf location), should be able to easily expand the set of formats supported by that swf, reusing the remainder of the Flash tech's communication to the swf.

[heff]

I suggest that adding an option, much as suggested by @heff and @rafaelrruiz, is the best solution to that problem.

@ericberdahl, or anyone, do you you have any ideas on what you'd expect that option to look like?

I've confirmed this issue so we just need to design the API and build it.

[ericberdahl]

@heff I propose that we build on the ability, added in VideoJS 4.7.3, for developers to assign the selectSource function. Thus, VideoJS would implement two functions, one that selects tech over source and another that selects source over tech. By default, Player.prototype.selectSource would be assigned to the implementation that selects tech over source (the current implementation), but both implementations would be name-exported such that developers could re-assign in their pages.

// (default) Player.selectSource will prefer tech order over source order.
// This may cause a secondary source to be selected if it is playable by a primary tech
videojs.Player.prototype.selectSource = videojs.selectSourceByTech;

// Player.selectSource will prefer source order over tech order
// This may cause a secondary tech to be used if it plays a primary source
videojs.Player.prototype.selectSource = videojs.selectSourceBySource;

This sidesteps the question of what the option is named, builds on the ability of the developer to re-assign selectSource to whatever algorithm they like, and leaves open the possibility that VideoJS may implement yet other algorithms in the future. It also allows developers to tinker/experiment/hack with their own exotic source selection algorithms (some of which may eventually be proposed as additions to the base VideoJS distribution).

[heff]

I'm going to close this issue because:

• We've exported selectSource so you can override the tech/source priority
• We've released the source-handlers interface and started migrating the adaptive formats to it

The root of this issue was that we didn't have support for HLS in HTML5, and we're getting close to having a solution.

HLS was always awkward as a "tech" because it didn't fit the concept of a tech, like Flash, HTML5, and YouTube. With the source handler API, there will be an HLS-Flash and HLS-Html5 source handler, and they will use the existing Flash and HTML5 techs instead of extending them to create a new tech.

Feel free to continue the discussion or ask questions here.

[gkatsev]

#2847 implements an option sourceOrder which would loop through each source and then for each source each tech.
The default stays the same.
It'll be released as videojs 5.3 which will be a @next on npm.