Mark Fisher

Mark Fisher on

MLB World Series 2017 – Los Angeles Dodgers vs Houston Astros

Live Streaming to Massive Audiences:

Bases Loaded and …. “Playback Failure” …. Still Need Scale and Quality to Win the Game

Houston won game 5 last night after a sometimes wacky but certainly memorable contest that ran for a grueling 5 hours and 17 minutes in Houston. The marathon event was long enough to rank near the top 20 longest MLB games ever played and serve as a reminder of baseball’s unique attribute among professional sports, the game can go on forever.

For those cord-cutters and cord-nevers who stream these epic live events, there may be occasions when a full five+ hours of baseball comes in handy.

One such occasion is when you are hit with a two hour streaming outage in the middle of the game.

You can stream the series this year from Fox Live, Hulu, MLB and PlayStation Vue. Depending on your choice of streaming provider, it’s possible you may get all or just some of the actual event. This was the case with those Hulu subscribers who attempted to tune into the game last week.

Hulu ran into problems on Saturday during Game 4 when their live stream crashed. Faithful Huluans saw a “Playback Failure” error message on the screen and were invited to “Retry”. Those brave enough to attempt the retry were then trapped in a video loop of a single pitch from the mound before the service crashed again. Hulu live streaming service was finally restored about two hours later. It almost goes without saying that a crash of two hours during the streaming of a live event will almost certainly take all the “live” out of the event. Except when the game lasts over 5 hours. So, there is that unintended benefit of Game 5.

Among the more entertaining tweets that ensued following the Hulu outage:

Hulu’s performance in Game 4 feels like Déjà vu when you recall Showtime’s streaming efforts in support of the Mayweather – McGregor fight in August where servers crashed and the boxing match was delayed to the dismay of millions of viewers. A class action suit against Showtime and a UFC offer of refunds to disappointed subscribers were some of the damage control responses when capacity, scale and quality all failed to materialize for this massive live streaming event. Unlike Hulu’s outage on Wednesday, Showtime customers had paid a relatively huge sum – $99 – to watch a single event. So, expectations were high and, in the midst of the outage, disappointment raged and spread like a wildfire from coast to coast.

Enter Open Caching and the Edge Cloud

In the midst of this streaming turmoil there are real signs of progress toward the scale and quality we need to stream live events to worldwide audiences of tens of millions. Last week, for example, Qwilt and Limelight Networks announced an initiative to integrate their respective infrastructures.

Qwilt and Limelight Networks to Offer New Integrated Caching Solution for Content Delivery

This announcement follows years of collaboration between Qwilt, Limelight Networks and many others in the Streaming Video Alliance where Open Caching specifications have been developed and approved by the members to encourage and guide this kind of integration. Limelight brings its global caching infrastructure with 80 points of presence and 25+ Terabits per second of egress capacity. Qwilt brings the massive open caching capacity deployed deep in ISP networks worldwide. Together, the combined integrated caching infrastructure delivers the best Quality of Experience (QoE) because content delivery is pushed to the deep edge of the network, close to consumers, where latency is extremely low. This is the new paradigm for content delivery on scale that was envisioned by the founders of the Streaming Video Alliance years ago.

The announcement was covered by Aditya Kishore at Light Reading:

Qwilt Takes Limelight to the Edge

Within the article, Aditya observes:

“In February 2017, Qwilt also announced the launch of its Open Edge Cloud platform, which combined its virtual caching nodes with a centralized management system and an open API. Its goal at the time was to create a programmable interface and use its open API to link to CDNs and content publishers. (See Qwilt Opens Up at the Edge.)

This integration with Limelight is built using that API, and provides evidence of its applicability. The same API can allow service providers to link to other commercial CDNs as well.

For content owners, this helps, as it further reduces the uncertainties of traversing uncontrolled networks by serving the content from closer to their consumers than even local PoPs on their CDN.”

The benefits of integrating Open Caching with upstream commercial CDNs are many. The content owner gets better QoE for their consumers. The CDN gets to extend their infrastructure (through Open Caching) into the last mile network which provides more capacity, lower latency, better quality and improved economics. Finally, the service provider becomes a member of the content delivery value chain and benefits from the network optimization, reducing capital and operational expense, that ensues when massively redundant OTT streams are converted to edge delivery because of a vastly superior content delivery architecture.

Open Caching and the World Series – A Case Study in Scale and Quality Streaming

We can quickly get a sense for the contribution Open Caching makes to overall scale when looking at the streaming traffic generated during a few of the recent World Series Games over the past week. This data comes from US ISP deployments around the US. In two cases, the streams are from Fox Sports and in one, from PlayStation Vue.

Below are a few snapshots of Open Caching Edge Delivery Performance from selected systems in US Internet Service Providers during the series.

Game 1 – October 24th

Stream Source: Fox Sports

Edge Delivery Performance:

Open Caching Supports 78% of total live streams at peak delivered from deep within the ISP last mile network at low latency.

Game 2 – October 25th

Stream Source: Fox Sports

Edge Delivery Performance:

Open Caching Supports 80% of total live streams at peak delivered from deep within the ISP last mile network at low latency.

Game 4 – October 28th

Stream Source: PlayStation Vue

Edge Delivery Performance:

Open Caching Supports 81% of total live streams at peak delivered from deep within the ISP last mile network at low latency.

Note the significant increase in streaming volume from this site (Vue) during the event. This shows the impact of live event streaming on the overall infrastructure – from CDN to Service Provider network – as traffic ramps up dramatically based on user demand for streams. In this case, peak traffic in this particular last mile network segment increased by over 10 times the normal daily volumes. Almost all of the increase in needed content delivery capacity was handled by the Open Cache during the peak event.


An Industry Grand Slam – The Future of Live Streaming is Bright

The announcement by Qwilt and Limelight is proof that the industry collaboration needed to scale streaming is happening and producing good outcomes for the entire ecosystem. Further, the real-world delivery results of Open Caching show how the future of streaming will evolve to achieve the capacity and quality to reach a mass audience of 10s of millions of viewers with broadcast quality.

We’re proud to partner with Limelight and equally proud to be one of the Open Caching innovators as we can see the future that this new open architecture for content delivery will enable.