MAXIMIZE VIEWER ENGAGEMENT WITH NEW OTT FLOW FEATURES

As the democratization of video content continues to empower consumers with more viewing choices than ever, content owners face enormous pressure to deliver a compelling experience to maximize audience engagement, convert new users, and reduce churn.

The power of OTT Flow is delivering a service that enables customers to get to market quickly with a low total cost of ownership. That said, this offering should not be construed as a compromise from a premium service. Apps utilizing OTT Flow are frequently featured in the various app stores alongside bespoke offerings. At Brightcove, we strive to provide the best for our customers and, as OTT Flow continues to gain traction within the industry, we continue to enhance its feature set and integrate with the latest platforms as they become available.

Brightcove deploys new features and functionality on a rolling basis, allowing our customers to focus on what really matters, customer acquisition, maximizing user engagement, and reducing churn. Below are  highlights of the new features and functionality, along with use cases to see the features in action.

CROSS-DEVICE FUNCTIONALITY

We have put a lot of energy on cross-device functionality. This is an important set of features because viewers are now able to mark content as a Favorite to be added to a Watchlist and are also able to Continue Watching where they last stopped across different platforms. This makes it easier for viewers to track their favorite videos, movies, or series and enables ease of access across devices keeping viewers engaged.

FAVORITES

Viewers watching on an OTT Flow powered service can easily mark a video or a series as a Favorite by clicking on the heart icon, adding it to a list of all their favorite content on any of their connected devices with the app. Viewers can now easily navigate to the content when and where they want to watch it.

CONTINUE WATCHING

When a viewer pauses or stops a video or a clip, there will be a bookmark automatically applied to the spot which enables them to resume where they left off. This is highlighted by a progress bar on the video details page, for video items in a collection page, as well as with a resume playback button, which will start the next logical video for the viewer.

CONTROL CHROMECAST FROM THE WEB

Enabling an advanced Chromecast sender experience allows viewers to fully control the video stream that is seen on the connected TV, while still being able to browse for more content on their PC. In addition to providing a great viewing experience both on the website and TV, viewers now have a true second screen experience.

BINGE WATCHING

Binge watching is second nature for many viewers in today’s OTT landscape.. When there is 15 seconds left in the video the viewer will receive a prompt with a countdown timer in the corner, informing them when the next video will be starting.

CONDITIONAL HERO CONTAINER TO OPTIMIZE CONVERSIONS

To help drive conversions, the Hero container has been optimized to better support promoting content, advertising the service, and providing a call-to-action that entices viewers with a free trial or subscription signup. Once authenticated, the viewer will not see this promotion and will receive direct access to the content.

There is now a fully configurable menu at the bottom of an OTT Flow app’s website. This allows content owners to set up links to drive users to other parts of the app as well as displaying contact information and respective social media networks.

Keeping viewers on the app as long as possible is the goal, so helping them to discover new content is critical. OTT Flow has extended support for presenting “related content” to the collection pages so that editors can promote similar content to keep users watching even more videos and increase engagement.

MULTI-CURRENCY SUPPORT

One of the benefits of an OTT service is that viewers can watch the content in any country the service is available. However, this can make billing for SVOD services difficult. To address this concern, OTT Flow has improved support for customers who are offering their service in multiple countries. By default, the current exchange rate is applied to each offer as it is made available in a different country. Now, customers have the ability to create country specific subscription models making it easier to present more attractive pricing to viewers worldwide and giving content owners full control on how they wish to price the service in the different marketplaces.

OTT Flow maintains an active roadmap with increased functionality being delivered frequently.  In addition to the above, we have recently released the following:

IOS UPDATES

  • Mobile “splash” screens to increase branding
  • Enable background audio playback from lock screen
  • Collection Page UI enhancements
  • Navigation Item UI enhancements
  • Hero Container Items can have have an option Call to Action button.

ANDROID UPDATES

  • Mobile “splash” screens to increase branding
  • Sortable search results
  • Hero Container Items can have have an option Call to Action button.
  • List Container Items support the Short Description field from Brightcove

We encourage our customers to explore these new features to delight your end users and look forward to continuing to build new innovative features and functionality to help content owners build premium and profitable OTT experiences.

SUPPORT FOR ROKU DIRECT PUBLISHER: ONE YEAR LATER

In 2016, we announced support for Direct Publisher, Roku’s web application that empowers content owners to launch channels without any development. Through our Brightcove Global Services group, we can help customers organize metadata from Brightcove into a feed that customers can simply drop into Direct Publisher.

A year later, we’re delighted to give a shout out to a few customers who have used the solution to efficiently get their content in front of viewers on Roku.

LIONSGATE ENTERTAINMENT

Lionsgate Entertainment launched its Vidmark channel of 50+ free full-length movies on Roku, under the genres of Hollywood Horror, College Humor, Plan of Action, Politically Correct, and Keepin’ It Real. With 200+ reviews just a few weeks after launch, they’ve seen impressive engagement from viewers out of the gate.

DALLAS BASKETBALL LTD

Dallas Basketball Limited launched an official Dallas Mavericks channel on Roku for fans to watch behind-the-scenes footage of players, game highlights and a few long-form videos like the 2011 NBA Championship Parade. As owner Mark Cuban explained in a statement, “We want our fans to have access to Mavs content wherever they are, and within any platform they’re using. Adding Roku to our repertoire adds another piece to the fan experience puzzle.”

BABYCENTER

BabyCenter, the digital destination for expert-approved information related to pregnancy, children’s health and parenting, launched a channel on Roku that brings 200+ educational videos, like “how to sleep like a baby when you’re expecting one” and “should I have a doula?” to the living room screen. “Leveraging OTT platforms enables us to make our content easily accessible to today’s young couples, parents and families,” said Linda Murray, senior vice president of Consumer Experience at BabyCenter. “Our audience now has easy access to expert-approved information when they need it, no matter where they are or what device they’re using.”

At Brightcove, we’re always excited to collaborate with key technology partners, like Roku, to help our customers grow their video businesses. Kudos to Lionsgate, Dallas Basketball Ltd and BabyCenter, among others, who have taken advantage of Brightcove’s support for Roku Direct Publisher.

HOW TO PLAY CONTENT IN-LINE IN MOBILE BROWSERS

Allowing mobile video ads to play in-line is a simple change that can help improve ad performance for publishers and provide a better viewing experience for their audiences. This is especially important in the context of the ever-growing mobile market.

Sixty-two percent of ad buyers foresee an increase in their mobile video advertising spending in 2018 according to an IAB report. This is on top of the 77% increase from 2017.

The user expectation of receiving advertising in exchange for free video content, even on mobile devices, has become the norm. However, publishers haven’t always achieved the same fill rate on their mobile inventory as on desktop.

While advertising on mobile web is not as restricting as an in-app environment, (mobile IMA SDKs do not allow VPAID and offer less macro functionality), it does come with a unique set of challenges presented by the ecosystem.

Through our work with some of Brightcove’s top media publishers, we’ve been able to identify one main area where some best practices can help: how to play content in line within mobile web browsers, thereby allowing skippable ad playback and muted autoplay.

ENABLING IN-LINE VIEWING OF MOBILE VIDEO ADS

By default, when a pre-roll ad is played on an iOS device using Mobile Safari, the video will play full-screen with a native look-and-feel and restricted behavior.

For publishers, this does not allow some features advertisers have come to rely on, such as support for skippable video ads, standard click-through implementation, and the ability to autoplay while muted. Additionally, video ads will only play full-screen and are automatically paused when minimized.

Click-through ads will also be tricky, instead of navigating to the advertiser’s landing page the ad will resume playback and go back to the full screen native player. However, there is good news, as of Chrome 53 and iOS 10+, Android and iPhone smartphones support video playback “in-line” with the web page.

iOS does require additional steps in order to enable support for this functionality. Implementing ‘playsinline’ is simple; you have to include it as an attribute in the video tag.

An additional option that you should consider taking advantage of is disableCustomPlaybackForIOS10Plus. When set to true, it will allow you to disable custom playback on iOS10+ browsers. Together with ‘playsinline’, it will allow skippable inventory to stay inline, thereby getting around the limitation of the full-screen player.

The ‘skip’ functionality itself is controlled via the ad server, such as DFP, but without the above settings, skippable ads will fail. There are still Google Interactive Media Ads (IMA) SDK limitations to the format, and iOS 10 is required for best support, but these changes will make for far less frequent ad delivery failures on mobile.

ADDITIONAL BENEFITS OF IN-LINE MOBILE VIDEO ADS

Some additional benefits of playing your video ads in-line include exposing the click through in the form of a “Learn More” prompt, as well as the ability to autoplay pre-rolls on mobile, with the player muted.

Without the ‘playsinline’ attribute present, tapping on the ad in the full-screen player will simply pause it. In-line, the link will typically appear on the top right corner of the ad and will be interactive. This should increase CTR just by the virtue of allowing greater ease of use. Playsinline can also unlock the autoplay functionality on supported devices.

There is one big caveat; the player must be muted by default to get around Apple’s strict autoplay restrictions. You can learn more about the new browser policies around autoplay, as well as read about best practices from our engineering team.

The world of digital video advertising can occasionally seem like an over-complicated and scary place, yet there are rules publishers can follow to achieve the best possible results. The ‘playsinline’ element is one such simple, yet powerful, change that can better help publishers make the most of the industry’s ongoing (and inevitable) shift to mobile.

ARCHITECTING A VIDEO STRATEGY AT THE SYDNEY OPERA HOUSE

More than an international landmark, the Sydney Opera House is one of the world’s busiest performing arts centers and Australia’s premier tourist destination. With more than eight million visitors and over 2,000 performances each year, it’s a building that, according to architect Frank Gehry, “changed the image of an entire country.”

But even with a growing network of global fans, the Sydney Opera House was operating with legacy systems. And this meant its digital marketing was unable to live up to the promise of this great building.

In 2015, the Opera House embarked on a major digital transformation project. A newly redesigned website and publishing workflow laid the groundwork for the organization to take its content and marketing strategies to the next level, with video at the forefront. This significant shift set the digital stage for the Opera House to more fully integrate with Brightcove and reinvent its video strategy and user experience.

DYNAMIC DUOS IN THE ARTS

Video marketing and the arts are a powerful combination that many organizations are only just beginning to explore.

Prior to its site relaunch and digital footprint rebuild in 2015, the Opera House’s 10-year old website was slow, outdated and didn’t play well with video and other content initiatives. Through its transformation process and a partnership with Brightcove, the Opera House can now publish video more easily and feature it more prominently as part of its revamped digital marketing strategy.

With over 40 shows, gigs, concerts, talks, and tours each week as well as a range of food and beverage experiences available, the possibilities for marketing are vast. The Opera House is using Brightcove and all its components to streamline video workflow processes, add substantial upgrades to the look and feel of its content, and take engagement to a new level. Now it can do more with video to reach a global audience and cross cultural and language barriers.

  • Hero videos. The hero video is a show piece for the Opera House’s current theme, and serves as a visual montage that attracts and drives exploration of its website. The dynamic background enables a richer experience for visual queues, ultimately encouraging increased engagement and discovery.
  • Event promotions. Video acts as the engagement medium for the Opera House, leading with teasers and clips to promote regular local acts and feature special events. The team at the Opera House can then utilize Brightcove’s analytics to better understand the reach and effectiveness of the videos.
  • Workflow improvements. The Opera House’s new Adobe Content Management System is fully integrated with Brightcove using the Adobe Experience Manager Connector. This deeper integration has improved the management of its large video library, greatly simplified their workflow, and accelerated video publishing.

ARCHITECTING THE OPERA HOUSE’S DIGITAL FUTURE

Now that the Opera House’s digital transformation is in full swing, the organization is setting its sights on the future. With a calendar packed with events and experiences, the Opera House has endless sources of inspiration for video to develop a strong voice that tells compelling stories straight from the famous Aussie landmark.

The team at the Opera House is actively expanding its video marketing strategy to provide a more engaging customer experience and boost both digital and foot traffic. Brightcove will continue to work with the Opera House as it explores new ways to deliver a dynamic digital experience and engage with its millions of fans and visitors on the global stage.

AMP UP YOUR VIDEO MARKETING: TIE VIDEO TO REVENUE

In today’s digital world, information is everywhere. Content is multiplying at warp speeds, always on, constantly competing for our attention. Video helps you cut through this digital clutter. Video is human. Video is authentic. Video helps us feel. And more and more, how you make someone feel is the reason they choose to do business with you.

Marketers love video because video works. But creating great video content and making sure that video drives high return are two different things. Video leaders focus on the business impact of video investments. So the important thing you can do in today’s data-driven world is to measure the business impact of your video and then double down on what’s working.

Measuring video matters. According to Aberdeen Group, companies using video grow revenue 50% faster than companies that don’t. How can you attribute KPIs and revenue to your video and prove the the value of your video initiatives?

PRE-WORK: THE DATA BEHIND VIDEO

Before jumping into tactics, you need to understand the data involved. When it comes to data, video is unlike any other type of content. You don’t just see what viewers clicked, you also see how long they were engaged.

With a white paper, you know someone downloaded it, but you don’t know if they actually read it, let alone exactly how much they read. Knowing that someone has watched 7%, 64% or 100% of your product demo video is extremely valuable. When you combine an individual viewer’s video engagement metrics with their other online activity, you get a much clearer picture of their interests, pain points, and needs.

Where marketers fall short with video is we often use it in a silo, disconnected from the rest of our marketing. We spend a lot of time instrumenting our website, our email campaigns, our blog posts, and our whitepapers and e-book downloads. We set up beautiful nurtures and personalized campaigns based on what users click on and download, but we don’t combine our video data with the rest of our marketing programs.

STEP 1: INTEGRATE VIDEO WITH ALL OTHER MARKETING EFFORTS

To create a holistic view of your marketing results, you need to integrate your video into your tech stack. Specifically, into your marketing automation platform (MAP) such as Oracle EloquaHubSpot, and Marketo and Customer Relationship Management (CRM) such as Salesforce.

What’s important for video marketers to understand is that MAPs are blind to video and its deep metrics. Unless, of course, that MAP is connected to your video platform.

If you haven’t yet fully integrated your marketing stack, you have company. According to a 2017 survey by CMO Council and RedPoint Global, only 3% of marketers felt all of their automation, engagement, and deployment tools were fully connected, with data and insights traveling freely between the various technologies.

The good news is, the fix is easy. Once you integrate your video platform with your tech stack, video data will be added to all your other marketing data, allowing you to read the digital footprint of each individual viewer.

STEP 2: MEASURE MORE THAN VIDEO HITS

Once your video platform, MAP, and CRM are connected, it’s time to ensure you’re measuring the right things. When it comes to measuring video performance, many companies focus solely on the view count. View counts by themselves are essentially a vanity metric – not tied to leads, revenue, or ROI.

It’s not just about how many hits your video is getting, it’s about knowing your viewers. You should be looking at each individual viewer, who that person is, what they watch, and how much they watch. Video data integrated with the rest of your marketing data will give you this invaluable information.

This information helps you deliver the right information at the right time to guide them through the customer journey.

STEP 3: USE VIDEO DATA TO PERSONALIZE COMMUNICATIONS

When you know who is watching what video for how long, it allows you to personalize their experiences with your brand. According to the Aberdeen Group, video/rich media viewers are 65% more effective at maintaining consistent, relative, and personalized communications with customers and prospects.

Based on what video they watch, you can automate a campaign and lead score model. For example, if a product video is watched 100%, you can immediately route that contact to sales to follow up.

When you tie video to your tech stack, you also tie it to leads, to pipeline, to sales, and to retention. And more importantly, you can attribute your video to revenue.

HCSS is one of many of our customers using their video and MAP integration to drive success. A construction software company based in Texas, HCSS has connected Brightcove to HubSpot and Salesforce, using the data to demonstrate video’s business value to the C-level. Using video throughout the customer journey has allowed HCSS to shorten their sales cycle by 44%. Attributing video to critical business metrics has enabled them to increase their marketing team by 40%, create video programs at every stage of the customer journey, and redesign their website to be video-first.

STEP 4: EXTEND VIDEO ROI ACROSS THE COMPANY

Like many companies that begin by using video in marketing, HCSS is also now using video across their organization, including training. The Aberdeen Group reports 98% of video marketers support sales, 67% support the C-suite, and 51% support product development. By tying video into your tech stack, you can prove and improve company-wide results. According to the Aberdeen Group, companies using video platforms used their video-gleaned insights to achieve a 40% greater year-over-year growth in average deal size. They also reduced customer acquisition costs at nearly twice the rate of non-video platform users.

If you’re already doing video marketing, that’s wonderful. But before you do anything else, be sure to integrate your video platform with your MAP and CRM. Once you prove the ROI of video, it’s easier to get support to extend your video programs.

MAJOR BROWSERS ENDING AUTOPLAY VIDEO: WHAT YOU NEED TO KNOW

In 2017, Apple and Google announced their respective plans for doing away with autoplay video, which may have revenue repercussions for media publishers that rely on the functionality. Many of you may be aware of the existing autoplay restrictions on mobile, but the measures are new for desktop browsers.

WHAT IS HAPPENING?

Apple’s latest version of Safari, version 11, has autoplay disabled by default for videos with accompanying audio tracks. Apple’s implementation lets end users whitelist sites to enable autoplay (choices: “allow all” videos to autoplay, allow only videos without audio to autoplay, never autoplay). Most end users are likely not aware of their options and will not manually re-enable autoplay functionality.

As for Google, beginning with Chrome M65 (the public release is M61 right now), video with audio will not autoplay on desktop without a user interaction to begin playback. This change is being made by Google in order to more closely align expected desktop performance to mobile. Google explained the goals of the changes in a blog post, but the key points about the timing of the rollout are in the table at the bottom of this post.

Google’s post also explains their plan to allow some sites to be exempt from the new anti-autoplay measures based on Chrome’s Media Engagement Index (MEI), but the MEI is a work in progress as of this post.

WHAT WILL THE IMPACT BE?

The net effect of these changes is that publishers that rely heavily on autoplay pre-rolls are most likely going to notice a decrease in available video advertising inventory.

Brightcove has been testing players configured for autoplay against browser scenarios where autoplay is not permitted and the default behavior is for the player to simply become click-to-play. There may be scenarios where high MEI sites are allowed to autoplay but the rules around which media buyers consider which autoplay scenarios viewable is tricky. For example, as of today, Publicis will count autoplay videos as viewable but not muted ones. GroupM doesn’t consider autoplay ads viewable even with sound on but recently released new guidance that they’ll count as viewable autoplay with sound off on “native placements” like Facebook.

Unfortunately, Brightcove does not have control over browser-level restrictions and cannot provide a workaround. In addition to an inventory decrease, possible impacts may include discrepancies in metrics and an increase in errors. VPAID creatives in particular may be adversely affected, and if they don’t handle this condition effectively they may appear broken. We will work to do everything possible to make sure your Brightcove player is able to handle these roadblocks gracefully and that your content plays back successfully.

ANY ADVICE?

We suggest preparing for these changes by aligning with your ad partners. Decide if it’s better to autoplay muted on these browsers, which may not count as a viewable impression, or whether you prefer changing the UX to click-to-play and focusing on making sure you have engaging thumbnails to invite viewing. It might be valuable to create a table, for your internal use, summarizing any risk of campaign under-delivery or revenue shortfall that these browsers’ changes may have to your business. Here are the dates to consider for Chrome.

 
September 2017 -New autoplay policies announced
-Begin collecting Media Engagement Index (MEI) data* in Crome M62 Canary and Dev
December 2017 -Site muting available in M64 Beta
-Autoplay policies available in M65 Canary and Dev
January 2018 -Site muting available in M64 Stable
-Autoplay policies available in M65 Beta
March 2018 -Autoplay policies available in M65 Stable

 

The risk will likely depend on what percentage of your inventory is driven by autoplay, what the split is, across browsers, of your viewership, etc. Reach out to your Brightcove account team with any concerns or questions.

Brightcove has been adopted as a video platform that utilizes “one content, multiple uses” content across multiple media.

CREATING CONTENT THAT RESONATES WITH A TARGETED AUDIENCE

TOYO TIRE is a company that manufactures and sells tires for passenger cars, SUVs, pickup trucks and other large vehicles. In particular, large-diameter tires for pickup trucks are very popular in North America.


Mr. Ryosei Morikuni, General Manager of the Brand Communication Department at the company, says, “I think that tires are a product category that is difficult to differentiate, except for core car enthusiasts. The frequency of tire replacement is about every three years, but after purchase, not many people are strongly aware of the brand name of the tires they have installed. In other words, I think they are a product category with low involvement.


For consumers, the three main factors that influence their decision to purchase tires are price, brand, and the recommendation of a salesperson. It is difficult to judge the difference in performance of tires by looking at them, and there are also almost no opportunities to actually try out tires. “For this reason, I think it is important to raise brand awareness, especially in terms of pure recall,” he says.


Tires are a product that everyone who owns a car will definitely purchase, and TV commercials are essential for achieving widespread recognition. At the same time, in order to communicate deeply with specific target groups and increase engagement, the company is also focusing on digital communication using web videos. This initiative, which began around 2014, involves creating video content centered around the company’s motor sports sponsorship activities, and distributing it on YouTube and Facebook. While distributing a large number of videos, the company has found that if the video resonates with the target audience, they will watch even long videos to the end. At present, the company produces 50 to 60 videos a year.

I WANT TO DISTRIBUTE ALL OF MY VIDEOS TO MY OWNED SOCIAL MEDIA CHANNELS AT ONCE.

It’s only been a few months since we started using this system, but I can already feel that it has reduced the amount of work I have to do by about 30%. The stress on the production side has been greatly reduced, and we now have an environment in which we can easily use a single video in a variety of media.

Yoshiyuki Morikuni
General Manager, Brand Communication Department, Corporate Planning Division, TOYO TIRE & RUBBER CO.

However, we soon encountered a problem. The amount of internal management work required to deliver our videos was increasing. When we first started using digital video for branding, it was enough to just post videos on YouTube. However, as the number of video contents gradually increased, the number of media that could effectively use video also increased, including owned media and social media. This was because we had been working to “aim for one content, multiple uses” (according to Mr. Morikuni), turning various communication activities into video content and using that video on multiple media. However, each time a video was posted, it took time to upload it and tasks such as embedding tags had to be done. The workload of the person in charge was increasing.

“For example, if you create a video report of an event, it will not be a one-off event, and you can expand the opportunity to experience the brand to people who did not participate in the event. Also, even if it is a single video, by creating different lengths and optimizing it for each medium, we think that the contact points will also expand. However, as the number of media posting that video content increased, the burden on the person in charge also increased.” (Same person)

In 2018, the company learned about Brightcove Video Cloud. They were attracted by the function that allows them to store videos in Brightcove Video Cloud and automatically distribute them to various social media with simple procedures by using Brightcove Social. They were also attracted by the fact that the video portal site, which was previously a static site, could be changed into a dynamic site by using Brightcove Gallery, and that the site could be automatically generated by linking it with Brightcove Video Cloud. Above all, they felt that the biggest attraction was that this would greatly reduce the workload of the person in charge.

“I feel like it has reduced the workload by about 30%.”

The new video platform went live in December 2018. By storing all video content in Brightcove Video Cloud, the video distribution work has been greatly streamlined. This is because it is now possible to manage the posting of videos to YouTube, owned media, and multiple SNS in one go.

Mr. Morikuni says, “It’s only been a few months since we started using it, but I can feel that it has reduced the workload by about 30%. Also, we no longer have to repeatedly log in to each social media site to check that we haven’t forgotten to press the publish button. The stress on the front line has been greatly reduced, and we now have an environment in which we can easily use a single video in a variety of media.”

The renewed video site “TOYO TIRES MOVIE” was designed so that videos can be updated simply by selecting whether to publish or unpublish them in Brightcove Video Cloud. The site, which uses a grid layout and allows users to access videos through an interface that is easy to understand and easy to see, uses a template from Brightcove Gallery.

The company has a security policy in place that prevents YouTube from being viewed within the company. However, sales representatives sometimes discuss videos with customers, and the company also wants its internal employees to see its brand content. With the start of operations using Brightcove Video Cloud, employees can now freely watch videos from the company’s internal network, and this has also had the effect of internal branding. In addition, because there are no advertisements displayed in the videos shown on the company’s website, this also helps to protect the brand.

The company plans to continue to expand its video content to attract more fans and create fans of the brand.

BUILDING A VIDEO PLAYER FAILOVER STRATEGY

At Brightcove, we use Amazon S3 as part of our video player delivery, publishing, and player management solutions – and rightfully so as it’s a leader in the space and delivers great performance. Great performance does not make it infallible, however. We’ve spent a lot of time and effort hardening our services to withstand an outage in anticipation of such an event and our efforts have paid off.

AMAZON S3 SERVICE INCIDENT

On Thursday September 14th, 2017, for a brief period around 1:37 PM EDT and then for about an hour beginning at 2:40 PM EDT, the Amazon S3 service began unwarrantedly throttling requests in the U.S. East region. As a result, it was rejecting a lot of perfectly valid requests to get and put files.

Many S3 customers were impacted, including some of the most visited websites on the internet that rely on this highly-reliable cloud infrastructure. While Brightcove did have an increase in error rates for some cold cache video content, we are happy to report that player loads, whether cached or uncached, went unaffected. What could have been a much bigger event was very much contained.

Here is some info about the incident as reported by The Register. The information that Amazon shared during the incident was as follows (times adjusted to EDT):

  • 2:58 PM EDT. We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.
  • 3:20 PM EDT. We can confirm that some customers are receiving throttling errors accessing S3. We are currently investigating the root cause.
  • 3:38 PM EDT. We continue to work towards resolving the increased throttling errors for Amazon S3 requests in the US-EAST-1 Region. We have identified the subsystem responsible for the errors, identified root cause and are now working to resolve the issue.
  • 3:49 PM EDT. We are now seeing recovery in the throttle error rates accessing Amazon S3. We have identified the root cause and have taken actions to prevent recurrence.
  • 4:05 PM EDT. Between 2:40 PM and 3:56 PM EDT we experienced throttling errors accessing Amazon S3 in the US-EAST-1 Region. The issue is resolved and the service is operating normally.

At 1:37 PM, well before the Amazon acknowledgement, the first errors triggered our alerts. Rather than tell us that there was a problem with our services, the alerts told us that corrective action had automatically kicked in.

During this entire incident, which occurred at peak-times for Brightcove, we responded to more than 28 million requests for the Brightcove Player without issue. This included players that were already active as well as 79 players that were published during the incident.

PLAYER FAILOVER TACTICS

Here is a glimpse of what our S3 setup looks like.pasted image 0

Although we do use S3 in the U.S. East region (where the incident occurred), we have implemented bi-directional cross-region replication options provided by Amazon.

We have also created our own open source Node.js project called s3-s3 which handles automatic switching to different S3 buckets as needed. This means that under normal operating conditions every file that we upload to U.S. East is also sent to our failover region.

Every time the Player Management and Publishing services interact with S3, they are ready for the worst. If there is an error, we automatically retry and if the retries fail, then we automatically failover to another region.

We handle failover at the individual request level. Requests always start with the primary region and fall back to the failover region if necessary. There is no need for manual intervention to switch systems over or back.

Thanks to our bi-directional replication, any files that must be pushed up to the failover region will be automatically replicated back to U.S. East when it is healthy again.

The graph below shows our automatic failover in action. The services continued to use our primary S3 bucket as often as possible but did end up failing over quite a bit in the process. The alternative: All of the requests on the right would have failed causing issues for our customers.

pasted image 0 (1)

So if you were blissfully unaware that there were any Amazon problems happening, that’s how we like it.