Reliable Observability for 25+ million concurrent live-streaming viewers

A cricket match can have over 25 million concurrent viewers. Games last about 3-4 hours, and systems are warmed up hours in advance in anticipation of the sudden surge in traffic. Significant ephemeral resources come online to last the game’s duration and be torn down soon after. Hundreds of engineers work on backend services and a robust infrastructure to enable the live-streaming of such high-ticketing events.

When something goes wrong, the war room team needs to immediately isolate the issue. The next step is to draft a Root Cause Analysis (RCA) and route it to the appropriate team for further investigation. Every additional second that is taken to diagnose a problem adversely impacts advertising revenue. Above all, it causes massive viewer dissatisfaction, given the criticality of missing a major sporting memory. Teams often find themselves investigating leading indicators of failure. These problems manifest on social media platforms, and before they spread, need to be triaged and fixed.

The Multifaceted Infrastructure Platform

A diverse and complex infrastructure platform powers our customer’s scale. Hundreds of microservices and a variety of data stores handle persistent data storage; some are fully managed by Cloud Hyperscalers, and the in-house team manages others. The scale-up necessitated by such games results in many ephemeral resources coming online. Consistent and uniform Observability across these disparate sources is incredibly challenging; to observe their health all of these sources continuously emit metrics and in plenty. Over time, the team’s engineers noticed a sprawl of metrics and monitoring techniques, making it hard to standardize the telemetry and monitoring.

The team was using an in-house setup based on VictoriaMetrics (a popular open-source time series database) and InfluxDB for metrics management. For visualizing data and managing alerts, Grafana was used.

Growing scale and concurrent access woes

Thousands of dashboards were created by multiple teams presenting unique challenges. Grafana dashboards and alerts were concurrently accessing the same underlying metrics storage. These underlying databases could not keep up with massive ingestion and simultaneous queries. Inevitably, the storage would ultimately go down leaving the teams oblivious to the health of their infrastructure. Instead of focusing on features and innovating on the product, the engineering team spent countless hours keeping the Observability platform up.

To reliably support the team’s incredible infrastructure growth, they needed a next-generation Observability platform. Given their unique challenges and incredible scale, the team needed a product that could withstand “cricket scale”, sustain uptime, be globally available, and not explode costs.

Last9’s Levitate is a globally available time series & events warehouse designed for scale, high cardinality and long term retention.

Open Standards

Levitate ingests data from multiple open standards, such as Prometheus exposition, OpenTelemetry Metrics, OpenMetrics, and InfluxDB. This ensured no migration effort was needed at our customer’s scale of hundreds of micro-services. Hundreds of engineers were onboarded to existing and new workloads on Levitate within weeks, given interoperability and ease of integration. Since Levitate is fully compatible with Open standards on the output layer, the team could keep using their existing dashboards and alerting workflows.

Within a month, Levitate was the source of truth for all metrics workloads across our customers’ teams.

SLA Guarantees

Levitate is a managed service with Service Level Agreement (SLA) guarantees and clawbacks for both Read and Write workloads. This eliminated the toil and upkeep to manage and scale our customer’s in-house metrics setup.

Long Term Retention

With the previous in-house metrics setup, teams could not retain data beyond a month for critical analysis. Imagine having billions of data points of consumer behavior, but being unable to use them for growing business needs.

Levitate’s automatic data tiering and retention policies paved the way for long-term time series storage. This helped the team with capacity planning and business insights year after year. By default, the latest data is available in all tiers, but their retention policies vary.

Levitate’s Data Tiering capability is also used on the query layer, creating policies for accessing the Blaze tier only for alerting. The other tiers can then be used for deeper exploration and analysis. This resolved the concurrent access issue they faced with the in-house metric setup.

Levitate has improved query speeds, dramatically reduced the Total Cost of Ownership (TCO) by 50%, and is currently the bedrock for the customers’ entire infrastructure.

Bring Your Own Cloud Model

Levitate comes with a Bring Your Own Cloud (BYOC) model - we can deploy in our customers’ cloud directly offering all the features Levitate comes with.

Levitate is currently the bedrock for the customers’ entire infrastructure.

With optimized auto-tiered storage, warehousing control levers, and availability guarantees, we’ve reduced the toil of managing a time series database and the engineering overheads that come with it — something seldom factored in while calculating the cost of running your own Observability team.