Scaling Metrics Maturity in a Cloud-Native World
Technology-focused companies across all industries rely on cloud infrastructure & microservices to deliver value to customers; and by extension, profits for the business.
The benefits of a performant infrastructure must be very apparent, and so must their degradations. The measurement and attribution of this performance in a complex software environment is called Observability (o11y).
And hence, Observability has become a first-class engineering citizen in such organizations.
As you climb the ladder of reliability, metrics increase and, correspondingly, the breadth of questions. Modern Time Series systems don't have to grow along a single axis of Cardinality, Coverage, or Retention alone.
Instead, the rate of ingestion and exploration warrants an expansion on all three axes.
- Coverage: More metrics are to be observed
- Retention: Save metrics for a longer duration
- Cardinality: Same metrics for more entities
In this initial stage, the scale of data is easily manageable. Small teams, low complexity, and low business maturity mean essential incident detection is the focus, rather than deeper intelligence over extended periods. Data retention needs are also at their lowest. There's no pressure to do better centralization, or formally democratizing access to time series data.
Tolerance towards failures diminishes. There’s more focus towards feature velocity. This is typically where you see an explosion of infra and monitoring bills. To deliver quality customer experiences, teams aspire to find problems before their customers. Teams must implement alerting & dashboarding to detect & diagnose incidents. Friction between business, engineering and infrastructure teams start growing. Formalized DevOps processes are considered at this stage.
At this scale, metric data is needed by different teams — across product, finance, support, & customer success, to properly run a business. Data growth explodes across all dimensions. Enterprises must dedicate full-time engineering resources to manage their time series database. Greater business maturity moves the focus of engineering teams from code quality to customer experience quality.
Inspired by principles of Data Warehousing, where Data Tiering is a common phenomenon, Levitate introduces tiers for the Time Series storage.
Additionally, Levitate offers powerful features to identify time series your team isn't using and trim data according to Data Policies you create. Access Policies enable you to control how tiers are engaged.
Our existing customers have more than halved their storage costs with Levitate, excluding advantages over reduced engineering overheads and their management.
Policy Engine Capability to express, evaluate data storage, and access rules.
Query Routing Engine Based on tokens + source, a query routing engine routes queries to their rightful tiers. The overhead of evaluation is minimal.
Sync Engine Separating write and read channels, guaranteeing ingestion no matter how heavy the read loads are.
Consumption Engine The consumption engine keeps track of metrics being consumed, and makes them available to all other engines.