# Why you should (not) build your own observability stack > If you are able to build it better than your vendor, then change your vendor. Not build it. Source: https://last9.io/blog/build-vs-buy-observability-stack/ With AI, engineering orgs are confident that they can build something in-house tailored to their needs at a lower cost than their vendors. This argument sounds right, but the decision is not that simple. It needs deeper deliberation. **I will make the case from an observability standpoint.** With Grafana, Prometheus, ClickHouse, and VictoriaMetrics all having an open source version, a decent enough observability stack can be built in-house in a few weeks time. Reaching **80% of an observability solution is not a problem** and can be achieved in weeks with proper planning, requirements, architectural decisions, the right set of people, prioritization, and business justification. However, **the remaining 20% is the place where magic happens or magic fails.** That 20% is where the build-vs-buy decision actually lives. **When you buy, you buy a complete solution, when you build - you build the software and then the entire solution.** With AI the building of software is easy and fast, however building a solution which solves for a long run can be challenging and needs a long term commitment from stakeholders. And as you get closer to 100%, perfecting every single-digit improvement becomes painful and challenging. ## You are buying a solution, not software A solution: 1. **Is different from writing software** - software is a part of the solution and not the solution itself. 2. **Solves for a long term**, has features which solve your current problems as well as your future ones. 3. **Runs like a well-oiled machine**, reliably, and can scale with your business aspirations without your intervention. Scaling should be just automated or a message away. 4. **Evolves with changing needs.** New features keep getting developed and tailored to your needs. 5. **All the challenges of developing, operating, maintaining, testing, scaling and reliability** are hidden behind an abstraction of the solution team / vendor team. However, for the vendor team developing that solution, the work is to **maintain the abstraction** promised to the customers (internal or external). That means: 1. **Evolution** with new challenges and needs 2. **Operations** to support high scaling demands and reliability 3. **Consistent maintenance** 4. **Integration support and onboarding** for customers (internal and external) Self built systems can reach that 80% easily with open source and AI, but **the 20% will need constant energy and time** to ensure coherence with the changing business needs. We have written before about what that 20% looks like in practice, in [the challenges of running Prometheus at scale](https://last9.io/blog/challenges-with-running-prometheus-at-scale/). The market sorts itself into open source, paid, and managed open source, and we have written about [the trade-offs of each](https://last9.io/blog/observability-oss-vs-paid-vs-managed-oss/) before. This post is about the option underneath all three: owning it yourself. > When you buy, you buy a solution, not software. ## What it costs to develop and operate an in-house observability solution The cost follows from the requirements, so start there. ### Requirements from an observability solution Two of these requirements carry most of the weight, so they are worth stating in full. **Close to zero room for errors.** Availability only guarantees that data will be written to the system. A failure or bottleneck can still happen after the system has accepted the data — ETL lag, a sudden high read query load, or the database running TTL cleanup or merges — each of which opens a gap between data written and data read. A lag of even **3 minutes** during ingestion will fail a lot of alert rules, through loss of signal and incorrect arithmetic results. The resulting alert storm cannot be identified as incorrect for another 5 to 10 minutes, and it impacts every team that depends on the observability stack. **Latency, in both directions.** Ingestion needs subsecond latency, because higher write latency puts backpressure on the senders. Reads need low latency, because high read latency results in slow debugging and frustration. Both have to hold even across cross-geo deployments. The rest of the list: | Requirement | What it takes | | ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | **Operates at 99.9%+ availability** | The baseline that every other requirement assumes | | **Maintain cost SLAs with varying needs** | Ingestion patterns, data size, and data volume all change, and each one moves the cost in a different way | | **Has cost effective infrastructure** | Tiered storage, autoscaling, data transformation, and filtering | | **Evolves with changing requirements** | The same changes ripple into the ETL pipeline, the backend databases, and query performance | | **Boring (aka reliable)** | Observability is the core of a reliable software product. It should not excite your engineers, and there should be no fire fighting at this layer | | **Throughput** | Handles the different traffic patterns across the entire 24 hours, plus burst traffic, sudden read load, and data exports, without hurting latency | | **User interface** | An intuitive and pleasant looking interface that eases debugging and solving incidents in high pressure situations | | **User experience** | Easy, stable, and mature integration — and wide integration, covering CDNs, Kubernetes, Kafka, databases, nodes, and applications | | **Other features depending on business needs** | Backups, disaster recovery, multitenancy for different teams or environments, and cross geo or per geo deployments | | **Adherence to compliance frameworks** | HIPAA, PCI, SOC 2, depending on the business | | **Support** | Round the clock support for various issues, and consulting for observability from SMEs | ### Cost of engineering team The same question at a single layer is instructive: we compared [self-managed Prometheus against managed Prometheus](https://last9.io/blog/self-managed-prometheus-vs-managed-prometheus/), and the stack below is that decision made seven times over. To build a solution with the above requirements, what will be the engineering team and skills required? #### Engineering team needed 1. **Hard skills** 1. DevOps 2. Database skills for modelling the data 3. DBA skills for operating the database 4. ETL 5. Dashboarding 6. UI 7. Backend 2. **Soft skills** 1. Stakeholder management 2. Product management 3. Internal customer management 3. **Operational skills** 1. **Round the clock on-call rotation** 1. **Errors outside office hours need immediate fix:** Even if your traffic pattern does not serve customers outside of office hours or beyond 9 to 9, you cannot ignore errors because, at this layer, all the other teams are dependent on it. A batch system running in the night also needs observability. 2. **Error free deployments** in the observability stack at ETL, backend and database layer. UI can be down for some time for maintenance 1. A failure here makes you blind to failures at other layers. 2. A false alert storm takes the engineering teams that rely on your system away from their own work, let alone the panic and stress it causes. #### What will the observability engineering team cost? For a thin solution with low customization, an aggressive and lean team would be: 1. **2 generalist senior engineers** 1. Handle most of the work and on-call rotation 2. Use open source software for most of the components 3. Deploy and maintain 4. Customize the stack for internal use cases 5. Dedicated initially to observability, then shared with other work when the observability solution matures 2. **Manager** 1. Takes care of the soft skills, drives adoption with the customer 2. **20% of their time** For a custom solution, with good UX and platform stability guaranteed and tested from the start, the team would be: 1. **Observability SME** 1. Consults teams on how to implement observability better, optimizes and customizes observability as per teams. Dedicated initially, shared post maturity, will be needed in L3 on-call 2. **[ClickHouse](https://last9.io/blog/clickhouse-high-cardinality-at-scale/) and TSDB expert** 1. Dedicated initially, shared post maturity, will be needed in L3 on-call 3. **Backend and ETL expert(s)** 1. Dedicated initially, shared post maturity, will be needed in L3 on-call 4. **UX and UI engineer(s)** 1. Dedicated initially, shared post maturity 5. **DevOps engineers** 1. Dedicated initially, shared post maturity. At least 2, and they will share other load 6. **Manager** 1. Takes care of the soft skills, drives adoption with the customers 2. **40% of their time** if shared **Checklist** 1. What will be the direct cost of the above team? 2. What will be the indirect cost of that team? 3. Is the cost of the above teams and work less than the cost of buying it from a vendor? ### Infrastructure cost The cost will be divided into 3 major sections 1. **Compute** 1. At minimum, you will need to run two DB / Data store replicas plus the ETL fleet. 2. **Storage** 1. In a tiered storage environment, even with finely selected warm data the storage cost will be around **2 cents per GB per month** of data stored. Hot data storage cost is higher, around **~9 cents per GB per month**. 3. **Network** 1. Observability data is huge, and its size is very different from application data. 2. This adds a significant amount of network cost as well. **Architecture on day 1** 1. The day 1 architecture will be **significantly different** than that on day 100 and day 200 2. The sub-optimal architecture on day 1 will cost more than day 100 **Monitoring your observability setup** Apart from the above, what will be the cost of monitoring your monitoring infrastructure? **Wastage** Apart from the business as usual traffic, the solution needs to reliably handle: 1. **Different traffic patterns during the day**, high during business hours, low during off hours, very high during marketing campaigns like Sales, or very very high during long running tournaments like FIFA or Cricket world cups. 2. **Event storms** due to applications needing more visibility, or an incorrect configuration at their end To solve this, you either: **Provision headroom**, wasting resources and increasing cost. As the architecture evolves this wastage will go down. However it will need engineering energy and bandwidth. Or **optimize and engineer the solution**, thus incurring more engineering cost and needing more expertise. **Checklist** 1. What will be the day 1 infra cost? 2. How much will the cost per GB ingested differ on day 1 vs day 100 vs day 300? 3. Is the infrastructure cost transparent, or hidden inside the price? 4. Is the cost per GB a promise, or a speculation? 5. Can the infrastructure run in my environment so I can see the bill myself? At Last9, the infrastructure can run in the customer's environment, so the infrastructure cost is fully visible. Hold every vendor — including us — to that standard. ### Evolution Then there is change, continuous change. As the business grows, as the application code changes, and as the internal customers' behaviour changes, there are changes in the observability stack: 1. **Internal customer requirements** 1. New feature requests come and new functionalities are required. Each such requirement needs to be developed, tested, and deployed at scale 2. **Ingestion data volume** 1. Change in data volume impacts the entire backend database deployment, capacity planning, and schema 2. Changes here impact the cost calculations, budgeting, and cloud bills 3. **Ingestion traffic pattern** 1. Changes how to scale the system, considering multiple factors The above change is continuous in fast moving and growing organizations and it results in a continuous SDLC cycle where the platform needs to evolve with the changing requirements. **So the team which was supposed to be shared will be dedicated, and the product whose first 80% was delivered in a few weeks will need continuous development for its last 20%.** **Checklist** 1. How fast do custom requirements actually ship? Ask for a real example: request date → production date. 2. Does the solution handle my requirements list today, or is it "on the roadmap"? Roadmap means you're building it anyway — on their payroll and their timeline. ### Business justification How do we justify the business case of moving this in-house or building this in-house? For a car manufacturing company, spending time and energy on researching and developing tires is a distraction. Then there is a lead time and investment of getting the first tire rolled out of the factory. **In short, creating tires for a car manufacturing company is a distraction.** When your organization is finding its product-market fit, solving complex challenges on core business value or hyper-growing, **not getting distracted should be the primary goal.** > "What must we not build?" should be a standing question in your team meetings. A team should be asking: **What is the quickest way to solve this?** Or: **What is the most cost-efficient and time-efficient way to solve this problem?** Every requirement should be tested against two questions before being worked on: 1. **How does it align with the mission** we are working on 2. **How does it impact the revenue stream** of the organization directly or indirectly **Why build something which is already there, and is not aligned with the org's mission or does not directly impact revenue?** ### Compliance If the org complies with PCI, SOC 2, HIPAA, or any other such frameworks, then there is **additional energy and time involved.** There is additional engineering work required to build and maintain solutions that comply with these frameworks. The requirements themselves evolve, which means the system needs to evolve with them. ## Build vs buy: what you own in each case | | Build | Buy | | ---------------------------------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | | **Time to the first 80%** | Weeks, with open source and AI | Days, integration only | | **The last 20%** | Yours, continuously | The vendor's | | **Team** | 2 senior generalists at minimum, more for a custom stack | None dedicated | | **On-call** | Round the clock, at the layer every other team depends on | The vendor's | | **Infrastructure cost** | Compute, storage, network, and headroom, visible on your own bill | Inside the price, visible only if the vendor runs in your environment | | **When data volume changes** | Recalculate capacity, schema, budgets, and cloud bills | Contracted | | **Evolution** | A continuous SDLC cycle on your roadmap | On their roadmap, their payroll, their timeline | | **Compliance (PCI, SOC 2, HIPAA)** | Yours to build and maintain as the frameworks evolve | Inherited, if the vendor holds the certification | | **When it fails** | You are blind to failures at every other layer | Escalate to the vendor | | **Best when** | Buying costs more, you already run a 99.9%+ ops team, and the commitment is long term | Observability is not your core business | **Bottom line**: buying transfers the last 20% and the on-call rotation to someone else. Building keeps both, and keeps them for as long as the system lives. ## When to build 1. **When buying is costlier than building**, consider energy, focus, opportunity cost, and money (direct and indirect) 2. **Your organization already runs an operations team holding 99.9%+ availability targets.** Building a team from scratch is a different problem, because the stakes at this layer are higher. 3. **When the commitment is genuinely long term** and today's justification will still hold for the next couple of years from now. And if you are able to build it better than your vendor, then change your vendor. Do not build it. Replit went the other way. They ran their own Thanos-based stack, then [deprecated it and cut monitoring costs](https://last9.io/blog/how-we-reduced-monitoring-costs-and-deprecated-thanos-for-replit/) — the decision this post is about, made in production. ## Summing up 1. Building something is short, easy, and simple. **Extending and operating it is not.** 2. **Building is a long-term commitment.** The decision is not just whether you can build version one. It is whether you want to own **versions two through fifty**. 3. **TCO is much more than infrastructure cost.** It includes people cost, infrastructure cost, engineering cost, management cost, operational cost, compliance cost, opportunity cost, and the cost of continuously evolving the system. AI changes this equation, but perhaps not in the way the current build-vs-buy conversation suggests. With AI, execution velocity has increased dramatically. The cost of getting to version one has fallen. But the other costs of ownership have not disappeared. AI can help you build faster. It can help a smaller team accomplish more. It can help engineers understand unfamiliar systems, generate code, automate operations, and troubleshoot problems. But the system still needs to run. It still needs to scale. It still needs to survive failures. It still needs to be upgraded. It still needs to comply. It still needs to evolve with your business. And someone still needs to own all of that. **AI has made building cheaper. It has not made ownership free.** That is what makes build versus buy more deceptive now. Version one is the easy part. If you are deciding whether you want to own versions two through fifty, [schedule a call with us](https://last9.io/schedule-demo/). --- ## FAQs ### Should we build or buy our observability stack? Open source gets you to roughly 80% of an observability solution in a few weeks. The remaining 20% is where the decision actually lives — evolution, operations, consistent maintenance, integration support, and round-the-clock on-call. Buy when observability is not your core business and the team is better spent on work that moves revenue. Build only when buying costs more once you count energy, focus, and opportunity cost. ### When does it make sense to build your own observability platform? Three conditions, and you need all three. Buying is costlier than building once direct and indirect costs are counted. Your organization already runs an operations team holding 99.9%+ availability targets, so the on-call and reliability muscle exists. And the commitment is long term enough that today's justification still holds in two years. If you can genuinely build it better than your vendor, change your vendor instead of building. ### What is the total cost of ownership of an in-house observability stack? More than infrastructure. TCO covers people cost, infrastructure cost, engineering cost, management cost, operational cost, compliance cost, and opportunity cost, plus the cost of evolving the system as ingestion volume and traffic patterns change. On infrastructure alone you pay for compute (at minimum two datastore replicas plus an ETL fleet), storage at around 2 cents per GB per month for warm data and ~9 cents for hot, network, provisioned headroom for burst traffic, and monitoring for the monitoring system itself. ### How many engineers do you need to run your own observability platform? A thin stack with low customization needs two generalist senior engineers who own the build, the deploys, and the on-call rotation, plus roughly 20% of a manager's time for stakeholder and adoption work. A custom stack with platform stability tested from the start needs an observability SME, a ClickHouse and TSDB expert, backend and ETL engineers, UX and UI engineers, at least two DevOps engineers, and about 40% of a manager's time. ### Has AI made it cheaper to build your own observability tooling? Yes, for version one. AI increases execution velocity, lets a smaller team accomplish more, and helps engineers work in unfamiliar systems. It does not remove the costs of ownership. The system still needs to run, scale, survive failures, be upgraded, comply, and evolve with your business, and someone still needs to own all of that. AI has made building cheaper. It has not made ownership free.