Housing.com Replaces ELK Stack with Last9, Eliminating Infra Overhead While Cutting Costs

Housing’s self-managed Elasticsearch cluster struggled with the complexity of standardizing logs across multiple business units and technology stacks. The team spent significant effort managing Grok filters at the Logstash level to parse raw application logs from diverse sources.

When developers changed log formats — adding new fields or modifying existing ones — the pipeline would break unless the team manually updated Grok patterns. This created a constant firefighting cycle. Over four years, Housing revamped their internal stack 2-3 times, each time encountering new scaling and reliability challenges.

The 3-4 day retention window created a particularly painful problem: teams would fix production issues first (the priority), but by the time engineers were freed up to conduct thorough root cause analysis, the relevant logs were already deleted. This incomplete evidence trail reduced RCA completion rates and prevented the team from identifying underlying systemic issues.

Housing evaluated multiple managed observability solutions but chose Last9 for its combination of cost-effectiveness, technical capabilities, and engineering-focused support. The support quality was a decisive factor — Housing needed a vendor that understood complex log aggregation requirements and could provide hands-on consultation rather than just documentation.

Last9’s remapping feature replaced Housing’s complex Grok filter management with a more maintainable approach. The team uses regex-based parsing with Last9’s testing tool to quickly validate patterns before deployment. If remapping fails, logs remain available in raw format rather than being lost entirely — a critical safety net that didn’t exist with their ELK setup.

The migration involved close collaboration between Last9’s success engineering team and Housing’s DevOps engineers. Multiple deep-dive sessions helped transition Housing’s team from four years of ELK patterns to Last9’s different visualization and querying approach.

Industry’s Most Complex Remapping Rules

Housing has one of the most complex remapping rule sets. This complexity stems from managing multiple business units with different logging mechanisms and no standardization across teams.

The remapping configuration handles:

• Service name standardization across inconsistent naming patterns
• URL extraction for API performance tracking in their tech scorecard
• Log routing between business units (Housing vs PropTiger) using field attributes
• Dynamic URL pattern aggregation using regex transformation to normalize URLs containing UUIDs and project IDs

For their Databricks integration powering tech scorecard analytics, Housing needed aggregated 5xx error counts and URL analysis. The team migrated from fetching data from ELK to Last9, extracting fields like URL, URL_path, and app_name for service-to-service communication tracking.

This standardization happens at ingestion without touching application code, and changes reflect within 10-15 minutes — compared to their ELK setup that required modifying Grok patterns and restarting services, often breaking the entire pipeline.

Built-in Alerting Replaces Custom Scripts

Housing previously ran auxiliary services that queried ELK data and sent alerts to stakeholders — additional infrastructure they had to maintain. Last9’s alerting capabilities eliminated this overhead entirely.

The team now uses:

• Scheduled searches for log pattern monitoring
• Service-level alerts with customizable thresholds
• No-data scenario detection to catch missing log indexes
• Slack integration for immediate team notification

All alert configuration happens within Last9 without external scripts or coordination with additional services.

Housing successfully deprecated their in-house ELK implementation, allowing their platform engineering team to focus on product development rather than observability infrastructure maintenance. The combination of extended retention, reliable ingestion, and built-in alerting has improved their ability to debug production issues and complete thorough post-mortems.

The team continues to work with Last9’s support on ongoing optimizations, including their Databricks integration for tech scorecard metrics.