Root Cause Analysis

Catalog Service Migration Failure During 4.2.0 Release

Date of Incident: March 27, 2025 09:29 UTC

Resolved: March 27, 2025 10:27 UTC

Duration: 58 minutes

Severity: P2

Affected Services/Functionality

• Catalog Service

• Data Reliability Features:

  • Policy Listing Page
  • Asset Discovery Page

‌

Incident Summary

During the 4.2.0 release deployment, a database migration failure in the Catalog Service caused partial

system degradation. The issue stemmed from a schema validation mismatch between the database

layer and application ORM, specically around tenant_id eld length validation.

Impact:

• Intermittent failures in Data Reliability features
• Inconsistent UI rendering
• Mixed-version service communication issues

‌

Root Cause Analysis

Primary Cause: Schema Validation Mismatch

Database vs. ORM Inconsistency:

*The migration failed when processing a tenant_id with 37 characters

Technical Breakdown:

1. Database Schema: Allows 64-character tenant_ids
2. ORM Entity:

‌ @Table(name = "tenants")

‌ class Tenant : IntIdTable() {

‌ val tenant_id = varchar("tenant_id", 34) // Constraint

‌ }

1. Migration attempted to process existing 35-character ID
2. ORM validation rejected the value before DB interaction

The migration failed when encountering a tenant_id exceeding 34 characters.

Failure Chain

1. Migration script aborted due to ORM validation
2. Catalog Service pods failed initialization
3. Kubernetes maintained previous version pods
4. Version mismatch caused API inconsistencies

‌

Detailed Incident Timeline

Investigation Phase

Hotx Development

Build & Deployment

‌

Resolution

Corrective Actions:

1. Deployed hotfix with:

   1. Updated ORM constraints (64 chars)
   2. Additional migration validation

2. Verified all tenant_id values in production

3. Validated cross-service compatibility

‌

Preventive Measures

‌

Key Learnings

Testing Gap: Need real-production-data migration testing