Index Migrator
The index migrator ensures Elasticsearch is ready to receive metric data. It runs at startup and creates or updates the ILM lifecycle policy, component templates, index template, and data stream.
Migration Steps
MigrateAsync() runs these operations in order:
- ILM lifecycle policy - controls data retention and rollover
- Component templates - reusable mapping fragments loaded from embedded resources
- Index template - combines components into the final template with time-series settings
- Data stream - created if it doesn't already exist
- Conflict check - validates no other templates share the same priority
Index Template
The index template configures time-series optimised storage:
| Setting | Value |
|---|---|
| Priority | 500 |
| Index mode | time_series |
| Timestamp field | @timestamp (strict date parsing) |
| Routing paths | gatewayId, asset.id |
Mappings are built from component templates generated by the schema pipeline. This means the Elasticsearch mappings stay in sync with the metric definitions automatically.
Lifecycle Policy
The default ILM policy manages data through two phases:
Hot Phase
- Rollover: after 7 days or 50 GB primary shard size
- Active for all new writes
Delete Phase
- Retention: 90 days after rollover
- Indices are deleted automatically
Component Templates
Component templates are embedded as JSON resources in Voltimax.Iot.Contracts. The ElasticsearchTemplateProvider loads them at runtime and the migrator uploads each one to Elasticsearch before composing the index template.
This approach keeps mapping definitions close to the generated metric types - when a new metric is added to the schema, its Elasticsearch mapping is generated alongside the C# code.
Naming Conventions
All names derive from the configured index name (e.g., iot-metrics-voltimax-gateway-prod):
| Resource | Name |
|---|---|
| Index template | {index-name}-template |
| Lifecycle policy | {index-name}-lifecycle |
| Index pattern | {index-name}* |
| Data stream | {index-name} |
Error Handling
- Each step validates the Elasticsearch response and logs failures with full context
- Migration failures log a warning but don't prevent startup - the platform continues with the existing configuration
- Conflicting templates (same priority) are detected and reported
Related Documentation
- Metrics Sink - writes telemetry data to the indices this migrator creates
- Logs Query - querying gateway logs
- Metrics Generator - generates the component templates used here