Normalise
Parse vendor payloads into stable fields for actor, source IP, action, target, outcome and event category. Preserve the original event for investigation.
Test:Known samples must parse without field loss or timestamp drift.Security engineering · Lab implementation
Building an operable security monitoring platform.
A vendor-agnostic monitoring architecture designed around useful telemetry, predictable routing and detections analysts can actually investigate.
5 source familiesCloud, identity, endpoint, network, SaaS
3 pipeline stagesNormalise · enrich · route
Tiered retentionUse case and risk driven
PortableTechnology-independent design
Problem
Collection is not visibility.
The lab represents a hybrid organisation with productivity-suite audit logs, cloud control-plane and identity telemetry, network security events, endpoint alerts and API-fed SaaS activity. The constraint is realistic: limited storage and analyst capacity mean every log cannot be retained forever or treated as equally valuable.
The architecture therefore starts with detection and investigation requirements. Collection, processing, schemas, storage and response remain separable capabilities, allowing each component to be replaced without redesigning the whole monitoring service.
Diagram 01 · Telemetry lifecycle
From source event to analyst decision
TELEMETRY SOURCES
IdentityCloudEndpointNetworkSaaS
API · SYSLOG · AGENT
PROCESSING PLANE
01Collect02Normalise03Enrich04Route
CANONICAL EVENTS
SECURITY OPERATIONS
DetectSearchInvestigateRespond
CONTROL HEALTHSource heartbeatPipeline failuresStorage capacityAlert delivery
Engineering detail
The pipeline is the control plane.
Enrich
Add asset criticality, environment, identity context and network ownership. Enrichment failure is observable and never silently drops the source event.
Test:Controlled events resolve to the expected asset and identity context.Route
Streams separate security, platform and compliance use cases. Rules route events to indexes with the correct access, retention and performance profile.
Test:Each fixture reaches one intended stream with no duplicate indexing.Diagram 02 · Storage design
Retention follows value, not habit.
Identity, endpoint and high-signal security events optimised for search and alerting.
→
Broader cloud, SaaS and network telemetry used for correlation and retrospective analysis.
→
Selected audit records retained according to business, legal and assurance requirements.
Operational safeguards
Monitor the monitor.
Source heartbeat alerts identify silent collectors, delayed APIs and falling event volume. Journal utilisation, processing failures, extractor errors, index health and disk capacity are treated as security control health—not merely infrastructure metrics.
Access is role-based, transport is encrypted, service credentials are scoped, and administrative activity is audited. A detection is only promoted after a known event triggers it, the alert contains enough evidence to triage, and failure conditions are understood.
Validation record
Ingestion
Replay representative source events and confirm timestamps, encoding and delivery.
Pass: searchable within the agreed delaySchema
Compare parsed fields with the original payload and canonical field contract.
Pass: required fields populatedRouting
Assert stream membership, index selection, access control and retention.
Pass: no loss or unintended duplicationDetection
Generate a controlled behaviour and follow the alert through analyst triage.
Pass: actionable evidence and owner