Which logs to send to your SIEM and which to leave out: priority sources, retention and cost

In its fifth annual detection risk report, CardinalOps analysed production SIEM environments and found that organisations ingest enough data to cover, potentially, 90% of MITRE ATT&CK techniques. They hold detections for 22% of the full set of techniques.

Volume is not the constraint. The SIEMs in that sample ingested an average of 23,746 distinct log sources across 259 source types, up 18% and 30% respectively on the previous year. More data, barely more coverage: the year before came in at 19%, and the five-year average sits around 20%.

So the useful question is not how many logs you send. It is which ones, carrying which fields, into which storage tier, and for how long. This guide works through those four decisions: the sources that earn their place in the expensive tier, the volume that can sit in cheap searchable storage without costing you an investigation, how to notice when a source goes quiet, and what Spain's National Security Framework (the ENS) and the delegated regulation behind DORA actually require on logging and retention.

Why does sending everything to the SIEM go wrong?

Connecting every available source has a surface logic: if the data is there, one day it will be useful. Three separate things break that logic, and only one of them is the bill.

The pricing model charges for weight, not for value

Nearly every platform bills per gigabyte ingested or per committed daily capacity, which means a debug log from an internal application that feeds no detection costs the same per gigabyte as the authentication events from your corporate directory. You pay by the gigabyte, and the gigabyte has no idea whether it is useful. The two candidates behind a runaway bill tend to be the same: raw network telemetry and verbose application logs.

A source with no rule detects nothing

Ingestion is necessary and nowhere near sufficient. A log sitting in the SIEM does not create a detection; somebody has to write the rule, map it to a technique, tune it and maintain it. The opening figure describes precisely that gap between what goes in and what gets detected, and it is worth being exact about how it is measured. The report defines coverage as having at least one rule mapped to an ATT&CK technique, which makes it a measure of presence, not of effectiveness. A rule can exist, be mapped, and still detect badly.

Rules break, and they break because of data

The same report found that 10% of SIEM rules were broken, against an average of 163 rules per organisation: around sixteen that will not work as written. That is an improvement on the previous year's 18%, but the cause matters more than the trend. Rules fail because of misconfigured data sources, fields that stop arriving and log parsing errors. Faulty logic is rarely the culprit.

A note on the source, in fairness to the reader: CardinalOps sells a detection posture management platform and therefore has an interest in these numbers looking dramatic. Its methodology is nonetheless one of the few built on configuration metadata from real production SIEMs instead of survey responses, and the report states plainly that covering 100% of ATT&CK is neither realistic nor what the framework was designed to encourage. And a clarification for anyone going to the report itself: the figures quoted here come from the 2025 sample reported in the body of the document, while its cover page gives 21% coverage and 13% broken rules as an aggregate across all five years of the series.

What should you ask before connecting a source?

The usual sequence runs backwards. Teams connect whatever ships with a native connector and then look for something to do with it. These four questions invert that, and each takes minutes to answer per source.

1. Which specific detection does it enable? If the answer is “general visibility”, the source does not belong in the expensive tier. You should be able to name a rule or a hunting hypothesis, and ideally the ATT&CK technique it addresses.
2. Does it carry the fields that detection needs? An access log without a user identity, a source address or an operation result will not sustain any authentication rule, however much space it occupies.
3. Is the format stable? In-house applications and SaaS services change their schema without notice and break parsing. That instability is a maintenance cost, and it is better accepted deliberately than discovered later.
4. Does it need to sit in the analytics tier, or is searchable storage enough? This is the question that saves the most budget, and the one asked least often.

Which sources earn their place in the expensive tier?

The joint guidance on event logging and threat detection published on 22 August 2024 by the Australian Cyber Security Centre, part of the ASD, alongside CISA, the FBI, the NSA, the UK's NCSC and ten further agencies from Canada, New Zealand, Japan, South Korea, Singapore and the Netherlands sets out a sensible prioritisation rule: start with critical systems and the data holdings most likely to be targeted, then internet-facing services, taking in remote access, network metadata and the underlying server operating system. In industrial environments the equivalent test is devices critical to safety and service delivery, air-gapped systems excepted, together with devices exposed to the internet and devices reachable across network boundaries.

Turned into specific sources, ordered by what tends to pay off in a typical corporate estate:

The logic behind the top half of that table is straightforward. These are the sources that record the use of an identity and the execution of code, the two things a large share of current attack chains touch at some point. An intruder arriving with valid credentials and working with built-in system tooling triggers no antivirus alert, but does leave a trail in the directory and in EDR telemetry. It is why detecting Kerberoasting, AD CS abuse or a Golden Ticket depends more on having directory auditing configured properly than on adding another source.

The network edge deserves a warning of its own. Perimeter devices tend to log sparsely and awkwardly, and they are the preferred way in whenever a mass-exploited flaw appears. We have written separately about why those devices are an EDR blind spot and how to cover them without an agent; for this guide the practical consequence is enough. A firewall or VPN that only emits connection-accepted and connection-denied events will not let you reconstruct an incident.

Identity sources carry a second argument that rarely makes it into a source inventory. Once an attacker holds a valid session, most of what follows looks like ordinary administration, which is why Microsoft 365 account takeover through AiTM and token theft is usually visible only in sign-in and audit telemetry.

Why does log quality matter more than source count?

The same joint guidance presses a point that source spreadsheets almost never capture: format and timeliness shape detection as much as existence does. Its recommendations reduce to four specific demands.

• A structured, consistent format such as JSON, with the same schema, format and ordering within each log type.
• Automated normalisation, which matters most for logs that change over time or without notice, such as software and SaaS logs.
• Time servers synchronised and validated across every environment, with UTC as the preferred standard, and capture of significant events such as device boots and reboots.
• Timely ingestion. Delay the generation, collection or ingestion of events and you delay the identification of the incident by the same margin.

For judging whether a given log is good enough there is a useful shortcut from an unexpected place. Spain's National Security Framework sets out, in control op.exp.8, the minimum content of an audit record: the identifier of the user or entity associated with the event, the date and time, which information the event affected, the event type, and the result, success or failure. Five fields. A log missing any of them will struggle to sustain a rule that survives an investigation, whether or not the ENS applies to you.

What happens when a source goes quiet?

This is the silent failure almost nobody discusses, and it sits directly alongside the broken-rules statistic. A source that stops sending does not raise an alert. It produces silence, and silence looks a great deal like nothing happening. An agent version change, a rotated integration credential, a new firewall rule or a full disk is enough for one event type to vanish for weeks unnoticed.

European financial regulation is unusually explicit here, and worth reading even if you are not a financial entity. The delegated regulation behind DORA requires, among its logging measures, mechanisms to detect a failure of the logging systems themselves. Not detecting the attack: detecting that the system meant to record it has stopped working. It is one of the few regulatory obligations that converts directly into an engineering ticket.

Implementing it takes discipline more than ingenuity. Establish a baseline of volume and frequency per source, alert when the flow drops below expectation within a defined window, and treat that alert as seriously as a security one. The sharper version watches for the presence of specific event types rather than aggregate volume alone, because a source can carry on emitting its usual noise while quietly dropping the one category that feeds your three best rules.

How long must you keep logs under the ENS and the DORA regulation?

A lot of invented numbers circulate on this. The texts say far less than the folklore does.

ENS: you set the retention, but you write it down

Royal Decree 311/2022 imposes no universal number of months. Control op.exp.8, activity logging, applies according to the level assigned to the traceability dimension, not to the system's overall category, and that distinction is frequently muddled. At the low level it requires the audit record with its five mandatory fields and activity logging switched on across servers. At the medium level four reinforcements apply: periodic and informal review of the records looking for abnormal patterns, system clock synchronisation, retention, and access control.

The retention reinforcement says this and no more: the system's security documentation must state which security events will be audited and how long records are retained before deletion. The access control reinforcement adds that activity records and any backups of them may only be accessed or deleted by duly authorised personnel, which stops the administrator of an audited system from erasing their own trail.

At the high level a fifth reinforcement is added, and it is the one that turns this into a project: tools to analyse and review system activity and audit information in search of possible or actual security compromises, plus an automated system for log collection, event correlation and automated response. Put plainly, at high traceability the ENS describes a SIEM with correlation and automation without using the word. Control op.mon.3, surveillance, sits alongside it, applies by system category, and at high category chains six reinforcements from event correlation through to periodic security inspections. If a certification is in view the detail matters, because not all ENS certificates are equal and scope decides their value.

DORA: five event categories and a reasoned retention period

Commission Delegated Regulation (EU) 2024/1774, which sets the technical requirements for ICT risk management under DORA, devotes Article 12 to logging. It requires financial entities to document procedures identifying the events to be logged, the retention period and the measures securing and handling that data, and to align the level of detail with the purpose so that anomalous activity can be detected effectively. It then lists five categories of event that must be logged in every case:

• Logical and physical access control, and identity management.
• Capacity management.
• Change management.
• ICT operations, including ICT system activities.
• Network traffic activities, including ICT network performance.

Three further requirements routinely get left out of implementation projects. Logging systems and logged information must be protected against tampering, deletion and unauthorised access at rest, in transit and, where relevant, in use. Failures of the logging systems must be detectable, the requirement noted earlier. And the clocks of each of the financial entity's ICT systems must be synchronised to a documented, reliable reference time source. On the period itself the regulation again declines to give a number, requiring instead that it be set with regard to business and information security objectives, the reason the event is recorded, and the results of the ICT risk assessment.

NIS2 pushes in the same direction through its risk management and incident handling measures, with the technical detail delegated to implementing acts for particular categories of entity. For anyone making the decision today the practical upshot is the same across all three frameworks: you choose the period, and you must be able to defend the choice.

How do you resolve the tension between detecting and retaining?

Separating the two resolves it, instead of paying detection-grade prices across the whole volume. Most platforms now offer storage tiers at different prices with different query capabilities. Microsoft, for instance, documents several Azure Monitor tiers with distinct costs and limitations, and other vendors have equivalents under different names.

The split that works is a blunt one. Into the analytics tier goes whatever real-time rules query and whatever the team touches daily during investigations: identity, endpoint, edge, cloud control plane. Into the cheap tier goes the volume needed to reconstruct an incident or satisfy a retention period but which feeds no rule at all: flow records, internal traffic, older history from the expensive sources.

The trade-off deserves stating in full, since vendors tend to leave it out. Data moved to the cheap tier generally loses real-time correlation, is slower to query and supports fewer operations. It is there to be searched, not always to detect. Moving a source is therefore a detection decision before it is a cost decision, and it is best taken by looking at which rules depend on that source before looking at the invoice. The joint guidance points at the same model when it recommends centralising and aggregating events first, then forwarding only selected, processed logs to the analytic tools.

Where do you start when the SIEM is already full?

Almost nobody starts from nothing. What you normally inherit is a SIEM with two hundred connected sources, a cost nobody can fully justify and a vague sense that too little is being caught. The sequence that works inverts the source catalogue.

1. Write down the threats that genuinely apply, drawn from your sector and your critical assets, not from the full ATT&CK matrix.
2. For each one, define the detection that would cover it and note the fields that detection requires.
3. Check whether any connected source delivers those fields today. This is where the surprises appear: sources present but missing the field, or delivering it truncated.
4. Review which existing rules depend on sources that no longer send or that changed schema. That review usually recovers more coverage than any new connection would.
5. Only then decide what to add, and send it to the tier its real use justifies.

It is slow work and no tool does it unattended. At Hard2bit we take it on as part of the managed SOC and of ENS and DORA readiness work. All three end at the same question, which is what gets recorded, at what quality, and for how long. If you are still settling the operating model before the sources, start with our guide to what a SOC is and which services it can include.

What the opening numbers suggest is counterintuitive and rather freeing. Most organisations do not need more data in the SIEM. They need detections built on the data already there, complete fields in the sources they already pay for, and an alert when one of those sources falls silent. Before signing off the next licence increase, try establishing how many of today's rules would fire if somebody reproduced, on your network, the technique each one claims to cover. That percentage will shape a budget better than any source catalogue.

This guide sets out general principles of logging and detection architecture for defensive and educational purposes. Source priorities, retention periods and tiering decisions must be adjusted to each organisation's threat model, architecture and applicable regulatory framework, and validated in a test environment before anything changes in production. The reading of the ENS, DORA and NIS2 offered here does not replace regulatory advice on a specific case. If you need your logging and detection strategy reviewed, Hard2bit can help.