59 EU domains scanned. The 11 emerging AI Agent Readiness standards show zero adoption. The only AI-related control with real traction (robots.txt bot blocking) is led by the public sector (60%) and ignored by retail (11%). Passive analysis, first-party data from Hard2bit Scanner, 2026-06-06.
Executive summary
On 6 June 2026 we ran Hard2bit Scanner against 60 public domains belonging to large European companies and agencies. 59 scans completed successfully, 1 timed out. We measured 11 emerging AI Agent Readiness standards, HTTP security headers, email authentication, SMTP transport, TLS posture and threat-intelligence exposure. What we found is an honest snapshot of the EU enterprise market as of mid-2026.
Six headline findings:
- 59 of 59 domains score zero emerging standards implemented: none publishes llms.txt, Content-Signal, MCP Server Card, Web Bot Auth, Agent Skills, OAuth Discovery, API Catalog or any of the other 11 indicators evaluated. Adoption isn't low: it's exactly zero, without a single outlier across banking, pharma, telcos, energy, retail or public sector.
- Only signal with real traction: AI bot blocking via robots.txt. 47% publish some form of policy, with the public sector leading at 60% and retail at the opposite end with 11%.
- 46% publish a Content-Security-Policy header, but only 20% configure it without
unsafe-inline— the rest is CSP theatre. - 100% publish SPF, 98% DMARC, but only 15% MTA-STS — SMTP transport security is still niche territory.
- Banking has the strictest DMARC (100% with
p=rejectorp=quarantine) and the worst CSP hygiene (0% withoutunsafe-inline). - TLS is universally solved: 93% support TLS 1.3, 0% offer weak ciphers.
Message to CISOs: if your organisation is thinking about enabling AI agent discoverability (publishing llms.txt, exposing a public MCP Server Card, declaring APIs via API Catalog), this report shows that no consolidated best practice yet exists across large EU companies. Moving now means defining the de facto standard; getting it wrong opens new exposure surface.
Methodology
What we measured
For every domain we evaluated four blocks of controls:
- AI Agent Readiness — the 11 emerging 2025-2026 standards:
llms.txt,llms-full.txt, Content-Signal (inrobots.txt), Markdown negotiation, API Catalog (RFC 9727), OAuth Discovery, OAuth Protected Resource (RFC 9728), MCP Server Card, Agent Skills, Web Bot Auth (RFC 9421) and Link Header AI. We also tracked AI bot policy inrobots.txt(GPTBot, ClaudeBot, Google-Extended, PerplexityBot, etc.) — a defensive technique that predates 2024 but sits in the adjacent space. - HTTP security headers — HSTS, Content-Security-Policy (with
unsafe-inlinedetection), X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy. - Email authentication and transport — SPF (presence and
~all/-allmode), DKIM visible via CT/DNS, DMARC (presence andp=mode), MTA-STS, BIMI. - TLS and exposure — TLS 1.3 support, weak ciphers, OCSP stapling, threat-intelligence exposure (Spamhaus, URLhaus, PhishTank, Google Safe Browsing).
How we measured
Fully passive analysis with Hard2bit Scanner. Public information only: DNS, TLS certificates, HTTP headers, Certificate Transparency logs, robots.txt, security.txt, llms.txt, public threat-intel lists. Zero anomalous traffic, zero entries in logs or WAF systems, equivalent to what any browser or public DNS resolver would do when visiting the domain.
When and where
Scan run on 2026-06-06 from European infrastructure (Hetzner, Helsinki datacenter). Sample size: 60 domains (10 per sector × 6 sectors). 59 successful scans, 1 isolated timeout.
Sectors and volume
| Sector | Domains analysed | Description |
|---|---|---|
| Banking | 10 | Systemic EU banks and relevant credit institutions |
| Healthcare / pharma | 10 | Pharma multinationals and major healthcare providers |
| Retail / e-commerce | 10 | Retail chains and marketplaces with EU presence |
| Telcos | 10 | Large telecom operators and ISPs |
| Energy / utilities | 10 | Electric, gas and grid operators |
| Public sector | 10 | National cyber agencies and EU data protection authorities |
Limitations worth declaring
- Sector sample size (10): sufficient for aggregate trends, insufficient for granular per-sector statistical inference.
- Single observation point (Europe): we don't detect geo-specific configurations served only to other regions.
- Standards still in draft: absence ≠ rejection. For
llms-full.txtor OAuth Discovery sub-checks, zero adoption may reflect not-yet-implemented rather than actively-rejected.
Per-sector anonymisation
Figures are published aggregated by sector. We do not attribute individual findings to specific domains. This is deliberate: the report describes the state of the market, not specific companies. The full list of scanned domains is available on reasoned request for academic researchers and accredited auditors.
Ethics
Public domains of well-known organisations, non-intrusive analysis, equivalent to journalism on public digital transparency. No internal access, no active probing, no anomalous traffic generation.
Finding 1 — Eleven emerging standards, eleven 0% rates
The first result is the clearest: of the 12 AI Agent Readiness indicators measured, 11 show exactly zero adoption across the 59 domains analysed. The only one with real traction (AI bot blocking in robots.txt) isn't even an emerging standard, it's a defensive technique consolidated before 2024.
| AI Agent Readiness indicator | Presence (n/59) | % |
|---|---|---|
| llms.txt | 0 | 0% |
| llms-full.txt | 0 | 0% |
| Content-Signal (robots.txt) | 0 | 0% |
| Markdown negotiation | 0 | 0% |
| API Catalog (RFC 9727) | 0 | 0% |
| OAuth Discovery | 0 | 0% |
| OAuth Protected Resource (RFC 9728) | 0 | 0% |
| MCP Server Card | 0 | 0% |
| Agent Skills | 0 | 0% |
| Web Bot Auth (RFC 9421) | 0 | 0% |
| Link Header AI | 0 | 0% |
| AI bot policy in robots.txt | 28 | 47% |
Interpretation: although the standards have been under IETF discussion and community proposals since 2024-2025, adoption among 59 large European organisations as of June 2026 is statistically zero. The gap between spec publication and enterprise adoption remains the usual bottleneck.
What is llms.txt and why its absence is telling
llms.txt is a plain-text file at the root of a domain that gives AI agents a curated index of the site's important content in Markdown format. Proposed by Jeremy Howard in September 2024 with community backing. Its total absence across 59 large EU organisations indicates that AI-agent-oriented discoverability is simply not on the agenda of web teams yet. The first organisation to implement it well defines the de facto pattern competitors will follow.
What is Content-Signal and what its absence says about AI governance
Content-Signal is a W3C/IETF proposal that extends robots.txt with metadata about which uses of content are allowed: classic indexing, model training, generative search, etc. It enables granular consent. Total absence in the sample means that organisations blocking AI bots do so in binary mode (yes/no), without differentiating between use cases (training vs inference vs search). Consistent with the immaturity of the standard itself.
What is MCP Server Card and why its absence isn't surprising
MCP Server Card is the public declaration of a server compatible with Model Context Protocol, Anthropic's standard for connecting agents with tools. Exposing a public MCP server is a deliberate decision and uncommon at this point in time; zero adoption mostly reflects that most organisations don't yet operate public MCP infrastructure. Expected, but confirms that the enterprise ecosystem is in a very early phase.
Web Bot Auth (RFC 9421) — the absence with most risk
RFC 9421 introduces HTTP signatures that let you distinguish a legitimate agent (signed by its operator) from a malicious scraper spoofing its User-Agent. Without Web Bot Auth, robots.txt and X-Robots headers are purely honorary signals: an attacker ignoring Disallow cannot be technically identified. Total absence means the market still depends on AI operator fair-play, not cryptographic verification.
If you want a quick baseline of how your own domain stands against these 11 standards, scan your domain with Hard2bit Scanner in under a minute — free anonymous scan, no card required.
Finding 2 — The public sector blocks more AI bots than banking
AI bot blocking via robots.txt is the only AI-related control with real adoption. Where present, it tends to be a binary block against all detected AI user agents. Public sector leads (60% of domains with policy present and active blocking). Retail sits at the opposite extreme (11% blocks, against 22% that publish any policy):
| Sector | Policy present | GPTBot | ClaudeBot | Google-Ext | Perplexity |
|---|---|---|---|---|---|
| Public sector | 60% | 60% | 60% | 60% | 60% |
| Energy | 60% | 50% | 50% | 50% | 50% |
| Telcos | 50% | 40% | 40% | 40% | 40% |
| Banking | 50% | 40% | 40% | 40% | 40% |
| Healthcare | 40% | 20% | 30% | 30% | 30% |
| Retail | 22% | 11% | 11% | 11% | 11% |
Public sector leads with 60%. Reasonable hypothesis: combination of regulatory mandate (AI Act, GDPR, ENISA guidance), conservative tradition on algorithmic transparency, and low commercial pressure to appear in generative answers.
Retail / e-commerce sits at the bottom with 11%. Hypothesis: web teams focused on lead capture, explicit willingness to appear in Perplexity and ChatGPT answers, underestimation of the training-without-consent risk.
Important observation: organisations that block, block all AI bots equally. We see no differentiated policy in the sample between, say, generative-search bots (desirable for answer SEO) and training bots (undesirable for unconsented datasets). This suggests that in 2026 policy is applied as total defensive blocking, not as granular consent governance — consistent with the immaturity of the Content-Signal standard, which if adopted would allow that granularity.
Finding 3 — Content-Security-Policy is theatre in 80% of cases
46% (27/59) publish a Content-Security-Policy header. But only 20% (12/59) configure it without unsafe-inline — a directive that allows inline JavaScript execution and neutralises most of the XSS protection CSP is supposed to provide. 25% (15/59) publish CSP with unsafe-inline. The rest don't publish.
| Sector | CSP present | CSP without unsafe-inline |
|---|---|---|
| Energy | 80% | 40% |
| Telcos | 50% | 30% |
| Public sector | 40% | 20% |
| Retail | 22% | 22% |
| Banking | 20% | 0% |
| Healthcare | 40% | 0% |
Energy leads with 80% presence and 40% without `unsafe-inline` — the sector that best combines coverage and quality. Banking has 20% presence and 0% without `unsafe-inline`: when present, it's broken. Healthcare repeats the pattern (40% present, 0% clean). Retail is curious: low coverage (22%) but everything published is well configured (22% without unsafe-inline).
NIS2 implication: Article 21 demands technical, operational and organisational measures to manage cybersecurity risk. A CSP header with unsafe-inline doesn't meet the spirit of an effective technical measure — it's a checklist compliance signal rather than real protection. NIS2 and ENS auditors should inspect header quality, not just presence.
Actionable recommendation for CISOs: if your organisation publishes CSP, audit it with a serious evaluator (Mozilla Observatory and Google's CSP Evaluator are free). If you find unsafe-inline, plan migration to nonce or hash within 6 months at most. Medium-high effort but real defensive return. Until then, CSP with unsafe-inline disabled performs almost as little as no CSP at all.
Finding 4 — Email armoured, transport vulnerable
Outbound email authentication (SPF, DKIM, DMARC) is universally adopted in large EU organisations. Inbound transport security (MTA-STS, requiring TLS on receive) is still niche:
- SPF: 100% (59/59). 97% (57/59) in strict mode (
-allor~all). - DMARC: 98% (58/59). 83% (49/59) with
p=rejectorp=quarantine. - DKIM visible via CT/DNS: 73% (43/59).
- MTA-STS: 15% (9/59) — still a minority.
- BIMI: 20% (12/59) — more adopted than expected, mostly in retail.
| Sector | SPF strict | DMARC strict | DKIM visible | MTA-STS | BIMI |
|---|---|---|---|---|---|
| Banking | 100% | 100% | 70% | 0% | 30% |
| Public sector | 100% | 90% | 70% | 20% | 10% |
| Healthcare | 100% | 90% | 100% | 20% | 20% |
| Telcos | 100% | 80% | 70% | 20% | 10% |
| Energy | 100% | 80% | 70% | 20% | 0% |
| Retail | 90% | 60% | 60% | 10% | 56% |
Reading: the frontend of email security (sender authentication) is mature. The backend (mandatory encrypted transport between mail servers) is still an improvement area. Banking is paradigmatic: perfect DMARC, zero MTA-STS.
NIS2 supply-chain implication: several documented EU 2025-2026 incidents (including the M365 account takeover patterns) involve compromised email at suppliers. Without MTA-STS, inbound email can be intercepted in transit by on-path attackers (STARTTLS downgrade). For NIS2-bound entities receiving sensitive data via email, MTA-STS stops being optional.
Retail at 56% BIMI is the most striking data point: reflects brand sensitivity and marketing maturity. BIMI requires DMARC in p=reject or p=quarantine mode + VMC certificate, so the sector with most BIMI is also the one investing most in strict DMARC — coherent.
Finding 5 — Banking: internal contrast between email and web perimeters
Banking-specific analysis, where the contrast between email-perimeter maturity and web-perimeter maturity is the most pronounced in the sample:
- Strict DMARC: 100% (best in sample)
- Strict SPF: 100%
- CSP without `unsafe-inline`: 0% (worst in sample)
- X-Frame-Options: 60% (medium)
- HSTS strong: 60% (medium)
Interpretive hypothesis: specialised teams defend distinct perimeters. The email security team in banking has been integrated with anti-fraud and anti-phishing for 20 years through specific regulation (PSD2, ECB cyber resilience, EBA recommendations). Very high maturity. The web security team is usually integrated with development and AppSec, operates with different priorities and carries more legacy in HTTP headers.
Reading for the profession: when there's specific regulation targeting a concrete risk (phishing using a banking domain), the sector responds with armoured DMARC. When regulation is generic (reasonable technical measures), practices relax. The conclusion: specific regulation works, and DORA does well in demanding technical granularity rather than generic invocations of best practice.
Insight for banking CISOs: if your organisation has perfect DMARC and CSP with unsafe-inline, it's not coincidence nor an isolated oversight. It's a structural pattern. Make sure AppSec gets the same sustained attention as email anti-fraud. An independent web security audit flags these imbalances without internal politics in the way.
Finding 6 — TLS and crypto: the battle won
The only block of the report where the result is mostly positive:
- TLS 1.3 supported: 93% (55/59)
- Weak ciphers: 0% (0/59)
- OCSP stapling: 41% (24/59) — the only remaining gap
- Threat-intel exposure (Spamhaus, URLhaus, PhishTank, Google Safe Browsing): 0% (0/59) — no domain listed
HTTPS cryptography is done. Good news that breaks the otherwise mostly grim pattern of the report. Collective efforts of the last decade — Let's Encrypt democratising certificates, HSTS preload lists, effective deprecation of TLS 1.0/1.1, coordinated browser pressure — have worked.
OCSP stapling at 41% is the only weak point. Improves privacy (browser doesn't query the certificate issuer directly) and revocation latency. Incremental quick win: enabling it at the reverse proxy / load balancer takes hours, not weeks. Recommended for leading sectors (energy, telcos) as the next step.
Implications for your organisation
For CISOs
- If your organisation is thinking about enabling AI discoverability (publishing
llms.txt, exposing public MCP server, declaring APIs via API Catalog), you're a pioneer. That means responsibility to configure it well before exposing it — and opportunity to establish the practice your competitors will copy. - Prioritise auditing what you already have misconfigured (CSP with
unsafe-inline, MTA-STS absent) before adding new vectors. Allms.txtpublished on a site with broken CSP adds exposure without adding protection. - Document the decision to block or allow AI bots. The policy matters when the next NIS2 or ISO 27001 supply-chain audit comes around.
For auditors (NIS2, DORA, ENS, ISO 27001)
- CSP with
unsafe-inlineis a clear signal of a merely formal technical measure — ask for evidence of a planned migration with a calendar. - Absence of MTA-STS in organisations receiving sensitive data via email is a documentable supply-chain weakness.
- AI Agent Readiness configuration (presence or absence, applied policy) should be included in future frameworks as part of auditable exposed surface. Nobody asks today; tomorrow they will.
For cyber consultants
- The market has a real gap in combined AI Agent Readiness + traditional web security assessment. Current offer splits between SEO/AEO (discoverability without security) and red team / pentesting (classic vulnerabilities without AI). The middle ground is open.
- Data from this report is direct material for client conversations: "do you have
llms.txt? have you audited it for exposures? is your CSP real or theatre? are you receiving email over MTA-STS?"
For developers / SRE
- If you decide to implement
llms.txtor MCP Server Card, do it with the same discipline as any other endpoint: review, audit, monitoring, credential rotation if applicable. - Start with the simple stuff (well-curated
llms.txt,robots.txtwith explicit AI policy). A public MCP server requires additional infrastructure (auth, rate-limit, logging) — treat it as a production-grade service, not as an experiment.
Frequently asked questions
These questions and answers are also published as FAQPage JSON-LD to make them usable by search engines and AI agents.
Conclusion
The 11 emerging AI Agent Readiness standards are the new exposure perimeter of your website. As of June 2026, large European companies and agencies haven't started configuring it. Whoever moves first defines the best practices the rest will copy. But getting it wrong opens new vectors — content exposure without consent, unauthorised scraping, public MCP infrastructure without robust auth — on top of the classic surface that the same report shows riddled with incomplete configuration (CSP theatre, missing MTA-STS).
Honest recommendation: audit what you already have before adding more. Crypto is done, email authentication too, but Content-Security-Policy quality and SMTP transport remain pending homework in most sectors. The AI Agent Readiness layer is built on those foundations, not in parallel.
Scan your domain for free at scan.hard2bit.com — passive analysis, 30-60 seconds, no card. If you need professional audit with defined contractual scope, let's talk or learn more about our cybersecurity audit service.
Methodological appendix
Scanner version and date
- Tool: Hard2bit Scanner (SaaS version June 2026)
- Scan date: 2026-06-06 (all domains in the same window, same day)
- Scan origin: European infrastructure (Hetzner, Helsinki datacenter)
- Sample size: 60 target domains, 59 successful scans, 1 timeout (mediamarkt.com)
- Volume per sector: 10 domains each
Checks evaluated
List of checks run by the scanner on each domain, grouped by the product's internal category:
- Network: TLS / SSL, DNS health, Exposed ports
- Web: HTTP security headers (HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy), Detected technologies, Known public vulnerabilities (CVE), Cookie configuration, Mixed content on secure pages
- Identity: Email security (SPF, DKIM, DMARC, MTA-STS, BIMI), Domain status, Certificate Transparency
- Data exposure: AI-era security posture, Exposed cloud storage, Leaks in pastes and repositories, Subdomain takeover risk, AI dataset exposure, AI bot blocking, Certificate Transparency subdomains, Vendor breach exposure
- Reputation: Threat intelligence (URLhaus, Feodo, Spamhaus, PhishTank, Google Safe Browsing)
- Compliance: security.txt file, Compliance signals (cookies, privacy, GDPR, accessibility), robots.txt file
- AI Agent Readiness: 11 emerging standards — llms.txt, llms-full.txt, Content-Signal, Markdown negotiation, API Catalog (RFC 9727), OAuth Discovery, OAuth Protected Resource (RFC 9728), MCP Server Card, Agent Skills, Web Bot Auth (RFC 9421), Link Header AI
How to reproduce or verify the data
Any domain can be re-scanned individually at scan.hard2bit.com with the same tool and obtain results directly comparable to those published here. The complete list of scanned domains is available on reasoned request for academic researchers, accredited auditors and technology journalists at hard2bit.com/en/contact/.
Bibliography and references
- llms.txt specification (llmstxt.org)
- Model Context Protocol (modelcontextprotocol.io)
- RFC 9421 — HTTP Message Signatures (Web Bot Auth) (IETF)
- RFC 9727 — API Catalog (IETF)
- RFC 9728 — OAuth 2.0 Protected Resource Metadata (IETF)
- W3C Content-Signal — proposals in progress (W3C)
- ENISA — European cybersecurity agency (enisa.europa.eu)
- NIS2 Directive (European Commission)
- AI Act (consolidated text)
- DORA Regulation (EU 2022/2554) (EUR-Lex)
About the authors and editorial methodology
This report has been produced by the Hard2bit S.L. R&D team. Hard2bit is a Spanish cybersecurity company with over 10 years of experience, certified ISO 27001, ISO 9001, ISO 14001, ISO 22301 and ISO 20000-1, operating with ENS HIGH category. Active members of ISMS Forum, ASLAN, CyberMadrid and UN Global Compact. The AI-cybersecurity intersection research line is run by the R&D team.
Hard2bit Scanner is the proprietary SaaS tool that generated the data for this report. Published at scan.hard2bit.com with a free anonymous tier and paid plans starting at €19/month.
Methodological notice: this report is first-party research with declared sample size (n=59) and public methodology. The figures and percentages exactly reflect the result of the 2026-06-06 scan; they are not extrapolable to the entire European corporate universe without replication with larger samples. The report does not replace a professional audit with defined contractual scope. For formal evaluations with professional responsibility, contact the Hard2bit team.