AI Security Policy Defined in Plain Terms
An AI security policy is a written document that sets the rules for how an organisation secures the AI it builds, buys or lets staff use. It covers acceptable use, data handling, access, the model life cycle, human oversight and incident response and it names who is accountable for each.
The problem is rarely that leaders doubt the value of such a policy. It is that AI use has outrun governance. Nearly three in four European IT and security professionals say staff already use generative AI at work, per ISACA’s 2025 research, yet just 31% of organisations there have a comprehensive AI policy in place.
That gap is the risk. Every day without a policy is a day of ungoverned data going into tools the organisation does not control and ungoverned output coming back out.
What an AI Security Policy Covers
An AI security policy is more than a one-line statement of intent. It turns a broad position on responsible AI into day-to-day security practice with a named owner and a control behind each rule.
It helps to separate three documents. A broad AI policy states the organisation’s position on responsible and ethical AI. An AI security policy is narrower and names the threats, the controls, the owners and the monitoring. The existing cybersecurity policy still applies and the AI security policy extends it into the model lifecycle and the new ways AI can be attacked or misused.
A workable policy covers these areas.
- Scope: Which AI counts from the tools staff use to the models you build or embed.
- Acceptable use: What staff may and may not do, and which tools are sanctioned.
- Data handling: What data may enter an AI tool, classified by sensitivity, and what never may.
- Access and identity: Who can use which AI systems, under least privilege.
- Model lifecycle security: Securing the AI you build or buy from design through to retirement.
- Human oversight: Where a person must review or approve AI output and who is accountable for it.
- Incident response: How an AI-related incident is classified, reported and handled.
- Monitoring: Logging AI usage so the policy is enforced rather than merely written.
- Supply chain: Vetting AI vendors, models and datasets before they are trusted.
- Training and review: Who is trained and how often the policy is revisited.
The Risks an AI Security Policy Must Address
The controls only make sense against the risks they answer. The most referenced map of those risks is the OWASP Top 10 for LLM Applications which a policy can be built around. Five groups matter most for a business.
- Prompt injection: Crafted input that overrides an AI system’s instructions, typed in directly or hidden in a document, email or web page it later reads. OWASP ranks it the top risk.
- Sensitive data disclosure: An AI tool leaking confidential or personal data through its output often because that data was pasted in or retrieved without controls.
- Supply-chain and poisoning risk: A compromised model, dataset or component pulled from an external hub, carrying hidden behaviour into everything downstream.
- Excessive agency: An AI agent wired to take actions such as sending mail or changing records, doing more than intended once it is manipulated.
- Shadow AI: Staff using unsanctioned AI tools with no oversight, the blind spot that makes every other risk harder to see.
The Business Impact of a Missing or Weak Policy
A missing policy does not stay a paperwork problem. It becomes money and risk through four mechanisms.
The first is data loss. When staff paste source code, customer records or strategy into a public tool, that data can sit on servers the organisation cannot reach. If it includes personal data it can become a breach reportable to IMY within 72 hours under GDPR Article 33.
The second is legal liability. A business is answerable for what its AI tells a customer. A 2024 Canadian tribunal held Air Canada liable for wrong advice its website chatbot gave, rejecting the claim that the bot was a separate legal entity.
The third is regulatory exposure. The EU AI Act makes AI literacy a legal duty and sets fines of up to 35 million euros or 7 percent of global turnover for the most serious breaches. NIS2, now law in Sweden, adds fines of up to 10 million euros or 2 percent and can hold board members personally accountable.
The fourth is the attack surface. Every AI tool is a new way in. In 2025 Microsoft patched a critical Microsoft 365 Copilot flaw, nicknamed EchoLeak, that could have let an attacker steal data with no user click. Flaws like this go unmanaged when no policy treats AI as part of the attack surface.
Real-World Cases
Samsung and the Source Code Leak
In early April 2023 engineers in Samsung’s semiconductor division used ChatGPT for coding help and to turn a meeting into minutes. To do it, they pasted confidential source code and internal notes into the tool.
Three such incidents were recorded within about a month, as first reported in South Korean media and later covered by Bloomberg. Because the tool stored the input on external servers, Samsung could not retrieve or delete it. The company restricted generative AI on its devices soon after.
A blanket ban tends to push use into the shadows. The stronger control is an acceptable-use rule paired with a sanctioned, data-protected AI channel so staff get a safe route and the data never leaves the organisation’s control.
Air Canada and the Chatbot’s Invented Policy
A grieving customer asked Air Canada’s website chatbot about bereavement fares. The bot told him he could claim the discount retroactively which was not the airline’s actual policy.
When the refund was refused, he took it to the British Columbia Civil Resolution Tribunal. In Moffatt v. Air Canada (2024), the tribunal rejected the airline’s argument that the chatbot was a separate legal entity found negligent misrepresentation and ordered it to pay damages.
Customer-facing AI needs an owner. The policy should require its answers to be grounded in verified sources and reviewed and state plainly that the organisation, not the bot, is accountable for what it says.
EchoLeak and Microsoft 365 Copilot
In June 2025 Microsoft patched a critical flaw in Microsoft 365 Copilot, tracked as CVE-2025-32711 and nicknamed EchoLeak. A crafted email carried hidden instructions.
When the user later asked Copilot a routine question, the assistant read the email, followed the hidden instructions and exfiltrated internal data, with no click required. It was reported as the first prompt injection to steal data from a production AI system and Microsoft said it found no sign of real-world abuse.
The lesson is to treat AI tools as attack surface. The policy should require security testing, least-privilege access to data for AI assistants and prompt patching, the same discipline applied to any other software.
AI Security Policy and Compliance
An AI security policy does not sit on its own. It is how several obligations are met in practice so write it with the relevant regimes in view.
The EU AI Act, Regulation (EU) 2024/1689, entered into force on 1 August 2024. Since 2 February 2025 it has required AI literacy, meaning staff who use AI must be trained to use it responsibly.
Breaches of the most serious prohibitions carry fines of up to 35 million euros or 7 percent of global turnover. A written policy and a training record are how an organisation shows it meets the literacy duty.
Two frameworks turn the law into practice. The NIST AI Risk Management Framework, released in January 2023, is voluntary and built around four functions with Govern covering the policy, roles and accountability an AI security policy sets.
ISO/IEC 42001, published in December 2023, is the first certifiable AI management system standard and slots alongside ISO 27001. For building secure AI, the CISA and NCSC Guidelines for Secure AI System Development set a secure-by-design baseline across the whole lifecycle.
For organisations in scope of NIS2, the Swedish transposition, Cybersäkerhetslagen (SFS 2025:1506), has applied since 15 January 2026. Its required measures include security awareness training and its board-accountability rule means directors can be held personally responsible for oversight. An AI security policy is part of the risk-management measures the law expects and serious incidents must be reported to MCF (formerly MSB).
GDPR still governs any personal data that touches an AI tool. A leak of personal data into or out of a model can be a breach reportable to IMY within 72 hours under Article 33 so the policy must set what data may enter a tool and what must not.
Financial entities have a further duty. DORA Article 17 requires ICT incident management, supervised by Finansinspektionen and AI incidents fall inside it. The policy’s incident-response section should map to those reporting lines.
How to Tell Your AI Policy Is Failing
A policy can exist and still fail. These are the signs it is not working.
- No inventory: The organisation cannot say which AI tools staff use so shadow AI goes unseen.
- No named owner: Nobody is accountable for keeping the policy current or enforcing it.
- A ban with no alternative: Acceptable use says no but offers no sanctioned tool so staff work around it.
- No logging: AI usage is not recorded so the rules cannot be checked.
- No pre-deployment testing: AI the organisation buys or builds goes live without security testing.
- Written once: The policy was published and never revisited while the tools changed monthly.
- No training: Staff have never been shown the rules which is common. ISACA found 32 percent of organisations give no AI training to anyone.
A written policy is only the start. Without monitoring and a sanctioned route for staff to use AI safely, the gaps above persist no matter how well the document reads.
How to Write and Enforce an AI Security Policy
Writing the policy is the start. Enforcing it is the job. Work through people, process and technology, then make each rule concrete.
- Inventory the AI tools already in use before writing a word so the policy covers reality.
- Name an owner for the policy and its enforcement, usually the CISO, with cybersecurity advisory support where there is no in-house security lead.
- Write acceptable use and in the same breath give staff a sanctioned, data-protected AI tool so the rule has a safe alternative.
- Classify data and state what may go into an AI tool and what never may.
- Apply least privilege to the data AI assistants can reach so a single flaw cannot expose everything.
- Require security testing and red-teaming for any AI you build or buy and patch it like any other software.
- Demand human oversight of AI output where it matters and name who is accountable for it.
- Extend incident response to AI with clear classification and reporting lines.
- Log and monitor AI usage because a rule you cannot see is a rule you cannot enforce.
- Train every member of staff which the EU AI Act now requires.
- Review the policy on a set cadence, not once, and align it to the NIST AI RMF or ISO/IEC 42001.
Start with the inventory this week. Everything else in the policy depends on knowing what you are actually governing.
