AI Security Awareness Training, Defined
AI security awareness training teaches employees to recognise and resist attacks that use artificial intelligence such as deepfake calls and AI-written phishing and to use AI tools at work safely without exposing sensitive data. It is security awareness training updated for AI-driven threats and the everyday use of AI in the workplace.
The change is recent and fast. In its January 2024 assessment, the UK National Cyber Security Centre judged that AI will almost certainly increase the volume and heighten the impact of cyberattacks with its biggest effect on social engineering. The people in your organisation are now the main target and the tools they use every day are part of the risk.
The training therefore has two jobs. The first is defence, helping staff recognise and stop AI-enhanced attacks. The second is safe use, helping staff get value from AI tools without leaking data or acting on an unchecked answer. Older awareness programmes covered neither well.
Why AI Changed the Threat to Your People
The human being has always been the softest target. What AI changed is the cost. A single attacker can now produce fluent, personalised messages in any language, clone a voice from a short recording and generate video of a real person’s face from clips found online.
The National Cyber Security Centre reached the same conclusion. Its assessment found that AI offers the biggest uplift in social engineering and that generative AI removes the spelling, grammar and translation mistakes that once gave phishing away. The FBI has warned that criminals use generative AI to commit fraud at larger scale and to make each attempt more believable.
There is a second pressure, pointing the other way. The tools that attackers use are the same tools your staff adopt to work faster. When an employee pastes a contract or source code into a public chatbot, sensitive data can leave the company. So the human layer now faces two AI risks at once, better attacks aimed at it and new ways for it to cause harm.
The AI Threats Your Team Must Recognise
Effective training is built around the specific ways AI is used against people and the ways people misuse AI. The main categories are short.
- AI-written phishing and business email compromise: Fluent, tailored emails and messages with no obvious errors often impersonating a manager or a supplier.
- Voice cloning: A familiar voice on a call or in a voice note, built from a short clip of recorded speech, used to push an urgent payment or a password reset.
- Deepfake video: A live or recorded video that impersonates an executive or a colleague as in the Arup fraud below.
- Fraud at scale: Fake profiles, forged documents and convincing fake websites, generated quickly to support a scam.
- Shadow AI and data leakage: Staff entering confidential data into public AI tools or using AI apps the organisation has not approved.
- Over-trust in AI output: Acting on an AI-generated answer, summary or line of code without checking it.
Every item on that list is a decision made by a person. That is why the training focuses on behaviour rather than on the technology itself.
The Business Cost of an Untrained Workforce
The cost lands in four places. The first is direct financial loss. A single deception can move very large sums, as Arup found when one employee authorised transfers worth about 25.6 million US dollars.
The second is data and intellectual property. Confidential material pasted into a public AI tool can pass out of the company’s control and cannot always be recalled. The third is regulatory exposure. A resulting personal-data breach must be reported to Sweden’s data protection authority, IMY, within 72 hours under the GDPR and NIS2 duties apply with the board held accountable.
The fourth is time and trust. Investigation, recovery and reputational repair are slow and expensive and the money is often gone for good. Arup’s funds were not recovered and the case remained under investigation. Against this, the World Economic Forum’s 2025 outlook found that 42 percent of organisations reported a rise in phishing and social engineering in 2024 while most still had no process to check the AI tools their staff were using.
AI Attacks on Real Organisations
Two cases show the two sides of the risk, an attack from outside and a leak from inside.
The Arup Deepfake Video Call
In January 2024 a finance employee in the Hong Kong office of Arup, the London engineering firm, received a message that appeared to come from the group’s UK-based CFO about a confidential transaction. The employee was doubtful at first.
A video call removed the doubt. On the call the CFO and several colleagues looked and sounded exactly as expected and the employee then made 15 transfers totalling about 25.6 million US dollars (200 million Hong Kong dollars) to five Hong Kong accounts. Every other person on the call was an AI-generated deepfake, built from public footage. Hong Kong police reported the case in February 2024 and Arup confirmed it in May 2024.
The control that would have stopped it is the same step that eventually exposed it. Before paying, the employee needed to verify the request through a separate, known channel such as a call back to a confirmed number at head office. Arup’s own CIO described the incident as technology-enhanced social engineering, not a systems breach.
The Samsung Data Leak
In March 2023 Samsung’s semiconductor division allowed engineers to use a public AI chatbot to help with their work. Within about twenty days there were three separate leaks.
One engineer pasted proprietary source code into the tool to fix a bug. Another pasted code from an internal test sequence used to spot defective chips. A third recorded a confidential meeting turned it into text and pasted the transcript in to get notes. None of the three meant any harm. All of them sent confidential data to servers the company did not control.
Samsung’s response was to ban public AI tools and build an internal one. The more durable lesson for most companies is different. Give staff an approved AI tool and a plain rule on what must never be entered, because a ban alone tends to push the same behaviour out of sight.
AI Awareness Training and Compliance
For organisations operating in Sweden and the EU, awareness training is no longer only good practice. Several rules make it a duty.
The EU AI Act sets the most direct requirement. Under Article 4, providers and deployers of AI systems must ensure a sufficient level of AI literacy among staff and anyone operating AI on their behalf. This has applied since 2 February 2025 and national authorities are expected to begin enforcement from 2 August 2026. The Act does not prescribe a set course, so a proportionate, role-based programme is the practical way to comply.
Sweden’s Cybersäkerhetslagen (SFS 2025:1506) transposed the EU NIS2 Directive and has been in force since 15 January 2026. NIS2 lists basic cyber hygiene and security awareness training among its required measures in Article 21 and Article 20 makes the management body responsible for security with personal accountability for board members. See our guide to NIS2 compliance in Sweden.
For financial entities, DORA has applied since 17 January 2025. Article 13(6) makes ICT security awareness programmes and resilience training compulsory for all employees and senior management, at a level matched to each role. In Sweden this is supervised by Finansinspektionen. See our guide to DORA compliance.
Two further rules touch this area. Under GDPR, Article 33, a personal-data breach must be reported to IMY within 72 hours which an AI-enabled attack or an AI data leak can trigger. And ISO 27001 treats staff awareness as part of a working information security management system.
How to Spot an AI-Driven Attack
Some signs still give an AI-driven attack away. Watch for a few patterns.
- Urgency and secrecy: A push to act fast and to keep the request confidential.
- A changed detail or channel: A new bank account, a personal messaging app or a request to skip the normal approval.
- Requests that bypass process: An instruction to move money, reset access or share data outside the usual steps.
- Video and audio artefacts: Odd lip-sync, a flat or still face, audio that does not match the mouth or a reluctance to take an unscripted action such as turning the head on camera.
- Perfect writing, abnormal ask: A fluent, well-targeted message that still asks for something unusual. AI has removed the old spelling and grammar tells.
Here is the hard part. You cannot reliably catch a good deepfake by eye and the quality improves every month. Research backs this up. A 2025 controlled study of more than 19,500 employees found that security awareness training as usually delivered, did not meaningfully reduce how often staff fell for simulated phishing. So the reliable move is to verify the request, not to trust your senses.
How to Build AI Security Awareness That Works
A programme that changes behaviour rests on three things working together, the people, the process and the technology. Treat training as one control among several rather than the whole defence.
Make training continuous and specific. The 2025 study found that a single annual course did little and that susceptibility grew over eight months so replace the yearly module with short, frequent and role-based sessions. Include finance, executives and their assistants who are the usual targets of deepfake fraud, and brief the board since NIS2 holds it accountable.
Set one clear rule and make it normal to follow. Verify any unusual or urgent request for money, credentials or data through a separate, known channel before acting. Require a second authoriser and a call back on a confirmed number for large payments and make reporting a suspicious message quick and free of blame.
Support the rule with controls. Turn on multi-factor authentication everywhere. Give staff a sanctioned AI tool with clear guidance on what must never be entered and add data-loss prevention so confidential material is caught before it leaves. Run realistic simulations that include AI voice and video and use each mistake as a lesson.
None of this depends on anyone spotting a perfect fake. It depends on a habit. When your people verify before they act, the most convincing AI attack still has to get past a phone call it cannot fake.
