Technology has transformed the way we live, work and communicate, making organisations more efficient, connected and capable than ever before. Businesses, charities, schools, sports organisations and public bodies have all embraced digital platforms, software systems, artificial intelligence and connected devices because they promise faster decisions, improved communication and better outcomes. However, hidden beneath this rapid adoption is a growing risk that many organisations may not have fully considered.
Since becoming more involved in online safeguarding, one of the things I have found is that many people do not recognise the dangers they have let into their own life and home. Perhaps some of this is because we have started to confuse purchasing technology with a robust safety protocol included, much like we expect when buying a kettle, a car or any regulated consumer product. The dangerous assumption is that because a platform, software system or device has been created by a technology company, that the risks have already been identified, managed and removed before it was launched. The assumption is understandable because if a platform is created by a global technology company, with a target market of millions and supported by impressive security statements, it feels reasonable to believe that the risks have already been addressed, but the truth is that it’s more a case of “caveat emptor”.
In reality, technology is not just a product, it is an environment where human behaviour, vulnerability and unintended consequences can create entirely new forms of harm. In some cases, as I have seen documented by both senators and congressmen/women in the USA when questioning the CEOs of major social media platforms, the dangers have been identified and highlighted by consultants and their employees, yet ignored…
A product can be technically excellent and still create harm, a platform can work exactly as designed and still expose people to abuse, exploitation, harassment or psychological damage, a device can perform perfectly and still contribute to unhealthy behaviours, anxiety or stress and enable stalking or worse. Software can be secure from cyber threats while completely failing to consider the impact it has on the people who interact with it and that’s not just carefully hearding you down a one way tunnel where apps start to malfunction if they aren’t part of the “inner sanctum” – but I’ll shelve that ramble today.
This is why I believe that the concept of “safety by design” needs to move much higher up the agenda because it’s not simply about creating technology that functions correctly, but also recognising that every piece of technology exists within a human environment, and human environments are complicated. They involve emotions, vulnerabilities, mistakes, poor decisions and sometimes deliberate attempts to cause harm. The question organisations need to ask is no longer simply: “does this technology do what we purchased it to do?” The more important question is: “what could happen to people if this technology is misused, misunderstood or exploited?” This distinction matters because many organisations are currently approaching technology risk backwards; they identify a need, purchase a solution and then attempt to manage the consequences afterwards only if harm occurs. I can see the “logic” of “if it’s not broken, no need to fix it”, but what damage to your company when reports that one of your employees has been using the office smart glasses to create adult videos of one of their colleagues?
A school introduces a communication platform because it wants better engagement with parents and pupils.
A sports organisation sets up an online community because it wants to connect supporters and athletes.
A company introduces monitoring software because it wants to improve productivity.
A public body implements a digital service because it wants to deliver services more efficiently.
All of these decisions may be reasonable, the risk(s) appears when nobody asks what happens when the same tools are used in ways nobody intended.
A communication platform designed to bring people together can also become a vehicle for harassment.
A social platform designed to encourage participation can also become an environment where abuse spreads quickly, pictures are stolen, fake accounts are set up, etc.
An algorithm designed to increase engagement can prioritise attention over wellbeing.
A workplace system designed to improve efficiency can create a culture where employees feel constantly monitored and under pressure.
A set of smart glasses can be used to create adult videos of a colleague without their consent.
The technology has not necessarily failed because the real failure occured when the human impact was never considered or deliberately hidden by the originators.
For too long, organisations have relied heavily on compliance as a measure of safety. Policies are written, procedures are documented, training is delivered and audits are completed, and while all of these things are important, they can create a dangerous illusion that risk has been controlled simply because the paperwork exists. A safeguarding policy, however comprehensive, does not automatically make a digital environment safe, just as a procurement process does not guarantee that every foreseeable risk has been identified. Likewise, a contract with a technology supplier does not remove responsibility from the organisation that chooses to implement and rely on that technology. The reality is that responsibility follows the decision, and organisations remain accountable for the choices they make, the systems they introduce and the risks they accept.
Character.AI is perhaps one of the clearest examples of why the phrase “safety by design” has to become more than another corporate phrase used in presentations, policies and compliance documents. The principle behind safety by design is simple: when developing technology, particularly technology that interacts with people emotionally or collects personal information, the risks should be considered at the beginning of the process rather than addressed only after something goes wrong. The concept behind AI companion technology was understandably attractive. The idea that people could create digital characters, hold conversations and interact with something that appeared responsive, available and understanding represented an exciting development in artificial intelligence. For some users, particularly young people who experienced loneliness, isolation or simply wanted somewhere to express their thoughts without fear of judgement, these systems appeared to offer companionship and connection.
However, the difficulty is that artificial intelligence can simulate human interaction without actually understanding human emotion. It can generate words that appear caring and supportive, but it does not possess empathy, judgement or the safeguarding instincts that exist within a genuine human relationship. It can create the impression of friendship and emotional connection while lacking the ability to truly understand the vulnerability, circumstances or mental state of the person on the other side of the screen, and that distinction becomes critically important when the users involved are children, because young people may not always have the experience or emotional maturity to recognise the difference between a genuine relationship and an interaction generated by a machine.
Character.AI was developed by former Google AI engineers and quickly became one of the most talked-about examples of conversational artificial intelligence. The technology represented the excitement surrounding a future where machines could interact with people in increasingly natural and human-like ways, creating conversations that felt personal and engaging. However, as adoption grew, so did questions about whether sufficient thought had been given to the risks associated with creating systems specifically designed to encourage ongoing interaction and emotional engagement. Zero thought to the effects on a human body which has evolved to need social interaction, not isolation…
The lawsuits brought against Character.AI have forced those questions into the public conversation. Families have alleged that interactions with AI chatbots contributed to serious harm involving teenagers, including cases involving self-harm and suicide. One of the most widely reported cases involved Megan Garcia, a lawyer who was a speaker at a conference I attended in early 2025 who told us about her 14-year-old son, Sewell Setzer III, who died by suicide after extensive interaction with a Character.AI chatbot. The lawsuit alleged that the chatbot encouraged an unhealthy emotional attachment and failed to respond appropriately to signs of distress. Regardless of the eventual legal outcomes, the wider question extends far beyond one company or one product. It concerns how society approaches responsibility when developing powerful technologies that can influence human behaviour, particularly among vulnerable users who may not fully understand the limitations of the system they are interacting with.
This is exactly where the concept of safety by design becomes so important. No pharmaceutical company can release any medical drug without undergoing a series of complex safety and quality compliance; we would never accept a vehicle manufacturer launching a car and deciding that safety features could be added after accidents had already occurred. We expect risks to be identified, assessed and managed before a product reaches the public because we understand that innovation and responsibility must exist together. Yet with digital technology, particularly emerging artificial intelligence systems, we continue to see situations where products are developed and released at extraordinary speed, while safety considerations appear to follow afterwards. The technology evolves faster than the safeguards, and the consequences of that approach are often experienced by the very people who were supposed to benefit from the innovation.
I am not trying to imply that artificial intelligence should be stopped or that every new technology should be viewed with suspicion – innovation has always carried risks, and society has always had to adapt to powerful new tools. The issue is whether those developing these technologies are taking responsibility for foreseeable risks before those risks become real-world harm. An AI system may not have intentions, but the people who design, train and deploy that system do have responsibilities. If a product is capable of creating emotional dependency, influencing behaviour or becoming a substitute for human support, then those possibilities must be considered before millions of people begin using it, rather than after families are left trying to understand why those risks were not addressed earlier.
The lesson from Character.AI is not that artificial intelligence itself is dangerous. The lesson is that powerful technology without appropriate safeguards can create consequences that were not fully considered or ignored at the point of design. The future of technology cannot be built on the assumption that society will simply learn from harm after it has already happened. Safety cannot be something added once a crisis occurs. It has to be part of the decision-making process from the very beginning, because when technology is designed to feel human, the responsibility behind that technology must be human as well.
This should be a major consideration for the insurance industry because the consequences of us all failing to be supplied with “safety by design” principles will eventually become financial consequences which will impact their profits and result in higher premiums for us all. Insurance has historically operated around responding after something has happened. An incident occurs, a claim is made, liability is assessed and compensation is considered. But technology-related harm creates a very different challenge because digital risks can scale rapidly. A poorly designed platform, an unsafe digital environment or a failure to consider foreseeable misuse can create widespread harm across communities, workplaces and public services. The next generation of claims may not simply involve physical damage or data breaches. They may involve psychological harm, online abuse, harassment, emotional distress, reputational damage and failures in safeguarding responsibilities and let’s not forget the claims made by SatNav users when the software had made them drive into a deep pond in the dark… The question insurers should be asking is whether their current risk models are prepared for this.
Are organisations being assessed on whether they have simply purchased recognised technology, or whether they have genuinely considered the human consequences? Are insurers looking at whether a business has implemented technology responsibly, or whether they are simply relying on the reputation of the supplier? Are companies being encouraged to demonstrate that safety was designed into their systems rather than added after something went wrong? The impact will not only be felt through insurance claims.
The NHS and wider public services are likely to carry a significant part of the cost because the connection between digital experiences and physical health is becoming increasingly clear. Online abuse, harassment, intimidation and harmful digital environments can contribute to anxiety, stress, depression and long-term mental health challenges. Those experiences do not remain online because they affect people’s ability to work, learn, participate and live healthy lives, create demand for GP appointments, counselling services, mental health support and specialist care. Our police are increasingly becoming involved in reports that have originated from online conduct. When organisations fail to design safe systems, the cost does not disappear, it just moves somewhere else: insurance claims, legal disputes, employee absence, public healthcare, etc…
This is why I hope that safety by design should not be viewed as solely a technology issue alone, it surely involves risk management safeguarding and insurance issues amongst others. The organisations that understand this now will be better prepared for the future by asking harder questions before purchasing technology. They will hopefully look beyond technical performance and consider human consequences.
Those that do not may find themselves answering difficult questions after harm has already occurred.
Technology will continue to evolve. Artificial intelligence, connected devices, automated systems and digital platforms will become even more embedded in everyday life.
The answer is not to reject innovation, but to make it safer and perhaps subject to certain QA testing before it can be marketed. Safe by design means recognising one simple truth: every technological decision is also a human decision; organisations that understand that principle will not only help to reduce future risk, but build a digital world where technology works for people rather than leaving people to deal with the consequences.
And knee jerks by government ? Let’s hope not – imagine all sat navs systems being disabled for one …
The warning signs are already visible – the question is whether insurers, organisations and public bodies act now, or wait until the claims arrive before recognising what was predictable all along.
FAQs
What does “safe by design” mean?
Safe by design means creating, selecting and implementing technology with safety built into the process from the beginning rather than trying to fix problems after harm has occurred. It considers not only whether a system works technically, but whether it protects the people who use it.
Why is buying technology not the same as buying safety?
Purchasing a recognised platform, software system or device does not remove responsibility from the organisation using it. Technology can operate exactly as intended while still creating opportunities for misuse, abuse or unintended harm.
Why should insurers be concerned about safe by design?
Insurers need to consider that future claims may increasingly involve digital harm, psychological injury, safeguarding failures, online abuse and the wider consequences of poorly implemented technology. The scale of digital platforms means one decision can potentially affect thousands of people.
Could unsafe technology create pressure on the NHS?
Yes. Digital harm does not stay online. Experiences such as harassment, intimidation, online abuse and harmful digital environments can contribute to stress, anxiety and mental health challenges, creating increased demand for healthcare and support services.
Is safe by design only a technology issue?
No. Safe by design is a risk management issue that involves technology companies, organisations, insurers, regulators, public bodies and anyone responsible for introducing digital systems into people’s lives.
What questions should organisations ask before adopting new technology?
Organisations should ask whether the technology has been assessed for foreseeable misuse, whether vulnerable users have been considered, whether safeguarding risks have been identified and whether safety has been built into the system rather than added after problems appear.
How can insurers help organisations reduce future risk?
Insurers can encourage better risk assessment by looking beyond whether an organisation has purchased established technology and examining whether it has considered human impact, safeguarding responsibilities and responsible implementation.
Why is safe by design becoming more important now?
Because technology is becoming more embedded in everyday life. Artificial intelligence, connected devices, automated systems and online platforms are increasing the potential benefits, but they also increase the scale of possible harm when safety is not considered from the start.
References
Information Commissioner’s Office (ICO) — Data Protection by Design and Default
National Institute of Standards and Technology (NIST) — Cybersecurity Risk Management Framework
UK Online Safety Act guidance
World Health Organization — Digital technologies and mental health considerations
