Overview
A technically secure application can still be unsafe for the people using it. Messaging, content sharing, reviews, livestreaming, groups and other legitimate features can be repurposed for harassment, grooming, impersonation, coercion, scams and other user-to-user harms.
CIOs responsible for applications with meaningful user interaction should require a User-Harm Threat Model before launch and whenever interaction functionality materially changes. The objective is not to moderate everything or eliminate all risk. It is to identify foreseeable misuse, apply proportionate controls, assign accountability for what remains, and make an explicit release decision.
What Is Happening
Application security has traditionally concentrated on protecting systems, accounts and data from attackers. That remains essential, but it does not cover situations where a legitimate account uses legitimate application functionality against another legitimate user.
Regulators are increasingly approaching this problem through design and risk assessment rather than relying solely on post-incident moderation. Australia's eSafety …