Security reports from other sites are welcome. Why aren’t safety reports?

One of the most fundamental triumphs of the last few decades of open source culture is the open sharing of bugs and how to fix them. There are mature, effective systems for responsibly reporting things like security issues, and entire ecosystems of players that coordinate on getting the issues resolved, even when doing so requires non-public coordination between many of the biggest and most powerful organizations in the world.

It's a quiet miracle of unexpected collaboration that literally enables the Internet as we know it to keep running.

There are a lot of lessons we could learn from this example, but what's most striking is where we haven't learned the lessons of this powerful form of collaboration: handling safety online.

Ignoring the symptoms

Imagine if a bank got a security report saying "there's a severe known security bug in some software that you're using, and we've fixed it over at our bank — wanna hear what we learned?" and then they responded with "No, we'll wait until the security issue has been used to steal from our bank."

This is the current state of the art when it comes to safety threats on social and communications platforms online. The vast majority of large-scale social platforms have an explicit policy of ignoring the harms or destructive actions that someone commits on any platform other than their own. When people have deliberately targeted others for abuse, spread harmful propaganda, or even bilked people out of money or opportunities, it's very common for a company to say, "That happened on another platform, and we only judge users by what happens on our own platform."

That's a mistake, and it's one that is frequently exploited by some of the worst actors on the Internet. Many times, those who intentionally want to cause harm to others will coordinate across platforms, narrowly walking within the lines of the worst things each platform permits, and spreading out the ill intentions in a way that gives them plausible deniability on each individual platform, while collectively enabling truly awful outcomes for their targets.

This is an absurd state of affairs, and one that empowers victimizers instead of good users. That is not to say that it's trivially easy to change from an "ignore the rest of the world" policy, but that it's worthwhile to attempt to do so, for the good of a platform's users and community, but also for the good of the Internet as a whole.

How to make a world-aware policy

When considering issues of community health, trust or safety, it's almost always vital to start with a framework that considers context and power when making decisions and policy. A simple example here is if we consider situations that involve publishing the IP address for a person. In a framework that's informed by the context of an individual, and the relative power that they hold in that context, we can easily see that revealing the IP address of someone who is fleeing an abuser is reducing the power they have over their own safety, because it could be used to help deduce their location. By contrast, if someone is publishing the IP address of another user in order to reveal that they're creating content from a computer located in the House of Representatives, that's a context where the action is being carried out by a person who bears a lot of power that may need to be held to account.

What this doesn't do is give a trust and safety team some easy checklist to follow, that simply says "this action is always bad". (Though, to be clear, there are some actions that are always bad.) Instead, it provides a framework for quickly and effectively making decisions that protect users and the community, in ways that are predictable enough to be trusted by the community.

Once we think about creating such a framework, we realize that this is the kind of model where the more useful data we have to help form our point of view, the better a decision-making process we can create. If we understand our challenge to be how to rapidly gain sufficient context with which to make an informed decision, then it becomes obvious we should consider a user's behavior off-platform.

Putting this kind of framework into practice can yield some unexpected benefits. For example, if we can see that someone is usually reasonable and thoughtful on other platforms, but has been transgressing within the context of our community, we can assume that they're either having a bad day or were prompted by something out of the ordinary in their experience that pushed them into not being their best selves. Many times, simply nudging them with a reminder of how much more constructive they are elsewhere can be more than enough to bring them back into being more positive community members. I've personally had a surprisingly large number of people respond with an apology and a near-instant change in attitude when given this kind of prompt.

There are, of course, the less heart-warming examples. Sometimes you see someone transgressing in a community, take a look at what they're doing elsewhere, and are completely horrified at what you find. In those contexts, it's easier to make a proactive decision to ban (or limit the privileges of) a user, because you can have higher confidence that they're not simply having a bad day.

It's hard. It's supposed to be.

The thing about building a community management process that considers the rest of the world to be valuable input is that, well... it's hard. It can be difficult to ascertain that a user with a certain identity on your platform is exactly the same person on another platform. (This is especially tricky if the bad action they're doing on one of the platforms is trying to impersonate someone else.) Sometimes different people have the same name, or avatar, or location, or all of the above. If the information you find on other sites is older or possibly out of date, you have to consider whether it's fair to judge the user by that context; people do (thankfully!) grow up and mature, or get out of the circumstances that caused them to be a jerk online. You can't always just write someone off for having some old account where they were breaking the rules.

You'll also find things you don't understand. Jokes or memes that may have been okay to the people who were in a particular context, even if they don't seem acceptable outside of it. Things that have such a specific context that they're impossible to parse for outsiders. Pages that are broken, images that are missing, clues that are gone. Since you'll be limited for time and resources, you very often won't have perfect information for making your decisions. Sometimes you'll have to dismiss the outside info entirely because you can't make heads or tails of it.

But more often than not, you'll find enough context to see the warts-and-all humanity of the person who's now part of your community. That's especially valuable because most of the time when you're doing all this work, it's because they've done something unpleasant enough that it prompted all this effort. The natural bias would be to not cut them a break.

And that gets to the hardest, and most rewarding, part of this approach: bringing them back in. The goal of building a community policy that considers a person's broader presence online is not just so you can ban people more quickly. (Although it does enable that for people who are clearly just being awful!) Instead, you'll very often find community members who are misguided but redeemable. Maybe they're used to other forums and communities online where people act like jerks to each other, and they don't know any better. As I said before, maybe they're just having a bad day. Maybe it's even because of the same shortcomings that drive you nuts about your own community!

What we end up with is a process where, most of the time, people can be brought back into the fold. Folks who started out as annoyances or even real problems in a community can turn into productive community members, and sometimes even community leaders. Others around them in the community can see that the inclination of the community is not punitive, but constructive. (This is, yes, an online community management framework informed by many of the principles of restorative justice.) Done right, it's good for the community, but it's also great for the trust and safety team, because their job shifts from mostly being about policing bad behavior into encouraging and even rewarding good behavior. Because the whole community can see it, you can often catalyze a positive feedback loop that builds more good behavior over time.

We need to do it

The current default model of waiting until someone is repeatedly, irredeemably awful, and then banning them so they get passed off to whatever website is still allowing people to be jerks to each other simply isn't working. Almost a dozen years ago I said "If your website is full of assholes, it's your fault", and it's still as true today as it was then.

This is an approach that, plain and simple, requires more investment in people and resources than the default model. This costs more money. It is worth it, because it yields a sustainably better result. But it is an investment, and should be treated as such. It will also likely be new and unfamiliar to your lawyers and policy people; they should be excited about the chance to learn the newest, most modern way to manage a community online, but if they're not, have them reach out to me and together we'll teach them how to do better. (Someone will come up with a catchy name for all of this, and it'll be a super marketable skill!)

We've seen for far too long the deep and painful costs of having unhealthy, destructive online communities where bad actors can hop from platform to platform doing their worst. We also see lots of communities where an ordinary person can have a bad day, make a mistake in an interaction, and then get no corrective feedback, and so fall into a cycle where they act worse and worse. It doesn't have to be this way.

In outlining this vision, I'm sharing broad principles from a few decades of striving to run communities this way, but of course the devil is in the details and there are a lot of specifics that I can't get into in this kind of piece. What I'd suggest above all is something simple:

Community safety policy should be as informed by the ecosystem as technical security policy. Learn from the ecosystem. Pay attention to the rest of the world.