The Future of Eating Disorder Content Moderation

This guest post is the second in a series of writings on the state of content moderation from ATM participants and colleagues engaged in examining content moderation from a variety of perspectives. We welcome Dr. Ysabel Gerrard in the post below.


In recent months, there have been various public furores over social media platforms’ failures to successfully catch — to moderate — problematic posts before they are uploaded. Examples include two YouTube controversies: vlogger Logan Paul’s video of a suicide victim’s corpse which he found hanging in a Japanese forest, and other disturbing videos found on the YouTube Kids app, featuring characters from well-known animations and depicting them in upsetting and frightening scenarios. By and large, it’s easy to see why these cases hit the headlines and why members of the public called on platforms to rethink their methods of moderation, however high their expectations may be. But it’s harder to straightforwardly categorise other kinds of content as ‘bad’ and decide that they don’t have a place on social media, like pro-eating disorder (pro-ED) communities.

In early 2012, Tumblr announced its decision to moderate what it called ‘blogs that glorify or promote anorexia’. Its intervention came five days after a widely-circulated Huffington Post exposé about the ‘secret world’ of Tumblr’s thinspiration blogs. Facing mounting public pressures about their roles in hosting pro-ED content, Instagram and Pinterest introduced similar policies. All three platforms currently issue public service announcements (PSAs) when users search for terms related to eating disorders, like #anorexia and #thinspo, and Instagram blocks the results of certain tag searches. They also remove users and posts found to be violating their rules.

“Important questions remain unanswered.”

Almost six years later, important questions remain unanswered: why did platforms decide to intervene in the first place, given their hesitancy to remove other kinds of content despite public pressures? How often do moderators get it wrong? And, perhaps most crucially, what are the consequences of platforms’ interventions for users? I can’t answer all of these questions here, but I will address some of the issues with platforms’ current moderation techniques and propose some future directions for both social media companies and researchers.

The politics of the tag.

Platforms’ decisions to moderate hashtags implies that people use certain tags to ‘promote’ eating disorders. Hashtags tell moderators and users what a post is about, and moderated tags currently include ‘#proana’ and ‘#proed’, but they also include more generic phrases that aren’t explicitly pro-ED, like ‘#anorexia’ and ‘#bulimia’. A blanket ban on all eating disorder-related tags — ‘pro’ or otherwise — implies a disconnect between what platforms think people are doing when they use a certain tag and what they are actually doing and why they are doing it. In a forthcoming paper, I show how Instagram and Tumblr users continue to use ED-related tags despite their awareness of content moderation and of the volatility of ED hashtags. Without speaking to users (an ethically tricky but nonetheless important future research direction), it is difficult to know why they continue to use tags they know are banned. Perhaps their actions are not intended to ‘trigger’ or encourage other users — if indeed people can ‘catch’ an eating disorder solely by looking images of thin women — but to find like-minded people, as so many other social media users do. After all, this is the ethos on which social media companies were built.

For example, one anonymised Tumblr user addressed one of her posts to users who report ‘thinspo’ blogs. She called these blogs a ‘safe place’ for people with eating disorders, warning them that reporting posts can be triggering and assuring them that she will always create replacement blogs. Her post was re-blogged over 10,000 times and seemed to echo the sentiment of other Tumblr users, ‘pro-ED’ or otherwise.

Tumblr said it would not remove blogs that are ‘pro-recovery’, but this user’s feed — along with many others’ — is a tangled web of pro-eating disorder, pro-recovery and other positionalities. Users do not always conform to a stereotypically and recognisably ‘pro’ eating disorder identity, if indeed platforms (and anyone, for that matter) would know how to recognise it when they saw it. If social media offer ‘safe’ spaces to people whose conditions are socially stigmatised and marginalised, then platforms need to exercise greater care to understand the content of social media posts. But as I explain below, this might not be feasible for platforms, whose moderation workforces are already pushed to their absolute limits.

Why blanket rules don’t work.

The fact is that blanket rules for content moderation do not work, and nor should we expect them to. They always and inevitably miss the subtleties; the users who sit somewhere in the murky middle-ground between acceptability, distaste and often illegality. In social media’s eating disorder communities, various discourses — pro-ED, pro-recovery, not-pro-anything, amongst many others — are entangled with each other. But while platforms took measures to address eating disorders, they evidently did not adjust their own moderation mechanisms to suit such a complex issue. As Roberts explains, Commercial Content Moderators (CCMs) have only a few seconds to make a decision about moderation. It is concerning that individual posts can be de-contextualised from a user’s full feed — through no fault of a CCM’s, given the speed with which they must make decisions — and be removed.

For example, in its Community Guidelines, Pinterest includes an example of an image of a female body that would be ‘acceptable’ in a moderator’s eyes. The image’s overlaid text: ‘It’s not a diet, it’s a way of life. FIT Meals’ de-situates it from pro-eating disorder discourses:

Picture1

(Image source: https://www.pinterest.co.uk/pinterestpolicy/self-injury-examples/).

“Perhaps we are expecting too much of moderators, but not enough of platforms.”

But in the absence of hashtags and a text overlay, would a content moderator know not to categorise this as ‘pro-ED’ and ban it? How do they know how to interpret the signals that ED communities have become infamous for? How can a moderator do all of this in only a few seconds? And what happens to users if moderators get it wrong, something Facebook recently admitted to in relation to hate speech posts? Perhaps we are expecting too much of moderators, but not enough of platforms.

The future of eating disorder content moderation.

If eating disorder content continues to be moderated, are the current approaches appropriate? Perhaps social media companies should not encourage users to ‘flag’ each other in their public-facing policies, given the historical and problematic surveillance of girls’ and women’s bodies. Maybe Instagram should not continue to chase hashtags and ban the ones the ones emerge in their place, given the minimal space they occupy at the margins of social networks. Platforms could also provide full and exhaustive lists of banned tags to help users navigate norms, vocabularies and cultures. In the follow-up to its original policy, Tumblr admitted that it was ‘not under the illusion that it will be easy to draw the line between blogs that are intended to trigger self-harm and those that support sufferers and build community’. This was admirable, but what if Tumblr became more transparent about how its moderators make these decisions?

Moving forward, one suggestion for social media companies to foster collaborations with social scientists to research and better understand these cultures (a point made recently by Mary Gray). Those with an understanding of eating disorders know that their stigmatisation has historically prevented people from seeking treatment; a problematic and gendered issue that platforms’ panicked decisions to intervene have potentially worsened. An in-depth exploration of online eating disorder communities and a more open dialogue between researchers and platform policy-makers might help social media companies to promote an alternative view of eating disorders than simply being ‘bad’.


Ysabel-Gerrard-smallDr. Ysabel Gerrard is a Lecturer in Digital Media and Society at the University of Sheffield. She co-organises the Data Power Conference and is the current Young Scholars’ Representative for ECREA’s Digital Culture and Communication Section. One of the aims of her next project is to talk to social media workers who have been/are involved in making decisions about pro-eating disorder and self-harm content moderation. If you are, or know of anyone who might want to share their experiences, please email her at: y.gerrard@sheffield.ac.uk.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s