While online harassment is recognized as a significant problem, most scholarship focuses on descriptions of harassment and its effects. We lack explanations of why people engage in online harassment beyond simple bias or dislike. This article puts forth an explanatory model where networked harassment on social media functions as a mechanism to enforce social order. Drawing from examples of networked harassment taken from qualitative interviews with people who have experienced harassment ( n = 28) and Trust & Safety workers at social platforms ( n = 9), the article builds on Brady, Crockett, and Bavel’s model of moral contagion to explore how moral outrage is used to justify networked harassment on social media. In morally motivated networked harassment, a member of a social network or online community accuses a target of violating their network’s norms, triggering moral outrage. Network members send harassing messages to the target, reinforcing their adherence to the norm and signaling network membership. Frequently, harassment results in the accused self-censoring and thus regulates speech on social media. Neither platforms nor legal regulations protect against this form of harassment. This model explains why people participate in networked harassment and suggests possible interventions to decrease its prevalence.
Digital Social Norm Enforcement: Online Firestorms in Social Media
Actors of public interest today have to fear the adverse impact that stems from social media platforms. Any controversial behavior may promptly trigger temporal, but potentially devastating storms of emotional and aggressive outrage, so called online firestorms. Popular targets of online firestorms are companies, politicians, celebrities, media, academics and many more. This article introduces social norm theory to understand online aggression in a social-political online setting, challenging the popular assumption that online anonymity is one of the principle factors that promotes aggression. We underpin this social norm view by analyzing a major social media platform concerned with public affairs over a period of three years entailing 532,197 comments on 1,612 online petitions. Results show that in the context of online firestorms, non-anonymous individuals are more aggressive compared to anonymous individuals. This effect is reinforced if selective incentives are present and if aggressors are intrinsically motivated.
Digital Social Norm Enforcement: Online Firestorms in Social Media
Actors of public interest today have to fear the adverse impact that stems from social media platforms. Any controversial behavior may promptly trigger temporal, but potentially devastating storms of emotional and aggressive outrage, so called online firestorms. Popular targets of online firestorms are companies, politicians, celebrities, media, academics and many more. This article introduces social norm theory to understand online aggression in a social-political online setting, challenging the popular assumption that online anonymity is one of the principle factors that promotes aggression. We underpin this social norm view by analyzing a major social media platform concerned with public affairs over a period of three years entailing 532,197 comments on 1,612 online petitions. Results show that in the context of online firestorms, non-anonymous individuals are more aggressive compared to anonymous individuals. This effect is reinforced if selective incentives are present and if aggressors are intrinsically motivated.
Harm Through the “#RizzCam”: Smart Glasses, Ambient Capture and Invisible Harassment
In online safety research, harassment is commonly understood as involving overt and unwanted acts that create a hostile environment for individuals or groups. Smart glasses, or wearable digital devices that are often indistinguishable from ordinary eyewear but can record video, audio and location data in real time, enable new forms of online harassment and gender-based harms that may not be recognised by current online safety or platform governance frameworks. Drawing on an interpretive content analysis of 350 point-of-view Instagram videos posted by individuals self-associated with the pick-up artistry (PUA) community, we examine how smart glasses can be used to harass women through what we call ambient capture. Ambient capture includes clandestine recording of people and social interactions within public, semi-private and private spaces. We argue this can constitute harassment when it is targeted at women and shared online without their consent. In our study, we found videos using ambient capture practices to harass women that had gone undetected by Instagram’s content moderation systems. These findings suggest online safety regimes and content moderation approaches should be assessed to ensure they account for the more invisible forms of harassment introduced by smart glasses and other wearable technologies.
Custodians of the Internet
A revealing and gripping investigation into how social media platforms police what we post online—and the large societal impact of these decisions Most use...
Trust: Foundation of Human Knowledge yet online Platforms won't allow | David Karger | TEDxMIT Salon
Estimating the effect size of moral contagion in online networks: A pre-registered replication and meta-analysis
Abstract. Over 5 billion people now use social media platforms. As our social lives become increasingly entangled with online social networks, it is import
Criticism of Facebook
Facebook (and parent company Meta Platforms) has been the subject of criticism and legal action since it was founded in 2004.[1] Criticisms include the outsize influence Facebook has on the lives and health of its users and employees, as well as Facebook's influence on the way media, specifically news, is reported and distributed. Notable issues include Internet privacy, such as use of a widespread "like" button on third-party websites tracking users,[2][3] possible indefinite records of user information,[4] automatic facial recognition software,[5][6] and its role in the workplace, including employer-employee account disclosure.[7] The use of Facebook can have negative psychological and physiological effects[8] that include feelings of sexual jealousy,[9][10] stress,[11][12] lack of attention,[13] and social media addiction that in some cases is comparable to drug addiction.[14][15]
Play nice! How the internet is trying to design out toxic behaviour
Online abuse can be cruel – but for some tech companies it is an existential threat. Can giants such as Facebook use behavioural psychology and persuasive design to tame the trolls?
Overperception of moral outrage in online social networks inflates beliefs about intergroup hostility
As individuals and political leaders increasingly interact in online social networks, it is important to understand the dynamics of emotion perception online. Here, we propose that social media users overperceive levels of moral outrage felt by individuals and groups, inflating beliefs about intergroup hostility. Using a Twitter field survey, we measured authors’ moral outrage in real time and compared authors’ reports to observers’ judgements of the authors’ moral outrage. We find that observers systematically overperceive moral outrage in authors, inferring more intense moral outrage experiences from messages than the authors of those messages actually reported. This effect was stronger in participants who spent more time on social media to learn about politics. Preregistered confirmatory behavioural experiments found that overperception of individuals’ moral outrage causes overperception of collective moral outrage and inflates beliefs about hostile communication norms, group affective polarization and ideological extremity. Together, these results highlight how individual-level overperceptions of online moral outrage produce collective overperceptions that have the potential to warp our social knowledge of moral and political attitudes.
Cleaning Up the Streets: Understanding Motivations, Mental Models, and Concerns of Users Flagging Social Media Content
Social media platforms offer flagging, a technical feature that empowers users to report inappropriate posts or bad actors to reduce online harm. The deceptively simple flagging interfaces on nearly all major social media platforms disguise complex underlying interactions among users, algorithms, and moderators. Through interviewing 25 social media users with prior flagging experience, most of whom belong to marginalized groups, we examine end-users’ understanding of flagging procedures, explore the factors that motivate them to flag, and surface their cognitive and privacy concerns. We found that a lack of procedural transparency in flagging mechanisms creates gaps in users’ mental models, yet they strongly believe that platforms must provide flagging options. Our findings highlight how flags raise critical questions about distributing labor and responsibility between platforms and users for addressing online harm. We recommend innovations in the flagging design space that enhance user comprehension, ensure privacy, and reduce cognitive burdens.
Beyond the hashtag: Circumventing content moderation on social media
Social media companies make important decisions about what counts as “problematic” content and how they will remove it. Some choose to moderate hashtags, blocking the results for certain tag searches and issuing public service announcements (PSAs) when users search for troubling terms. The hashtag has thus become an indicator of where problematic content can be found, but this has produced limited understandings of how such content actually circulates. Using pro-eating disorder (pro-ED) communities as a case study, this article explores the practices of circumventing hashtag moderation in online pro-ED communities. It shows how (1) untagged pro-ED content can be found without using the hashtag as a search mechanism; (2) users are evading hashtag and other forms of platform policing, devising signals to identify themselves as “pro-ED”; and (3) platforms’ recommendation systems recirculate pro-ED content, revealing the limitations of hashtag logics in social media content moderation.
Data Feminism for AI
Attacks on women and trans people are not happening in a vacuum. Communications scholars W. Lance Bennett and Marianne Kneuer recently argued that we are witnessing the intensification of "illiberal public spheres" around the world (Bennett and Kneuer, 2023). These are characterized by a set of anti-democratic communications tactics that include the denigration and exclusion of minoritized people, targeted attacks on the press and political institutions, and transgression of norms of civil discourse. Such tactics are enabled and amplified by the social media platforms owned by Big Tech, whose business model elevates attention above all other metrics, such that they profit richly off extremism, threats, spectacle, and lies. The success of this corporate partnership with right-wing extremism is evident in the proliferation of book bans (Magnusson, 2024), the demonization of DEI and critical race theory (Crenshaw, 2022), and mob tactics to doxx, intimidate, misrepresent, and otherwise silence people that study misinformation, speak out against racial and gender violence, or rule against insurrectionists (Edmonds, 2024). At the same time, we have seen the continued work of feminists, including Latin American, Indigenous, and abolitionist feminists in the US and globally, doing the work of imagining alternate worlds (Cullors, 2021; Davis, 2022; Ricaurte, 2019; Simpson, 2017). This paper represents our own first attempt to imagine a more equitable, sustainable, feminist world for AI research.
Frontiers | Cyber-mobbing, technology-facilitated abuse and online violence
This article collection will explore the pressing issues of cyber-mobbing, technology-facilitated abuse and online violence, including online abuse, stalking...
Online abuse: banning anonymous social media accounts is not the answer
In the wake of the tragic death of Sir David Amess, some politicians are calling for a law banning anonymous social media profiles. But it’s not the right way to tackle online abuse.
Online abuse: banning anonymous social media accounts is not the answer
In the wake of the tragic death of Sir David Amess, some politicians are calling for a law banning anonymous social media profiles. But it’s not the right way to tackle online abuse.
But anyway, papers since you asked: - DOI:10.1371 Digital Social Norm Enforcement: Online Firestorms in Social Media - Deliberation and Identity Rules: The Effect of Anonymity, Pseudonyms and Real-Name Requirements on the Cognitive Complexity of Online News Comments doi.org/10.1177/0032321719891385
