Examining Toxicity in Software Related Discussions

Earlier this week, The Register carried an interesting analysis of a study by CMU into online toxicity and, in particular, how it manifests in open-source projects.

The paper doesn't actually suggest that this is an Open Source Specific problem, just that the traits they identified are more common in OSS communities than other forms of toxicity are.

This seems to fit well with the idea that this is an issue around software communities, Open Source or otherwise.

As a timely example, the lead on the (forthcoming) game Return to Monkey Island has announced that he won't post on his blog about Return To Monkey Island anymore, specifially because of the abuse he's receiving.

The abuse is being sent because RTMI uses a different style of art-work to the original Monkey Island games.

That's right, because the artwork is different to the original 30 year old series, the lead on a game has been abused until he lost the will to share news/previews. It's an utterly shitty thing to do.

It seem's unlikely that Ron will read this, but if you are: mate, the game looks amazing and I'm so stoked that it's going to be a thing.

It's been a while since I've written a proper opinion piece, but in this post I want to analyse a few examples of toxicity (or potential toxicity) in the context of CMU's definition.

---

The Definition

One of the key findings in CMU's study was that toxicity tends to manifest in a different way in OSS projects than it does elsewhere.

The study looks at the form that toxicity takes in other locations (there's a tabulated example in the paper covering Reddit, Wikipedia, Twitter, Stack Overflow and Github).

The base definition of toxicity used is

rude, disrespectful, or unreasonable language that is likely to make someone leave a discussion.

When releasing the study, the lead author - Courtney Miller - said that

Toxicity is different in open-source communities, it is more contextual, entitled, subtle and passive-aggressive.

The news release notes that only about 50% of the toxic posts they examined contained obscenities.

So, we're not necessarily looking for strong language (and strong language doesn't imply toxicity), we're looking for communication that's unreasonably entitled, passive-aggressive or just downright unreasonable.

---

Case 1: Elementary OS

CMU give a specific example in the form of Elementary OS #348, so that seems as good a place to start as any.

The opening sentence of that issue is provided as a good example of an entitled and demanding approach

The ticket is combatative from the very outset, and whilst you can see what the user is complaining about, that opening sentence really does set an unconstructive tone.

This is definitely open to debate though, and there was some interesting discussion on El Reg about whether or not it actually meets the definition when viewed in a wider context.

However, the same author takes things up a notch when he's not agreed with

Within that issue there are other (minimised) comments which undoubtedly meet the more traditional definitions of toxicity.

The issue was ultimately (and rightly, IMO) locked and closed because of Code of Conduct violations (we'll come back to those later).

---

Case 2: simh Toxicity creates Toxicity

This year's developments around the simh project is (IMO) quite an interesting example of how toxic behaviour from users can elicit toxic behaviour from developers.

In July 2021, a slightly grumpy user opened issue #1059

Their frustration is evident in their tone, but this opening message ticks almost all the boxes of CMU's most common form of OSS toxicity:

• It's contextual: it relates to specific functionality
• It's entitled: it demends a fix, you can almost hear the tone of voice in "Please!"
• It's passive-aggressive: there's a strong implication that the feature, and whoever built it is stupid

The one thing it isn't, is subtle.

The opening message doesn't disclose it, but the developer (Mark Pizzolato) quickly stumbles upon the fact that the user has a somewhat unusual use-case.

The XKCD 1172 is incredibly string with this one.

It's quite hard, at this point for sympathies not to rest with the dev: the feature was introduced 15 months before the issue was raised, and now they've got complaints from a user who seems to object to using a filesystem on their removeable media.

Worse yet, the user knows that they can work around it, and appears to instead be complaining on principle:

I know that I can use the transfer size with dd but it was never necessary,
because the containers were correctly [under]sized. Like I said, your 
changes are not backward compatible

The further down the thread you go, the worse things seem to get.

Additional users start chiming in to disagree with the dev, who continues to respond and defend his implementation throughout the onslaught.

If we fast-forward to April this year (9 months after the ticket was first opened) there's a post to note that one of the users had been using Twitter to try and encourage people to brigade the ticket

At this point the Dev's responses are also getting a little short/blunt

Thanks for these wonderful suggestions which you are welcome to use on any 
simulator you work on.

Meanwhile, the design for the subject of this issue is long done and about 
to see the light of day.

I think a fair number of us would probably be striking this tone having been harangued for 9 months.

But (with the benefit of hindsight), this was also an early warning that the dev's attitude toward (at least part of) the userbase was turning toxic, with worse yet to come.

On 12 May 2022 a [commit was made]((https://github.com/simh/simh/commit/ce2adce632e1a22e6d76d4bf726d6b863373c550) to the repo which updated the license so that it contained the following text

Any use of this codebase that changes the code to influence the behavior of 
the disk access activities is free to do that as long as anyone doing this 
is explicitly not licensed to any subsequent changes to any part of the
codebase made by Mark Pizzolato after that functionality was implemented by 
Mark Pizzolato.  Changes that qualify for this restriction at least include: 
changing the behavior or default of SET AUTOSIZE/NOAUTOSIZE, or any code 
in sim_disk.c or any simulator components that use the sim_disk routines.

(the wording was subsequently amended again to mention two files - sim_disk.c and scp.c).

It's fairly obvious what motivated this change, but it also meant that the project's license was (by the dev's own admission) no longer FLOSS compatible because imposed proprietary terms on two files within the codebase.

On 3 Jun 2022, the simh project was forked and the Open SIMH project was announced. The community seems to have moved to the fork (at least based on pull-request and issue tracker activity).

It seems reasonable that we should be able to subsitute "language" for "actions" in CMUs definition (otherwise it wouldn't, for example, apply to toxic behaviour at in-person events), and with that minor change, CMU's definition of toxicity seems to fit very well

unreasonable language/action that is likely to make someone leave a discussion

In just one commit, the developer pulled the FLOSS rug out from under the project's entire userbase and drove away more or less the entire community.

It's a sad set of events, but what's interesting here is that at it's core, it's a similar phenomena to that observed with Ron Gilbert this week.

Before he changed the license, Mark Pizzolato endured nearly a year of abuse and harrassment from users who disagreed with one of his design decisions. Ron, of course, has had to endure abuse about his team's design choices too.

Although they've taken very different approaches, both Ron and Mark decided that they didn't want to interact with an abusive community on that topic any more, and in effect said "I'm taking my toys and I'm going home".

The difference, of course, is that Mark's change alienated the entire user-base, where Ron's doesn't affect use by those who don't object to the graphics.

---

Case 3: You're criticising my work

So far we've looked at examples involving other people, but now I'm going to turn the spotlight onto one of my own interactions.

I've never felt overly comfortable about how this interaction played out, as although I don't think my input was problematic, the dev clearly took issue with it.

I thought it'd be interesting to see how the conversation stacks up against CMU's characterisation of toxicity (obviously, there is a risk of bias here).

For background: Like many others, I use Kodi to watch/manage my digital media library. Kodi uses the an API provided by TheTVDB to get metadata and episodic information about media that it indexes.

Back in January 2020, scans started failing and Kodi started complaining that it was it couldn't parse XML when scraping:

11:28:01.942 T:1549005808   DEBUG: could not parse input XML
11:28:01.943 T:1549005808   DEBUG: XSLT: xslt transformation failed
11:28:01.944 T:1549005808   ERROR: Run: Unable to parse web site

I found a (longstanding) thread on Kodi's forums that seemed to be used when scraper broke, and others were complaining of breakage, so I posted to confirm that it was broken for me too, with what I'd found so far.

I feel fairly confident in claiming that there's no toxicity exhibited in that message, it's simply a statement of what I've found and a little bit of confusion about mismatched clues.

There were lots of replies complaining about TheTVDB changing their api again (suggesting this was something that happened regularly) and some users had even posted on TheTVDB's forum complaining of a change (I commented on one to ask if there were any recent changes, but said nothing about XML/JSON etc).

The next day, however, a Kodi dev appeared and posted a message in reply to another user, but obviously referring to my earlier note

For me, the wording of this opening sentence rates as being somewhat toxic.

I took some issue with it, but worded my reply a little more evenly than the comment I put into my own JIRA:

Thanks for the update, seems to be working now.

> Please stop spreading this ignorant rumor about JSON vs XML. 

Sorry but given the API response is JSON and the debug log explicitly 
complains about XML you can see how I'd come to that conclusion.

If people arriving at that conclusion is an issue for you, then perhaps 
improving the logging is in order - particularly given that Kodi's default 
logging level is particularly unhelpful for this issue 
- "Unable to parse website" really doesn't tell you much.

From a quick look at the pull and the addon, it looks like the JSON is being
parsed with regex rather than a... well... JSON parser. This seems to be why
changes in layout breaks things, if the JSON were being deserialised this
wouldn't be an issue, unless I've missed it it's not like they've changed any
attribute names, just changed the order in which they appear.

Whilst I stand by the point I was trying to make, with the benefit of hindsight, the wording of this was perhaps a little overly combatative.

Although I communicated it badly, it wasn't just the developer's tone that I objected to. The thread was full of people castigating TheTVDB for having made a "breaking change" to their API "again" when that wasn't actually the case.

To understand the issue: The API returns JSON in it's response body, and initally the response looked something like this

{ 
  "aliases" : [], 
  "banner" : "http://someurl/banner.jpg", 
  "network" : "BBC", 
  "image" : "http://someurl/picture.jpg",
  "seriesName" : "Boris the hedgehog",
  "slug" : "boris-the-hedgehog"
}

The change that TheTVDB made was to order attributes alphabetically, so the JSON became

{ 
  "aliases" : [], 
  "banner" : "http://someurl/banner.jpg", 
  "image" : "http://someurl/picture.jpg",
  "network" : "BBC", 
  "seriesName" : "Boris the hedgehog",
  "slug" : "boris-the-hedgehog"
}

The attributes network and image swapped places (these were the only two which moved).

RFC 7159 states that JSON is unordered, so there's a reasonable expectation that such a change would not break any clients, and by ignoring the standard, any non-standards compliant parser has signalled that any breakage is their own problem.

The criticism that TheTVDB were attracting was unreasonable: Kodi wasn't using a JSON parser (which wouldn't care about the change), but was instead using a regular expression (which did), but was then allowing TheTVDB to shoulder the blame.

As the dev had replied to me, I tried to better express this

Others may disagree, but I don't consider that this rates for toxicity under CMU's definition: there's a statement of fact and no hostility.

Unfortunately, though, it didn't land very well:

For the rest, thank you for the pitch in your 3. post in the forum 
suggesting that my 10 years work on non RFC compliant scrapers is 
a piece of shit.

Maybe it's time to stop fixing things what I really don't like or 
use (e.g. tvdb in overall). 

Having obviously upset the dev, I tried to smooth over any hurt feelings

I think we've got off on the wrong foot here.

FAOD I'm not for a second saying that your work is a piece of shit. 
There's a hell of a lot of non-RFC compliant stuff out there that 
isn't a piece of shit too. As you yourself have said, it's stood the 
test of being in use for a decade. I get why you might have chosen to 
go the regex route in the first place - it gives you a great deal of 
flexibility in adding new things, at the cost of sometimes being fragile 
- but as I say, it's a little unfair to be pointing fingers at the 
TVDB's api when they've not changed anything that should matter to a 
RFC compliant client.

I'm also not underestimating the demand on your time or energy that Kodi 
as a whole makes.

As far as Kodi goes, I'm a user and I'm doing what users do - having a 
grumble. It was only really your tone that prompted me to look closer and 
reply earlier, otherwise I'd pretty much just have posted to confirm the 
fix was working for me.

Which probably should have been the end of the story.

Unfortunately, another (newbie) user commented on the use of regular expressions

That is good news indeed that a JSON parser will be used. As a SW dev myself, 
this sounds less fragile in implementation if perhaps requiring a bit more of 
a "coder person" to maintain it. That said, maintaining complex regexps like 
the ones proved to work here for a decade are ... um ... fragile. This is 
especially so when the upstream data provider who claims to be remaining 
backwards compatible with their output production is in fact NOT doing so.

When the devs get to it, please LOUDLY highlight the release that first uses 
JSON parse instead of regexps. This will be a motivation for at least one 
user (me!) to upgrade my Kodi boxen.

Which again elicited a defensive response from the Kodi developer.

I decided that there was little to be gained from re-entering the discussion, so didn't reply, but I did note my thinking

I've not posted in the Kodi forum in the two years since (although to be fair, I hadn't posted there before this either), and I've also not bothered investigating and reporting bugs that I've found (beyond finding a workaround that suits my needs), partly because it doesn't feel like I can necessarily provide the feedback in a manner that'd be welcome there.

---

Contributing Factors

Communication Style

Looking at my Kodi example, with the benefit of hindsight, I can see that there was a lot of potential for conflict to arise via gaps in expectations around communication.

The Developer's opening sentence uses the word ignorant, an incredibly loaded and (usually) rude term. It's fairly clear that that's how I received and perceived it.

But, if English is not the developer's first language, it's quite possible that it was not intended. It's perfectly possible to be ignorant of $thing without being ignorant in general. It's just that the broader definition is how the term tends to be used nowadays.

My interpretation of that message drove my response, and there is some palpable annoyance in the following

If people arriving at that conclusion is an issue for you, then perhaps 
improving the logging is in order - particularly given that Kodi's default 
logging level is particularly unhelpful for this issue - "Unable to parse 
website" really doesn't tell you much.

The paragraphs that follows are less problematic in that respect, but contain potential for another issue

From a quick look at the pull and the addon, it looks like the JSON is being
parsed with regex rather than a... well... JSON parser. This seems to be why
changes in layout breaks things, if the JSON were being deserialised this 
wouldn't be an issue, unless I've missed it it's not like they've changed any 
attribute names, just changed the order in which they appear.

All this is from a quick look, so perhaps I'm missing something, but it seems 
quite unfair to be pointing the finger at TheTVDB when the issue appears to be 
how you're handling the responses 

I noted above that I was communicating my point poorly in this reply, but I think it might be helpful to look at why because it's almost certain that other's run into the same thing.

I work well when doing detailed technical breakdowns/troubleshooting, and communicate them best by walking the reader through each step (my Bitfi write-up being an extreme example).

But, I try and avoid doing this unsolicited when posting in other people's forums (or in comments on issue trackers).

If I'd instead written something like

From a quick look at the pull and the addon, it looks like the JSON is being 
parsed with regex (link) rather than a full JSON parser. 

This is problematic because the regular expression relies on the object 
attributes being in a specific order, even though RFC 7159 Section 1 (link) states

> An object is an unordered collection of zero or more name/value pairs, where a 
> name is a string and a value is a string, number, boolean, null, object, or array.

Standards allow unrelated parties to make decisions about what is and isn't OK to 
change within their applications. There's a sort of unwritten contract that says 
that changes to a server that are OK within the standard can be expected not to 
break a standards compliant client, and that any deviation from the standard 
makes you responsible for resolving resulting issues.

Scraping broke because the parsing on Kodi's side made an unsafe and non
standards-compliant assumption: that attribute order is and would remain fixed in 
upstream API's response bodies.

We can see that this was the case in the commit that fixed scraping (link) - 
the attributes network and image swapped place because attributes are now ordered 
alphabetically.

With that in mind, it seems quite unfair to be pointing the finger at TheTVDB 
when the breakage was because the client/scraper design left it susceptible to 
standards-compliant changes in the response body.

It would have read a lot like I was trying to dunk on the dev, which wasn't the intent. It's probably 50:50 whether such an explanation would have been accepted or if it'd have knocked the overally toxicity up a notch too.

There's undoubtedly a middle ground somewhere between these two message, but the devil really is in the detail.

So, we can see that communication style is important here:

• We can't know for sure whether the dev's use of the word ignorant was intended to be offensive, but that's how it landed.
• Similarly, whilst it wasn't my intention to cause offense/upset, it's clear from the dev's subsequent responses that I did.

It's possible we were both engaging in good faith and were simply misinterpreting each other's meaning.

---

Communication Mechanism

If we accept that differences in communication styles contribute to tension, then it follows that we should also ask whether the same misunderstandings would have arisen if we were communicating via a different medium.

Textual communications often lack important context and it's generally left to the reader to interpret and infer the tone in which it should be delivered.

If we take the following

I can't understand why this is implemented this way.

This sentence could be

• critical of an implementation, or
• a poorly communicated attempt to say that the author doesn't understand the choice.

One is criticism, the other a request for help, two different ends of a spectrum that would yield very different responses and the response would almost certainly depend on who responded to it.

Taking this further, in the Kodi example, if the developer and I had been sat across a table talking, it's quite possible the developer's tone would have shown they weren't trying to be insulting, and mine would have shown that I wasn't implying their work was "a piece of shit".

---

Entrenched Views

If we look at the simh example we can see that there are two very entrenched and contradictory views:

• The reporter feels that the software shouldn't amend his disk images,
• The developer feels that his choice is defensible, has precedent and seems to work well for the majority of users.

The problem is, both parties hold these convictions incredibly deeply, so there's no potential for movement by either party. At best, what you're left with is a slanging match - one side yelling "no", with the other side yelling "yes".

Of course, this isn't unique to the simh example either - the same is true in the most, if not all, of the others.

---

Knock On Effects

As we saw in both the simh case and "Return To Monkey Island", eventually developers hit the point where they feel they can't participate in any discussion around that topic anymore.

Whilst understandable, this can then have a knock on effect on future users and contributors.

For example, I use HomeAssistant as part of my Home Automation strategy.

Some time ago, HomeAssistant introduced a change to the way that DNS is handled (routing it all through a docker container which then forwarded it via DNS-over-TLS to Cloudflare). For various reasons that don't matter here, I was impacted by this change quite a bit after it was first introduced.

The change caused me problems, and from looking at HomeAssistant's forums it was clear that it had caused other users issues, and some of those users had been very vocal and persitent about it.

As a result, before I even became aware of the change, the developer's views had reached the point of not wanting to discuss further

Now Pascal did mention a way the forwarding can be resolved, but that people 
in this PR consider too much work nor do commenters agree with it being marked 
as unsupported. You can't have it all. Stray off the easy path, accept the 
consequences. If you want it to remain supported, it means it becomes our problem, 
because we are supporting it then. We are not interested in taking up any extra 
work that a) slows down any future improvements and b) benefits only a minority of 
our users.

And as usual, any topic about DNS ends up with insults towards us, so it's closed.

We are not interested in further discussing this topic.

As a latecomer to the issue, this left me in a frustrating position.

My setup was having issues because of unintended consequences of the change (and it's since become clear I wasn't the only one). I'd have been perfectly happy to contribute a PR to help resolve them, but the development team were no longer in interested in engaging on the topic (and, given the history, who can really blame them).

I tried, in good faith, to engage by creating a new issue, with specific details but received no response from the developers.

Not wanting to pester the developers, I ultimately created my own add-on to override the behaviour and fix the issues I and others were experiencing, and eventually, the development team independently made a change which resolved the issues people were experiencing.

What we can from this, is that it's possible for toxicity by a relative few users to lead to others in the community being unable to report/address related or adjacent problems.

The development team don't want to open themselves back up to abuse, and so they ignore any issues raised that appear to be about controversial topics, leaving potential contributors unable to get sign off on improvements and fixes.

This can potentially feed into a viscious cycle where latecomers themselves become frustrated that they cannot get engagement to help fix $important_issue, lashing out and further increasing toxicity levels (which pushes the devs further away).

---

Enter The Code of Conduct

The simh and Elementary OS cases share a common factor: they both involve parties with deeply held and incompatible opinions.

Where they differ, is the simh toxicitiy dragged on and grew over the course of nearly a year, whilst the Elementary ticket was tied off within just 9 days, limiting the opportunity for the levels of toxicity and resentment to increase.

Elementary were able to achieve this because they could refer to their community's Code of Conduct (CoC) and close the issue based upon violations of that

Although the detractors almost certainly weren't satisfied with this answer, they were no longer able to use that issue as a mustering point for abuse and arguments with the developers (and any new tickets they did raise in a similar tone could quickly be closed).

In simh's case, the ticket acted very much as a rallying point, with detractors using social media to try and attract support. This isn't an uncommon tactic during disputes, and in fact there are allegations of similar behaviour in some of the HomeAssistant pull request threads.

The introduction of CoC's has been unpopular with some, but as a whole seems to have had a positive impact, with CoC's being applied for in-person events (like conferences) as well as online interactions.

Most importantly, they provide a codified means to close off conversations that aren't just unproductive, but are at risk of becoming toxic.

A CoC provides a framework that community members can communicate under, providing them re-assurance that the communications they receive will be framed within the same framework. In other words, it sets expectation - in advance - about what will and will not be considered acceptable, and assures community members that abuse won't be tolerated.

Of course, like all tools, the effectiveness of a CoC depends entirely on how it's used.

There was a high profile example earlier this year of KubeCon trying to use it's CoC to silence Kat Cosgrove's legitimate criticism of a last minute change to it's mask policy.

Unfortunately, this isn't a one-off, and there are occurrences like that in other communities too. I think we're probably all aware of individuals within some communities who can sometimes be problematic with the little power they've been granted, and it's a difficult problem to address.

So, whilst having a CoC is a useful tool against toxicity - the existence of a CoC does not automatically imply that there is no toxicity, unfair or uneven application of the CoC may itself cause toxicity.

---

Specific to Open Source?

Although CMU's focus was Open Source, their paper doesn't suggest that any of this is specific to Open Source.

The claim was simply that the the "contextual, entitled, subtle and passive-aggressive" form of toxicity is more common in OSS than other forms of toxicity are.

At the start of the article, I stated that it seems to be an issue that surrounds software in general, and provided a recent example affecting propietary software. But, I believe even that is too narrow a scope: for example, similar traits can be observed on car forums.

I believe this form of toxicity thrives best where

• there's a user base
• some of those users have (or think they have) a deep understanding of the workings of the subject

Understanding that something could have been achieved in a way the user prefer lends itself well to a entitled and passive-aggressive tone: it should work the way I want it, and you're an idiot for doing it the inferior way.

---

Conclusion

There's no denying that toxicity is a pervasive issue, both online and IRL.

As we saw in the simh case, toxicity (or an eventual response to it) can completely destroy a community.

Although simh is a particularly extreme example, the presence of toxicity often means that would-be contributors aren't willing to enter the fold (or, as in my HomeAssistant example, are willing but cannot get engagement from the project team).

If left unchecked, opposing views develop into flame-wars, only becoming more deeply entrenched as the battle of words rolls on.

Conflict can be sparked by unrecognised communication issues: where text's lack of tone lends a different meaning to the reader than that intended by the author. There's a certain type of toxicity that thrives in the gaps between our words.

It's not always (in fact, in the context of community discussions, amost never) possible, but hopping on a quick call early in the conversation often resolves those kinds of misunderstandings.

There is, of course, also a balance that has to be struck: contributors need to be able to accept constructive criticism (no-one should be forced to tell me my Pull Request is good if it's going to break things), whilst critics needs to remember to keep their criticism constructive ("this PR is crap" isn't constructive, what needs fixing, why is it crap, is it really all crap?).

Tools like a Code of Conduct (CoC) help a community's stewards intervene in toxic situations before the damage is too great, but the CoC can itself breed toxicitiy if it's unevenly or improperly applied (or widely perceived as such).

There aren't any good one-size-fits-all answers.

Avoiding toxicity means recognising that our words can have an impact that we didn't intend, and recognising that sometimes a discussion needs to end before it escalates and becomes self-feeding.

Sometimes that takes the form of a moderator closing a thread, but most of the time it should involve each of us recognising that a conversation is no longer constructive and ending our own involvement (something humans are terrible at in practice).

I've waffled a lot, so I'll close this off with a final ironic observation:

Some readers may remember how excited everyone was when System of A Down really hit the scene.

But, because of disputes between band members, the band which brought us Chop Suey never really lived up to it's promise. Some of the band members have said in interviews that they feel that SoaD missed it's peak because they went on hiatus instead.

The band that rocked us with Toxicity missed it's potential because of toxicity between band members.

Without care, the same might be said of other communities too, and every time that it happens it represents the failure of something that people were once passionate about.