Dealing with Down Voting

Dealing with Down Voting

Many users have expressed vehemently their opposition to down voting when reviewing Project Spark user generated content (UGC), however, the powers that be at Team Dakota, the creators of Spark, do not yet wish to remove down voting as there has not been a consensus on the topic, and there are some cases which there are good games that do have high ratings (ignoring the quality levels with low ratings (fine example: the newest addition to TD library, Guily All the Same, has received a ton of downvotes (86% rating at the time of this post) despite the quality effort put into the level), and also overlooking ignoring the marginal levels with high ratings, but I digress).  BigDaddy has already written an article explaining the opposition to down voting (found here http://www.project-spark.org/blog/entry/it-s-time-to-end-down-voting ), so this "opinionated editorial" will focus on identifying and briefly evaluating the potential alterations to the current down voting system which have been mentioned by Team Dakota, as well as mention some of my own ideas for reducing the impact of downvote trolls in Spark.

Inform the Voters

What is a down vote supposed to mean?  Tell us the intended use.  Is this supposed to a Zoidbergian troll call "Your level is bad and you should feel bad!"? Should it be an anti-DLC statement, "this level contains DLC which I do not own, and cannot remix it, feel my misplaced wrath!"?  How about remixes, "You didn't do most of the work, I am going to down vote so the original creator will get attention" (this may be addressed soon with the upcoming improved lineage system, but people may continue to vote in such a way)?  Or is it just to be "This game has a high rating, but I didn't quite like it at the level of 95%, I think it is more of an 85% game, so I will down vote to impose my will on the game rating"?  How about X1 compatibility, "this game doesn't function well on the X1, which I am using, it looks cool, and I want to play it, but I can't- BURN!"? None of the above, all of the above?  At least tell us before voting the intent so we can be consistent and have the scores be meaningful.



Captcha Censoring

I am not in a position to say if there are bots in Project Spark at the moment, but a captcha form being required for a down vote to be processed would have two effects.  Firstly, it would prevent down vote bots from becoming a thing, if they haven't already.  Secondly, basically everyone hates captcha, so if they want to down vote, they will need to hate the level more than they hate doing a captcha.  However, this will also likely result in levels ratings being skewed higher, with only especially bad levels being filtered down.  And of course those people would likely feel especially embarrassed.

Feedback Tags

Perhaps require feedback to be tagged with why the level was down voted.  These tags could be displayed in a histogram to the creator so that they may improve their level, or future levels.  This way a down vote will come with constructive feedback.  Heck, allow anyone to submit feedback tags even if they don't plop down a down vote.  It may be worth adding a "Be Trollin'" feedback tag in case any trolls feel like being frank about it.

Scale the Down Votes

First and Foremost

Regardless of which algorithms Team Dakota implements, please do not tell us, keep it a black box.  Do not tell trolls how to be effective trolls, if the trolls know which obstacles are up, they will figure out how to get the most troll for their time.

You Must Be This Tall to Down Vote

A minimum spark level for down votes to actually be counted may negate the effect of trolls who log in to Spark with no intent on creating or actually playing games.  If there were a level of 5, it wouldn't be hard for a real user to reach, however if a user switches to troll mode after losing interest, it would be a rather low barrier of entry for them, as I recall reaching level 5 within a day or two of playing.  Higher levels or 10 or 15, or what ever, would be more likely to filter out potential turn coats, but would have a minor side effect of also removing legitimate down votes from members who have not yet rising above the threshold.  This may be mitigated by decensoring the down vote after the user reaches the appropriate level.

Weighting the Down Vote

It has been mentioned that some factor could scale the impact of a vote, such as the user's spark level.  At first glance, this would also reduce the impact of a low level troll.  But at the moment, it isn't all that entirely difficult to quickly level in Spark, especially with a few tricks which any mildly clever user could come up with.  So do users who figured out these tricks really have a more important opinion when it comes to voting than a user who has ignored the dailies and didn't concern themselves with leveling?  The low hanging fruit aspect would be combining the minimum level discussed previously with the weighting.  Perhaps a hard cap and a soft cap, concepts that may be familiar to MMORPG players, but flipped upside down.  The hard cap could be that users of a spark level below 5 would carry a weight of zero, while users between level 5 and 15 would have a soft cap reducing their effectiveness by some amount.

Frequency of Voting

It may be useful to take into consideration how many down votes the down voter is doing, either per day or as a percent of how many ratings in total they have given.  If a user down votes every level they come across, their rating could mean less.  Or if they down vote a lot, but also up vote a lot, their vote could have a normal weight.  Or perhaps if they rarely down vote, and/or rarely up vote after playing a game, perhaps their votes should have more of an impact.

Voting Relativity

If a user down votes a level which has a substantially high number of up votes, perhaps the down vote shouldn't mean as much.  Even Babe Ruth didn't get 100% of votes on Hall of Fame Ballots (today writers use that as rationale for leaving surefire hall of famers off their ballot, essentially down voting, just because if the best player in their minds wasn't perfect, why should anyone be given a perfect rating? It's silly, but again I digress), so maybe the people in the minority shouldn't impact the score very much.

Subtract the Toxic Noise

Alternatively, find a "background noise" of level ratings, this view of this approach comes from my background in analyzing genetic/epigenetic microarrays, where even something that should be 100/100 or 0/100 will have its signal altered by the overall and/or nearby environment.  Essentially, look at all levels, with particular attention to the very good levels.  If even really good levels with 100 total votes is expected to get 5-10 down votes just from the toxic environment of the internet, reduce the impact of the first 5-10 down votes per 100 votes.

Other Suggestions

Do you have any other possible solutions worth considering? Post in the comments below.

Preview: "A Simple Task" by wertandrew (released)
Bungie Community Hosting a Competition in Project ...

Comments 8

 
An_Average_Recon on Saturday, 29 March 2014 21:25

This pretty much covers everything, nice post.

Another idea: Maybe you have to play 10 hours of ugc and create mode combined before you get to vote? This way people new to the game don't just downvote because their expectations are too high.

This pretty much covers everything, nice post. Another idea: Maybe you have to play 10 hours of ugc and create mode combined before you get to vote? This way people new to the game don't just downvote because their expectations are too high.
BigDaddy on Sunday, 30 March 2014 00:00

Thanks for the thoughtful entry moose. Another thing that could be tried which would make it a little bit better is to allow people to share but have the creator turn off up and down votes. It works with you tube why not do it here. Perhaps the creator just wants to share and doesnt care for up or down feedback. Why not allow them that option?

Thanks for the thoughtful entry moose. Another thing that could be tried which would make it a little bit better is to allow people to share but have the creator turn off up and down votes. It works with you tube why not do it here. Perhaps the creator just wants to share and doesnt care for up or down feedback. Why not allow them that option?
Brasten on Sunday, 30 March 2014 08:50

Agreed that mostly covers it.

Although going along with the Tags type is the need for a "better" feedback system. Comments will never be truly useful tool outside of us "Creators" who really care about the improving each others craft... or a space for trolls who want an early Flag.

This would go to rating core concepts in game design, almost looping back to Roles discussed in the Lineage system preview. A World could have a great story, but horrible controls, or lack luster Props. A simple Up/Down vote system gives no method for even broad brush feedback by Players. Up/Down reads out as Love/Hate, with no inbetween. If Players were asked to rate "Core" concepts, it would provide feedback a creator and see and understand where to improve.

I'll use Audible star ratings as the example. Audible asks users to give a rating to rate Overall, Performance, and Story. This is actually fairly useful, especially in the areas of Performance, because its an Audio book, and if the person who is reading sounds like an off-key-monotone it can kill even the best Story. Apply this logic to a Game. If we are talking about Controls and they were there only bad part of the experience, a Creator now knows where to focus their time, just like any developer.

If a Player feels like going deeper (perhaps prompted by having to actually think and rate 5 or so core areas) they can then use the Comment system or add Feedback Tags of a more detailed nature. It also offers a learning opportunity for "players" who aren't normally Creators or are new to Creating to see, experience, and think about core aspects of game design.

This could also help filter out Downvote mashing trolls. You can create a baseline for that game and a user demonstrating random "vote" selection outside the norm on a frequent bases across many games, can be noted by the system for a closer look by a moderator.

For controls I would also flag for what the Player was actively using during the game. If they were using a Controller have their Controls rating go into the Controller bin. If it was Keyboard and Mouse, the KB&M bin. This is going to be constant issue as people move outside of the 3rd Person Adventure game format and the Defaults start to break. How many games are getting down votes simply because their controls were built exclusively with one platform in mind and do not translate well to the other? How many RTS games will get totally burned by console players because they are solely KB&M driven? How many console FPSs will get burned because Left and Right trigger default to 1 & 2 on Keyboard?

Agreed that mostly covers it. Although going along with the Tags type is the need for a "better" feedback system. Comments will never be truly useful tool outside of us "Creators" who really care about the improving each others craft... or a space for trolls who want an early Flag. This would go to rating core concepts in game design, almost looping back to Roles discussed in the Lineage system preview. A World could have a great story, but horrible controls, or lack luster Props. A simple Up/Down vote system gives no method for even broad brush feedback by Players. Up/Down reads out as Love/Hate, with no inbetween. If Players were asked to rate "Core" concepts, it would provide feedback a creator and see and understand where to improve. I'll use Audible star ratings as the example. Audible asks users to give a rating to rate Overall, Performance, and Story. This is actually fairly useful, especially in the areas of Performance, because its an Audio book, and if the person who is reading sounds like an off-key-monotone it can kill even the best Story. Apply this logic to a Game. If we are talking about Controls and they were there only bad part of the experience, a Creator now knows where to focus their time, just like any developer. If a Player feels like going deeper (perhaps prompted by having to actually think and rate 5 or so core areas) they can then use the Comment system or add Feedback Tags of a more detailed nature. It also offers a learning opportunity for "players" who aren't normally Creators or are new to Creating to see, experience, and think about core aspects of game design. This could also help filter out Downvote mashing trolls. You can create a baseline for that game and a user demonstrating random "vote" selection outside the norm on a frequent bases across many games, can be noted by the system for a closer look by a moderator. For controls I would also flag for what the Player was actively using during the game. If they were using a Controller have their Controls rating go into the Controller bin. If it was Keyboard and Mouse, the KB&M bin. This is going to be constant issue as people move outside of the 3rd Person Adventure game format and the Defaults start to break. How many games are getting down votes simply because their controls were built exclusively with one platform in mind and do not translate well to the other? How many RTS games will get totally burned by console players because they are solely KB&M driven? How many console FPSs will get burned because Left and Right trigger default to 1 & 2 on Keyboard?
Sparkotron on Monday, 31 March 2014 12:42

Voting systems can be abused either way and/or may not be representative of 'why' a particular vote was used in the first place. The "Inform the Voters" section from Moose explains this well.

I will repeat myself and state that keeping track of both "download counts" + "average time played per download" would be much more representative of how much a UGC is liked, and would be pretty difficult to abuse. Levels that are not long by design or don't quite fit into the "game" category (like Guilty All the Same) could simply have a tag applied to them which would be taken into account and inform potential players that "played time" doesn't really apply here. Lastly, combine this kind of system with the ability to add feedback to the creator, and you have something pretty solid (imho).

Voting systems can be abused either way and/or may not be representative of 'why' a particular vote was used in the first place. The "Inform the Voters" section from Moose explains this well. I will repeat myself and state that keeping track of both "download counts" + "average time played per download" would be much more representative of how much a UGC is liked, and would be pretty difficult to abuse. Levels that are not long by design or don't quite fit into the "game" category (like Guilty All the Same) could simply have a tag applied to them which would be taken into account and inform potential players that "played time" doesn't really apply here. Lastly, combine this kind of system with the ability to add feedback to the creator, and you have something pretty solid (imho).
Snicken on Monday, 31 March 2014 17:44

As a creator, I would find it immensely helpful if the voters were required to give a reason for their down vote. Obviously any system can be abused but if the vast majority of the down votes came with a reason of "controls", it would help focus my attention when making tweaks. The inclusion or exclusion of the down vote itself is a good debate. I do not know where I align in that debate. Without the down vote, it will be hard to tell the truly BAD level from the "just not worth an up vote" level. But strictly showing the up vote to download ratio should be able to indicate if a level is truly awesome. I'll leave that to the people how are much smarter than I am. Thanks for giving this issue the attention it deserves!

As a creator, I would find it immensely helpful if the voters were required to give a reason for their down vote. Obviously any system can be abused but if the vast majority of the down votes came with a reason of "controls", it would help focus my attention when making tweaks. The inclusion or exclusion of the down vote itself is a good debate. I do not know where I align in that debate. Without the down vote, it will be hard to tell the truly BAD level from the "just not worth an up vote" level. But strictly showing the up vote to download ratio should be able to indicate if a level is truly awesome. I'll leave that to the people how are much smarter than I am. Thanks for giving this issue the attention it deserves!
invisibledoogey on Thursday, 03 April 2014 15:01

I also wanted to weigh in here. As a general concept, down votes are an okay way to deal with weeding out the good levels from the bad. But as stated by others, down votes don't have context AND I believe down vote bots are actually a thing in Project Spark now.

Case in point, I re-uploaded Jetsam last night with some nice optimization fixes and general game balance. My first version was inferior, but had an 85% approval rating. I re-uploaded knowing I'd lose all of those approvals. Within a few minutes, I had 10 downloads on the new level, each that was a down vote. It's slowly recovering now, but still hovering around a 34% approval rating. I have a feeling based on the rapidity and frequency of those initial down votes that these were down vote bots.

There needs to be an extra step added to downvotes, either in forms of tags, CAPTCHA fields or a required description. This is necessary to give creators constructive feedback on how to make improvements and reduce the ability for people to activate spam bots.

I have thick skin, so I can live with a 34% approval rating for now. BUT if I were new to Project Spark, spent dozens of hours toiling away on a passion project, then uploaded and immediately received down votes, that would most definitely turn me off from creating more in Project Spark.

We need to find ways to encourage people to upload and keep the practices of immediate rapid down voting from happening on levels. Otherwise, we'll lose a lot of passionate people who can't take the false negativity.

I also wanted to weigh in here. As a general concept, down votes are an okay way to deal with weeding out the good levels from the bad. But as stated by others, down votes don't have context AND I believe down vote bots are actually a thing in Project Spark now. Case in point, I re-uploaded Jetsam last night with some nice optimization fixes and general game balance. My first version was inferior, but had an 85% approval rating. I re-uploaded knowing I'd lose all of those approvals. Within a few minutes, I had 10 downloads on the new level, each that was a down vote. It's slowly recovering now, but still hovering around a 34% approval rating. I have a feeling based on the rapidity and frequency of those initial down votes that these were down vote bots. There needs to be an extra step added to downvotes, either in forms of tags, CAPTCHA fields or a required description. This is necessary to give creators constructive feedback on how to make improvements and reduce the ability for people to activate spam bots. I have thick skin, so I can live with a 34% approval rating for now. BUT if I were new to Project Spark, spent dozens of hours toiling away on a passion project, then uploaded and immediately received down votes, that would most definitely turn me off from creating more in Project Spark. We need to find ways to encourage people to upload and keep the practices of immediate rapid down voting from happening on levels. Otherwise, we'll lose a lot of passionate people who can't take the false negativity.
SwarmHammer on Friday, 04 April 2014 00:23

Force all downvoters to leave a reason. Have all votes being moderated for the type of comments that it has. One idea to make sure people just do not type anything meanigless, use somekind of a survey with key questions that the downvoter has to rate according to how he feels about certain aspects. Make those answers all required. If the voter decide to cancel the questionnary, his vote will not be counted. Each one of those survey would be accessible to the creator by means of a graph that represent a rating for each question, with the number of downvoters. Trolls will not like the system.

Your freedom of expression is expressed in this by deciding if you are willing to go through all the hoops to simply downvote a level because you are a lame person that hates everything that is good.

Force all downvoters to leave a reason. Have all votes being moderated for the type of comments that it has. One idea to make sure people just do not type anything meanigless, use somekind of a survey with key questions that the downvoter has to rate according to how he feels about certain aspects. Make those answers all required. If the voter decide to cancel the questionnary, his vote will not be counted. Each one of those survey would be accessible to the creator by means of a graph that represent a rating for each question, with the number of downvoters. Trolls will not like the system. Your freedom of expression is expressed in this by deciding if you are willing to go through all the hoops to simply downvote a level because you are a lame person that hates everything that is good.
SwarmHammer on Friday, 04 April 2014 00:24

And by you is a generic term I am barelly speaking to the downvoters at large

And by you is a generic term I am barelly speaking to the downvoters at large