How PencilDrop judging works, and why there are no likes
Judges see two entries from the same page side by side, with names hidden, and pick the one they prefer. Every entry gets seen, and its rating comes from all those comparisons.
If you've ever posted a page in a colouring group, you'll know the hearts have more to do with timing than with colouring. A page that took three evenings, with a sky you were proud of, can get eleven because it went up at lunchtime on a Tuesday. Likes measure who saw a thing and when.
So on PencilDrop, ratings come from people looking at two pages and choosing between them.
What do you see when you judge?
Two entries from the same drop, side by side, with no names or numbers on them. You pick the one you like more, or say you can't decide, which counts as a draw and still tells us something.
You judge what's on the screen, however long it took to colour. A tidy page done in one sitting can beat an ambitious one that lost its way in the waves.
You don't choose what you see, either. Our server picks the pairs and makes sure every entry is shown a fair number of times, including the one someone squeezed in five minutes before the page closed. You can't go looking for a friend's entry to vote it up, and nobody can go looking for yours.
How is the rating worked out?
Each judgement is one small fact: in this pair, this person preferred A to B. A busy drop collects thousands of them. At the close we fit a rating model over every one (Bradley–Terry, if you'd like something to look up), and an entry's rating is the number that best explains all the pairs it was in. Beating a strong entry lifts you a lot. Losing to the page that goes on to win barely costs you.
That's how an entry judged twenty times and one judged two hundred times can sit on the same board fairly. The model keeps track of how sure it is about each one, and an entry without enough judgements yet is marked provisional until it has them.
Why are names hidden?
Who made an entry, and where they're from, stays hidden while a drop is being judged. If you're a quiet fan of a few colourists, their names on the screen would nudge you towards them without your meaning to, as they would most people. With the names gone the rating is about the page, and someone's first ever entry can finish ahead of a person who's entered every day for a year.
Whose judgement counts for more?
Anyone can judge, though the weights differ. If you entered the same drop, yours counts the most, because you've spent an evening with those exact shapes and know which bits were fiddly. Someone with an account who didn't enter counts for less, and someone judging in a browser from a shared link, with no account, counts for a little.
Each judgement is also weighted by the judge's standing. A new account starts out counting for a little less and builds up as it judges normally, and accounts that judge suspiciously (far too fast, say, or always in step with another account) are discounted or quietly set aside. Entering is free, and paying for Plus changes none of it.
What if you're among the first in?
A new drop has an awkward start, because the first people to enter have nobody to be compared with yet. So being early is paid for: at the moment the first three entries of every competition earn 30, 20 and 10 Ink on top of the usual, and the card for a drop says so while those places are open. The first two in have nothing to judge at all, so their rank shows straight away and they're paid the Ink ten pairs would have earned. Their entries are judged as soon as there are two to put side by side.
When are the results final?
While judging is open, ratings are provisional and move as judgements come in. When the judging window closes, some hours after the page itself closes, the ratings are fitted one last time over every judgement and the placings are written down. From then on the results are frozen, and they only change if an entry is removed for breaking the rules. A person looks over the top hundred entries of every drop before any prize goes out, and the prizes are digital things that live in the app, with no cash value.
How you can help
The most useful thing you can do after entering is judge a few pairs. Each one nudges two people's results a little closer to right, and you get to see how someone else handled the sky you just spent an hour on. Ten fit nicely into the time the pasta water takes to come to the boil.
These are our own examples of what a page can become, each coloured to a written colour plan. They aren’t competition entries.
