PyCon Nove

Florence

19th - 22nd April 2018

PyCon Italia Community Voting: How to rate the talks?

Dear pythonistas, this post wants to recap some suggestions about the community voting and summarise the algorithm used to generate the talks ranking.

First of all however I would like to remember all the deadlines of PyCon Nove:

  • 07 jan 2018 - End of the Call for Proposals
  • 08 jan 2018 - Talk Voting begins
  • 21 jan 2018 - Talk voting closing
  • 04 feb 2018 - End of the Early Bird fare
  • 04 feb 2018 - Publication of the schedule
  • 05 feb 2018 - Start of the Regular fare
  • 31 mar 2018 - End of the Regular fares
  • 01 apr 2018 - Start of the On Desk fares

Voting criteria

An incomplete proposal is the largest negative factor I consider. If a proposal is incomplete, I can’t really say what the speaker will talk about, or even if they know the subject matter for their talk.

Finally, I consider whether the abstract is compelling. Without regard to the actual subject, is the abstract written in a way to attract an audience? Is it full of boring clichés, vaguely worded assertions about the superiority of one tool over another, cute metaphors, or hand waving? I look for an abstract that shows the speaker is excited about the topic, and that they will be conveying that same excitement to the audience. If a proposal does not have what described above It's important and useful ask the submitter to provide more detail, in this way we could have a fantastic talk or the possibility to give the right rate to the proposal.

So in case you want to ask clarification about the talk or if you want to suggest any edit, open a comment specifying your motivations and clarify what are you asking for.

When you're ready to rate the talks keep in mind that the algorithm used to calculate the rating is the Schulze Method

What is the Schulze method?

The Schulze method is a specific way of ranked voting which is in the CIA used for ranked voting (by the same proposal in Agora Bergamo). It was developed in 1997 by Markus Schulze as an electoral system for selecting a single winner by using votes that express preferences. Without getting into too many details, the method works with the following system: voters make an ordered preference list (aka ranking the options) where ties and/or leaving out options are allowed. Then for every two options, they are compared (ignoring the other options), checking which option is more preferred by the voters. After the options are compared, the system calculates the strongest paths (aka which option is the most preferred overall by all voters) and the final results are made.

Looks confusing? Here is an example supported by Wikipedia:

Let’s assume that there have been 5 talks submitted for PyCon Nove, named A, B, C, D and E. There have been 45 votes cast in the following order:

First, the pairwise preferences are calculated which will look like this:

Green means that the voters preferred the option in the column, red means that the voters preferred the option in the row (for example, 20 voters preferred talk A over B, and 25 who preferred talk B over A). There is no undisputed winner by only looking at the pairwise differences here. For the next step, the strongest paths are calculated (how it is exactly done is visualised with direct graphs in the Wikipedia article), which will look like this:

Green means that the voters preferred the option in the column, red means that the voters preferred the option in the row (for example, 28 voters preferred talk A over B, and 25 who preferred talk B over A, so with the Schulze method talk A is better than talk B). Based on the Schulze method, the ranking is E>A>C>B>D, therefore talk E is the first in the rank for PyCon Nove.

N.B. If no preference is expressed (from 1 star to 10 stars) the algorithm consider an avg vote. In our case, an un-voted talk is starred as 5.

So here we described the hard way 😅 used to calculate the general ranking of the talks. After that the ranked contribs are distributed among the community in order to fulfill the track of the conference.

Here the Jupyter Notebook used for Pycon Otto to organise the conference track: http://bit.ly/2Al210w
If you have any doubt or question please do not hesitate to write us at info@pycon.it


— Ernesto, 29 December 2017  ratingcommunity votingvoteabstract

0 Comments

Archive