How should we measure intelligence? Or, in general, how should we measure a quality that doesn’t really have a good definition (the ability to apply knowledge to deal with new or trying situations?)? If the quality has the property that those that possess it are capable of judging who possesses it better than those who don’t, and if there is a lot of subjects in our environment, we could try an iterative algorithm.

Ask every subject to assess everyone they know in some way. Extract the first iteration of the measure of the quality for everyone. Then extract the second iteration by computing the weighed average of the assessments, weighed by the quality itself. Continue until convergence is achieved.

For example, we could ask everyone to provide the IQ of everyone else. No definition necessary — just provide a number. The resulting IQ of everyone would be, at first, the average of everyone’s rating of them. Now that we have the initial IQ, we use it to weigh the calculations of a more accurate IQ. We keep doing it until a subsequent iteration doesn’t alter the IQ calculations in a meaningful way.

It’s a wonderful way to come up with a tangible rating for something that’s undefinable, based simply on the easy to accept assumptions that a large number of people will together come up with the right answer, and that people who have been judged more highly should have more to say.

What’s more interesting, though, are the specifics around how exactly to do the assessment.

• Asking everyone for a number suffers from the problem of unclear scale — one person’s 100 will not be the same as another person’s 100. A clever variation on the above is to ask people to compare others as opposed to rating them, because while we have a hard time quantifying things, we find it rather easy to compare options (an evolutionary ability?). In fact, if a measure doesn’t have any other meaning other than comparative, it probably should remain comparative only rather than quantitative (i.e. if a score of 100 doesn’t mean anything, why should we use scores). So, I will provide a ranked list of people, without much consideration for how much better the fifth person is from the sixth (they may even be just as good). We can then combine all such rankings into one global ranking with a simple rule: if person A is consistently above person B in individual rankings, person A should be above person B in the resulting ranking. In other words, for every pair of people, determine whether (A>B) occurs more often than (B>A) and place A above or below B, appropriately. There will no doubt be conflicts (e.g. A>B, B>C, C>A) which should be resolved in a way that violates the fewest individual inequalities. In such a case, the weighing can still be done by assigning some score to the resulting order, and maybe by taking into account how many conflicts I am contributing to (if many, my weight should be lower).
• Asking everyone for one number suffers from the problem of confidence. I may be highly confident of one assessment and not at all confident of the other. Including a confidence rating (e.g. provide a score that you think is most likely to be true, but also scores on either side that you think are as likely as they are not to be true) may provide more information.