A Break For The Crowd

The use of crowd intelligence seems to be continuing to grow. Perhaps decision democratization or committee consensus would be better descriptions, but neither has quite the cache of “intelligence”. Even if our crowds are not expert, (and they are supposed to be cast wider than just experts,) we should hope that they are wise.

The latest piece of crowd intelligence to catch my eye is Bernstein et al’s study in medical diagnosis (1). The diagnostic problem was the assessment of hip fractures. That is the problem of a break at the neck of the femur when, if there is displacement of the ball from the socket, the surgical solution is a hip replacement. If it is not displaced, pinning is the treatment. Diagnosis is done from X-ray pictures; nobody waits until they get to the table to decide.

The baseline for correct diagnosis was 69% for the twelve orthopedic surgeons chosen for the test. Note: no unlucky old folks were affected by these diagnoses. Next, small committees were formed of various sizes and the diagnosis was decided by majority vote.

Committees did better with 3-member committees hitting 77% and 5-member committees getting to 80% correct.

The big experiment was done next with a 40-member internet-based group. The critical issue here is response time. The input from a group of nine was taken as a sufficient “crowd” to get things right and this could be done in about an hour, with the longest time in a ten-case study being less than two and a half hours.

The authors conclude that their “wise crowd” should have people with diverse opinions and, of course, the democracy of the internet means that no one dominates the committee. So this may be a low-cost error-reduction procedure, which sounds good.

However, the “crowd” is still a crowd of “experts,” in other word, a “wise crowd” and this takes us away from the real idea of crowd intelligence where the non-expert is included thus enabling an unexpected solution to emerge.

Personally, this leaves me in a bit of a dilemma. The wise crowd decision over a broken leg seems to work OK, but it seemed to be a failure in Congress where six “wise experts” couldn’t achieve a consensus. The latter is clearly a case where true crowd intelligence is required.

  1. http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0027620

Leave a Reply