Reviews would theoretically be public, but no punches need be pulled. Reviews are for other readers, not to save the authors' egos. (Besides, I would be thrilled to kick off a Howett-style meltdown over reviews of free short stories.)

I trust that fanficcy people would be able to say, "I can see that someone really into [topic] would enjoy this story, but outside of that niche, meh." From what I've seen of literary reviews, I trust fannish ones better--they're more centered on "here's what works & what doesn't and what you might not care about" than attempting to claim some objective standard of literary quality.

The biggest problem I'd expect with short stories would be a tendency to slip into beta mode and start suggesting actual corrections for the stories that are *almost* good reads on their own. The number of self-publishing author who can't properly wield a comma is terrifying.