The rating system

[clifford_thornton]

So I write for a little online magazine called Tiny Mix Tapes. Kind of like a Pitchfork-esque publication, where I review jazz and outsider music.

They use a "dot" system (instead of stars) on a 1-5 scale, with five being the highest. Most things that they have heard of or are hyping get 4 or 5 dots, I'll say that off the bat. Most of the things I submit I give between 3.5 and 4.5 dots, with a few submitted as 5s. I rarely review anything I don't like. Unfortunately, rarely do any of my "gushing" reviews or even "very positive" reviews stand with the rating I submit. Usually half and sometimes a whole dot is notched downward, but I haven't complained... until now.

Here's part one of my exchange with the editor, and I'm interested to see how he responds.

Dear ____,

I'm getting a bit agitated by the fact that most of my reviews keep getting notched down from what I give them. I can understand notching a five down to a four and a half, but four down to three and a half or three seems excessive. If I think a record is excellent and review it as such, and the ratings are not indicative of that feeling, it sends a weird message. I rarely even bother to write about music that I don't think is far and away worth buying and listening to, especially within the realm of jazz and creative music. If it were up to me, for example, despite how good the new Blues Control is, the Halvorson would be heard and enjoyed by just as many people. However, one four-dot review is allowed to stand while the other isn't. A half-dot might not seem like much, but it's quite a difference in my opinion. The Charles Gayle from a while back is another example - I submitted it as a five, and it was given a four, even though the tenor of the review was gushing and it is pretty much my record of the year so far. As you know, I hem and haw with the system - it's often hard to decide, because the language of the review should speak for itself. But I do my best to shore up the words and dots. Often, I've had comments from my readers about how the review and the rating don't match, and they're curious why I rated something lower than what the language seems to indicate. I have to tell them it's the editors assigning the ratings, not me, because that's ultimately what happens.

Honestly, if my opinions weren't reliable - or I didn't feel they were - regarding the music I choose to write about, it'd be one thing. But I'm trying to carve out a life as a critic, writer, and supporter of certain kinds of music. If I want this collaborative venture between myself and this music to mean something, it gets tiring to explain to people that a "middling" rating wasn't my decision. I'd like to keep writing for TMT because it's a good and visible site that has helped me to hone my craft. Honestly, I think it's improved my writing to be part of the TMT team. And with fewer online/print publications to review for, I've got to hold on to what I can. But the rating system's vagaries are really frustrating sometimes.

All right, ranting over. Would be interested in hearing your thoughts.

Thanks,

I've had back-and-forth with people on whether or not it's worth giving something a "poor" review. I don't always give perfect marks, obviously, and have ripped things a new asshole on a number of occasions. But if I'm going to submit a review to a more "general" site like TMT, it's usually going to be for a release that I think is really worthy of the attention of a wider audience. Hence the usual positivity. I'd be interested in thoughts from the o-board, though, on whether my bitching is worthwhile, egoist, or something in between.

[CraigP]

I think it's weird that you write the reviews but don't assign the ratings. And given that, I think it's weird the editor is lowering your ratings; I would have guessed the opposite problem. It makes sense you're going to spend time writing about things you like, but perhaps the editor thinks you're overpraising too many releases. How many 4.5 or 5 dot albums have you reviewed? Five dot albums should be pretty rare.

[A Lark Ascending]

I'm not a great fan of 'rating' music - there's so much subjectivity involved and every listener comes from such a different context. Yet even knowing that, 4 or 5 stars perks my ears up, 1 or 2 has me cautious. I prefer places where you have to read the review to elicit how the reviewer feels.

Having your ratings adjusted so they end up not matching what you wrote does seem bizarre.

I have to deal with a rating system when being assessed as a teacher and when assessing others. I've no issue with commenting on a lesson where the myriad of things that go on, good and not so good, can be discussed with plenty of nuance and a general focus on how to get better (or even better). But I hate sticking the wretched 1-4 numbers on the end because they then become the label that the reviewed person takes away. "Gosh, I'm 'Outstanding' (1)" or "Drat, I'm just 'Satisfactory' (3)" or - worst of all - "They've just told me I'm 'Inadequate'(4)" when the reality is so much more complex.

I read a comment on a classical site yesterday where a woman maintained that top ten lists were a male thing. I wonder if the need to 'rate' in categories might be too.

Edited July 1, 2012 by A Lark Ascending

[papsrus]

Why not forget about the dots altogether, focus on the writing, and let the editors assign the dots (since that's kind of what's happening anyways)?

If you feel the need to, you could make this explicit in your writeup -- ("Although I don't assign the number of dots, ..." ).

Free yourself from the dots. Enjoy the music and the writing.

[JSngry]

Never forget what Dots are.

[clifford_thornton]

I think it's weird that you write the reviews but don't assign the ratings. And given that, I think it's weird the editor is lowering your ratings; I would have guessed the opposite problem. It makes sense you're going to spend time writing about things you like, but perhaps the editor thinks you're overpraising too many releases. How many 4.5 or 5 dot albums have you reviewed? Five dot albums should be pretty rare.

Yeah, I put the ratings in and they're lowered. I don't know, generally I only send them the stuff I think is really good and could use some cross-genre attention, since they mostly cover rock & electronic music.

I've done one 5 and one 4.5 in two years, most are in the three or four range. If anything's below three, I just don't bother with it at least for this pub. For NYCJR or even my own blog, I've definitely noted some 2s - or would, if a rating system was in use.

Would be curious if any o-board contributors to places like Down Beat have anything to add, since that pub. relied so heavily on ratings. If it were up to me the star/dot system would be kaput.

[paul secor]

If the editor is changing your ratings, he or she is basically changing your opinion, even if they leave your words intact. I can't speak for you, but if it were I, I'd tell the editor to leave my ratings intact or sayonara.

[AllenLowe]

I agree with Paul. It's a bizarre thing for them to do.

Or maybe you should rate everything one star higher than you think it deserves. That way when they cut the rating, it will be closer to your intention.

[clifford_thornton]

Yeah, both of those are good points. Thanks.

He responded in detail to my concerns, and I have yet to go through and itemize it (and I'm leaving town tomorrow). Part of me thinks diversification is good and having material in TMT really helps the visibility angle. But then again, I don't get a paycheck from them and the adjustable ratings system really cheeses (!) me off.

[Teasing the Korean]

A friend of mine is a respected film critic. For one of the rags he wrote for, he assigned a star rating geared towards people who glance only at the rating, and saved the more nuanced review for the narrative.

He tended to assign stars based on what your average chain-restaurant, mall-shopping American might tend to go for. His narrative reviews were directed more toward film aficionados.

Edited July 2, 2012 by Teasing the Korean

[CraigP]

But then again, I don't get a paycheck from them and the adjustable ratings system really cheeses (!) me off.

I think that it would cheese me off that I'm providing free content for a presumably for-profit publication, given that they accept advertising. But it sounds like you have a longer-term goal. BTW, did you mean to only give Bending Bridges 3.5 dots?

[david weiss]

I've heard a few stories about editors knocking off a star or so because they didn't agree with the rating or more often because it was in the 4 plus star range and the editor didn't think it should have such an exalted status.

I know of one instance where it happened to me. The writer forwarded the review to me before it went to press and it was given a 4 1/2 star rating. This was for a magazine that for years only went up to 4 stars and had only recently changed their rules to allow for the 5 star rating for what they thought would be reserved for classic records only. They took away the the 1/2 star when they went to press. The writer sent me a nice note saying he didn't know what happened but it seemed pretty clear to me......

Actually I just thought of a second.....

Apparently an editor for a French magazine took off a star because he hadn't heard of me.

I was friendly with the writer in question and the next time I saw him, I busted his chops a little (something I do try to refrain from doing) saying really, 3 stars for that CD (he had told me how much he liked it).

He then told me what happened.......I guess I should take that one with a grain of salt but I did believe him and had heard other stories about the editor in question......

It all seems like a huge waste of time......

[thedwork]

However, one four-dot review is allowed to stand while the other isn't. A half-dot might not seem like much, but it's quite a difference in my opinion. The Charles Gayle from a while back is another example - I submitted it as a five, and it was given a four,

agreed. it only seems like "not much" to someone not following the logic thru. using a 5 star system, deleting a full star as they did to your Gayle review, represents a 20% difference. that is huge.

I can't speak for you, but if it were I, I'd tell the editor to leave my ratings intact or sayonara.

i'd do the same. it makes no sense and is wildly rude to the writer who spends their time getting things as they want them. they should either use the rating submitted by the reviewer or not use the rating system at all. if the ratings i submitted were consistently altered without my permission i'd be pretty damn angry...

i've written for a few places and only one of them (Blurt-online.com) used a rating system. they never altered any of my ratings or words. my one review printed in STN had no rating and AAJ had no rating systems while i was writing for them; though they messed w/ words a bit (but in a pretty professional way...).

Edited July 8, 2012 by thedwork

[clifford_thornton]

But then again, I don't get a paycheck from them and the adjustable ratings system really cheeses (!) me off.

I think that it would cheese me off that I'm providing free content for a presumably for-profit publication, given that they accept advertising. But it sounds like you have a longer-term goal. BTW, did you mean to only give Bending Bridges 3.5 dots?

I gave it four dots!!!

[CraigP]

If the TMT site had a dot race, which dot would win?

[JSngry]

Let's run that and see..

[clifford_thornton]

Thanks guys. This was really helpful. Here's my gestated response to his email (which I'm not printing here).

Dear ________,

Okay, here we go. I was in Brooklyn getting set up with an apartment and all of that. Back online now with more regularity. Lesson in how not to toss an important query in someone's yard before leaving for 9 days.

Basically, what confuses me is the push for continued/broad cultural relevance, almost apart from the work at hand, as a justification or some sort of validity. Let me rephrase that - when writing reviews, having a lot of extraneous ideas seem to detract from the central focus, which is on the music. I don't think that every review, even of something "really good" or "great" (4 or 5 dots), should have to find a new spin on why the work is important. Sure, there isn't much out there like Blues Control, but this is also - for me - coming from the perspective of someone who doesn't listen to a whole lot of rock music. My noting of its incomparableness may reflect just as much on the fact that I don't have a lot to compare it to, as it does their ability to make stand-alone work.

Improvised music is, to me, a language I know well enough to feel that it's like breathing or eating, so perhaps the words I use to describe it could seem a little more pedestrian. In other words, how does one describe or validate one's oldest friend? That is where the writer's judgment of quality vis-a-vis expertise should come in. I'm reading writer X because I trust her opinions on music Q, because this writer's bias is informed through familiarity and love. Nothing wrong with that. Mary's record is a continued refinement of a certain aesthetic, and given the fact that her working methods are broad, that narrowing is interesting and important. I thought that came through in the review (that's why I gave it 4 dots), but maybe not.

For me, the critical endeavor has ceased to be about forcing some kind of "new" or "earth shaking" idea unless something is really way out of left-field. I'd rather talk about what the work presents within its own sphere, though that sphere can reach in a number of different ways. Social and political commentary aren't something I'm equipped to write about, which isn't to say that I or the musicians I write about aren't social/political, but I'd rather just let those situations arise as they may. Ex: Ishmael Wadada Leo Smith's "Ten Freedom Summers" is a reflection on the Civil Rights struggle. Hard not to write about that in context. The Thing's collaboration with Neneh Cherry? A good indie-rock record that has its roots in the Afro-Scandinavian arts scene of the late '60s. Tim Berne's Snakeoil? A damn fine record of improvised music that I wouldn't feel at all out of place giving four dots to, just on the fact that it is a sterling example of its kind. There are many reasons why something is "good," and for me as a writer, I'm translating (and justifying) what that means. Of course, it's more complex than that, because if it were that simple it would result in less mental constipation and more text. But you get the gist of what I'm saying.

The rating system is a real struggle for me, because it quantifies attempts to describe something which is itself a delicate play between material and immaterial. That said, I want to feel like when I assign a dot or a half-dot, it means something, and that my take is, for the time being, complete. If the system relies on adjusting the writer's final take based on editorial re-reading, that seems to invalidate our opinions on a certain level. I mean if I were blowing out five stars right and left on mountains of hyperbole, that would be one thing. I try to avoid hyperbole and feel that my ratings are entirely reasonable. Also, I tend to review things for TMT based on the fact that I think they deserve a broad audience and are already quality, saving the lukewarm reviews for other sites. Therefore, my 4's are because I save the cream of the crop for this publication, not because of some writerly laziness. I mean, most of us are doing this on top of our own insane schedules and we want to show the world that there's good music out there.

Anyway, I hope this is a somewhat sensible window into my brain when it comes to the rating issue, and gives us both some food for thought to continue.

Cheers & Thanks,

CA

The gist of his response was that anything above 3 / 3.5 dots required ample "socio-political and cultural context apart from the music," which is often hogwash anyway.

[cih]

It seems very inconsiderate and misleading to change your ratings. Concerning the actual marks though, there might be a valid argument - given that you only review things of a certain quality in the first place - that a '3' is higher within that range of quality than a more general '3'... So if all the reviews are 3.5 - 4.5, it might suggest they are all very close together in an already higher, narrower band, rather than a '2' which ordinarily indicates ordinariness, but in a higher range might reflect a lower level of very good.

Although on reflection I think that would be confusing

The gist of his response was that anything above 3 / 3.5 dots required ample "socio-political and cultural context apart from the music,"

As though listening (or playing) was the opposite of thinking

Edited July 11, 2012 by cih

[JSngry]

The gist of his response was that anything above 3 / 3.5 dots required ample "socio-political and cultural context apart from the music," which is often hogwash anyway.

aka he is intent on using both you and the music about which you write for promoting his agenda instead of paying attention to whatever the music sets as its (and how you see it so doing).

At this point, I'm kinda like...fuck him, and fuck all those like him.

Then again, there's so many of them that it's gonna take a helluva lot of fucking to fuck them all. Normally, a helluva lot of fucking is a job for which I would readily volunteer, but in the case of megalomaniac editors and other wannabe Gestapo Gestapos, uh...not so much.

[clifford_thornton]

Thanks; at this point I'm taking a couple months off - August and September - to get re-settled, so maybe I'll have more perspective come fall. But yeah, if you're going to have rating systems, you shouldn't futz with them if the writer is generally not bullshitting.

Though the point is taken that in my case, I tend to save the best stuff for a couple of spaces, so I could be shifting the bias. Maybe I should start reviewing lame-o CDs on that site too.

[paul secor]

Or you could do what Archie Shepp did on a Leonard Feather Down Beat blindfold test back in the 60's - give everything five stars. And let the editor know that's what you're doing. That way, people will have to read your text to find out what you think about the music.

The editor probably won't go for that, but at least you'll have made your point.

edit - should have said five dots.

Edited July 11, 2012 by paul secor

[clifford_thornton]

I thought about just giving everything 1 dot and seeing what happens...