Friday, January 5, 2007

Scoring

Following are the scoring rules for Fantasy IR.

You build a team of five academics. You may have two tenured, and three non-tenured/tenure-track academics on your team. If a non-tenured academic receives tenure while on your team, you may hold onto her, but if you cut her from your team, you must replace her with a non-tenured academic.

There are three categories of points in Fantasy IR, as follows below. Please feel free to discuss this scoring system in the comments section.

Publications
Books
(legend: sole author/co-author/editor/chapter author)
Published by Cornell, Cambridge, or Princeton University Press during year of play
50/20/10/5
Published by another University Press during year of play
25/10/5/3
Published by a non-University Press
20/5/3/1
Reviewed in Foreign Affairs or the APSR/Perspectives
5 (7 for feature-length review)
Reviewed in the New York Times
10
Reviewed in the Washington Post, Economist, LA Times, or Wall Street Journal
7
Articles
(legend: sole author/co-author)
Published in APSR, International Security, International Studies Quarterly, or International Organization
50/20
Published in American Journal of Political Science, Journal of Conflict Resolution, Security Studies, European Journal of International Relations, World Politics, Millenium,
25/10
Published in another peer reviewed scholarly journal, Foreign Affairs, or Foreign Policy
15/5
Published in a non-scholarly journal
7/3
if the article generates responses/letters in the year of play:
in APSR, International Security, International Organization
10
in PS, Journal of Conflict Resolution, Security Studies, European Journal of International Relations, World Politics, Millenium, International Studies Quarterly
5
Op-Eds
(legend same as articles)
New York Times
10/7
Washington Post, Financial Times, Wall Street Journal, LA Times, Boston Globe, Chicago Tribune, Times of London
5/3
all other op-eds
1

Conferences
APSA Paper presented (beginning of game year)
10
APSA Panel chair
7
APSA Panel discussant
5
ISA Paper presented
5
ISA Panel chair
3
ISA Panel discussant
2
MPSA/other conference Paper presented
3
MPSA/other conference Panel chair
2
MPSA/other conference Panel discussant
1

Tenure/Chair/Hire/Position
(legend: T1/non-T1. We define "T1" as one of the top 16 schools in the William and Mary study on IR, available at http://mjtier.people.wm.edu/intlpolitics/teaching/surveyreport.pdf specifically on page 25. The schools are Harvard, Columbia, Stanford, Princeton, Chicago, Yale, Michigan, Berkeley, UCSD, Cornell, MIT, Duke, Johns Hopkins, Georgetown, Ohio State, and Minnesota. Depending on comments from the rabble, this list may change.) These are event-based on the happenings during that year - a new tenure, a new chair position, etc.
Make Tenure
50/30
New Job (accept offer)
20/10
Endowed Chair
30/15
University Dean/Provost
25/15
Department Chair
20/10
Promotion (other than tenure)
20/10
Dissertation Advisee Placed at
10/5

Bonuses
Named APSA President
100
Named ISA President
70
Grawemeyer Prize
100
Selected for a Government Job (US/foreign)
(cabinet level/subcabinet level/working level)
30/10/5

32 comments:

Anonymous said...

Great idea, though I do hate that we have to wait until July!

Anonymous said...

IS = APSR?

PS, EJIR > AJPS, JOP?

What planet are you people living on?

Anonymous said...

Why "Top 16"?

UMN homer out there?

Anonymous said...

What about a book chapter? 20/15 in major university press; 15/10 lesser press.

Cullen said...

I would add Journal of Peace Research to the same echelon as Journal of Conflict Resolution, but otherwise I think that this looks pretty good.

Anonymous said...

awesome idea, although i wish the scoring could somehow distinguish important work from "important" work.

Anonymous said...

No points for AJPS or JOP? Come on now...

FantasyIRAdmin said...

Responding to January 10, 2007 5:05 PM

...JOP we saw as an Americanist journal. Same for AJPS. What are the major streams within IR that are published in those two?

Responding to January 10, 2007 4:18 PM

...If people think JPR needs to be included we can, but I don't know if it has the same impact as JCR on the subfield writ large.

Responding to January 10, 2007 1:33 PM

...Book chapters are included - fourth slash under books

Responding to January 10, 2007 12:56 PM

...We're open to expanding this to top 20, top 25. What other schools should be on there?

Responding to January 10, 2007 12:54 PM

...IS is the dominant journal in security studies, while APSR does little work on IR relative to other subfields. PS we can drop potentially. EJIR is fundamental for constructivists, and if you're arguing that AJPS and JOP need to be up there because their authors are "underrepresented" the same argument fits for EJIR.

General comment

We GREATLY appreciate the comments. Please keep them coming, and more importantly, please spread the word about Fantasy IR. At this point, we're dealing with rule manipulation. But at some point soon, we'll want to start thinking about who wants to play.

Anonymous said...

How are we defining the IR players? Is it just by how the department lists them? What about people who are listed IR but mostly work in comparative these days (a la Fearon)? What about people who are listed as in other subfields but have substantive interests in IR?

Anonymous said...

If you want to figure out how much to value a journal article, just go to the study you cite above (the same one you are using to rate IR departments). That study lists the top 25 departments.

On page 22 of the same study you get the ranking of IR journals based on thier perceived impact on the field.

Looks like your rules are slightly over-valuing APSR and dramatically over-valuing EJIR, Millenium, and PS. You are under-valuing ISQ and AJPS, but JOP is not rated very highly, so your current rules seem about right on JOP.

Using this or some other rating system allows you to avoid being arbitrary or inadvertantly imposing your own bias. An alternative ranking can be found in recent issue of PS article by Giles and Garrand.

How do you do the draft? Who will gather the statistics? Seems time consuming. Don't you have a dissertation to write?

Thanks for doing this. It sounds like fun.

Anonymous said...

Less than 50% for co-authored pieces is below current norms in the field. Either 50% or 60% would be more accurate, with some sort of additional breakdown for additional authors. You also need to include more major IR awards (look at the ISA and APSA website).

It might be interesting to break out some different leagues (same players, but different scores), such as a "quanthead" league in which JCR would be more valuable than IS.

Anonymous said...

MIT, Oxford UP? I think those should be there with Cambridge and others. PS...come on...if you have that you better have International Interactions up there too...

Anonymous said...

1/10, 12:54 here.

Honestly, to be seen as remotely credible, AJPS and JOP need to be in there. In most (U.S.) departments, IR folks are told that publishing in those places gives them their best shot at tenure, raises, etc. (because their comparative, AmPol, and theory colleagues know and respect those journals). Most IR folks I know would rather have an AJPS than an IS for those reasons alone.

Also, a fair bit of IR *does* get published in those two journals (check out the websites of either the forthcoming April 2007 AJPS or the February 2007 JOP for examples).

For those reasons, I'd put them both (AJPS, JOP) in category II (along with WP, JCR, etc.).

Anonymous said...

Have you thought about adding citation counts?

Anonymous said...

Umm so an endowed chair gets you 30, and that is equal to getting a Cabinet level position? So if I write 3 op eds for the NYTs, I’m as equally successful as the Secretary of State? I think the whole government point value needs to be bumped up.

Moreover you're missing the two mainstay strat studies journals from the UK; International Affairs and Survival, which would probably be considered equal to, if not more prestigious than EJIR.

Anonymous said...

So a chapter in a non-university press is worth as much as an op-ed? Howsthat? You may want to distinguish between university, academic, and non-academic presses.

FantasyIRAdmin said...

Next set of responses...

1) We've moved AJPS to second level, and ISQ to first. We've dropped PS, and are considering what to do with the other journals... I don't think we're overvaluing EJIR that much, but the big question is whether to add JPR. JOP, I think we're still safe leaving that as an "other peer reviewed journal."

2) You can draft whomever you want from a department, someone like Fearon who mainly does comparative is fine. But if he's not publishing in the major journals or the major presses, his score is going to drop.

3) Per a recommendation from an interested reader (not a commenter, although the comment was useful too) we're considering citation counts. How do we measure this? Remember, we want relative simplicity in scoring, at least in the first year. Is there a citation index we should use? How frequently is it updated?

4) The other point values are still in flux and we're considering whether to change them or not. Your comments will be a key indicator as to whether we do.

Thanks, and keep the comments coming.

Anonymous said...

Let's not be silly - almost no major security scholars publish in AJPS, much less JOP.

Anonymous said...

On citation counts the simplest thing to use is google scholar or SSCI. But this would get you away from your focus of production during one "season." After all, articles and books often don't get cited until long after they are published. For example, dead people continue to get cited in the current season. Is this really what you want to measure?

In terms of the amount of time it will take for administrators to run this league, citations will be a major time commitment.

International Affairs and Survival are not important journals in US IR. Just look at Giles' journal ratings or Garand's journal ratings. You won't find either of those journals among your first two tiers. Your current standards seem about right in terms of which journals to put in which of your three tiers.

Agree with earlier comment on co-authorship. Disciplinary norms give you much more than 50% value for a co-authored paper. According to Gary King there is no difference at all: A co-authored article should count the same as single authored article. I think this is not how most people feel about the issue, but King is right that you should get more than half credit for a co-authored piece.

Becoming secretary of state does not advance our knowledge of IR. In fact, given the performance of the current occupant (who does have a phd in political science), you might want to reduce points.

Your response on how to define an IR Player may need more thought. The comment did not imply that Fearon was publishing in bad outlets, it implied that he was not doing IR, but some other branch of political science. Hypothetical: If Mearsheimer writes an article about the voting behavior of African American members of the US Congress on domestic policy issues and that article gets published in APSR, does he get credit in the IR Fantasy league? See the point? Are eligible players defined ahead of time by you (and you provide a list to draft from) or do you look at the substance of the articles they are writing and decide whether they are IR articles or not? The former would be much easier from an Admin point of view.

FantasyIRAdmin said...

New drafting rule:

If you want to draft someone (professor or even if you're insane, grad student) she or he must be institutionally affiliated with the IR subfield in a political science/government department, or the IR element of a public affairs/international affairs school. So Steve Walt, while not affiliated with Harvard GOVT, would have to be identified with the IR element of KSG. On the Mearsheimer point, if John Mearsheimer can get a piece published on voting patterns in a 50-point journal, more power to him. I doubt it'll happen, but it's worth considering that IR is a relatively catholic discipline.

FantasyIRAdmin said...

Responding to January 11, 2007 1:51 PM

This is a study of IR writ large, not just security studies. If other IR scholars (non-security) write in AJPS, it should be in. Anyone have any evidence pro or con AJPS, JOP, etc?

Anonymous said...

This is a hysterical idea (modern sense of the word)-- I'm looking forward to playing.

Observation: the weights for conference participation are very high relative to publications.

10 points for an edited book from the big 3, 10 points for a coauthored book out of the big 3, 10 points for a coauthored peer-reviewed article in a journal ranked 5-10... and 10 points for an APSA presentation, or *12* points for a chair-discussant APSA twofer, or 10 for an APSA discussant/ ISA paper combo. Those last 3 just shouldn't be in the same ballpark as the publication ones.

Anonymous said...

This league will reward those with good connections more than it will those with a good nose for IR.

In a fantasy football league, you're drafting somebody based on past performance and an expectation about behavior in an unknowable future.

In this league, it's likely that some people will have "insider information" about things in the pipeline to which others do not have access. With forthcoming publications, the future is often knowable but also often private information.

I know a few people right now who have contracts with major university presses. Last APSA, it would have been very smart to draft those people, but not everybody had access to the same information. It's not like book contracts are publicly advertised.

Nothing you can do about this, but the winner of this league will be the person who knows the most about what's forthcoming, not what's already been published.

FantasyIRAdmin said...

Responding to January 11, 2007 4:55 PM

It does reward inside information. But inside information isn't always enough to determine the margin of victory. Not to mention, you'd have to have a HELL of a lot of inside information on every single player. But perhaps you draft people you know about. Perhaps you don't. Remember, you only get a team of 5 academics. Two tenured, three untenured. What if you can't get your person in the draft? Then you're, so to speak, hosed.

But the points attribution system also rewards knowing about IR. There are two necessary conditions that are jointly sufficient for getting points in this game. First is that something has to happen in your academic's career. Second is that you have to know about it and report it. So you need to know who's got the articles in the new issues of the journals. You have to know the new books coming out. Especially if you want to use the free agent system to pick up a hot commodity on the market.

Are there information asymmetries in the game? Absolutely. That's part of the fun. And remember, points are upon publication, not contract.

Responding to January 11, 2007 4:39 PM

We'll strongly consider how to balance these two. The points structure in general is still in flux, and if you have more detailed suggestions than would fit in a comment, feel free to email us at irfantasyleague@gmail.com

Anonymous said...

Could you tell us your id so that we don't draft you? You will be too busy adjudicating the disputes and revising stuff to get published :)
Just joking.

Anonymous said...

What about journal editorships? I know those don't change hands very often, but it seems like it should be worth something.

Anonymous said...

On AJPS: Here are my own subjective takes on what they've done in "IR" in the past year. (I included one or two IPE papers, but none on civil wars and other "IR-ish" domestic politics topics).

July 2006:

A Unified Theory and Test of Extended Immediate Deterrence -- Curtis S. Signorino, Ahmer Tarar

Locke on the Moral Basis of International Relations -- Lee Ward

October 2006:

Institutional Inconsistency and Political Instability: Polity Duration, 1800–2000 -- Scott Gates, Håvard Hegre, Mark P. Jones, Håvard Strand

Lies, Defection, and the Pattern of International Cooperation -- Milan Svolik

January 2007:

The (Other) Liberal Peace: Capitalism, Common Interests, Development and Democracy -- Erik Gartzke

April 2007:

Ideology and Voter Preferences as Determinants of Financial Globalization -- Dennis Quinn and Maria Toyoda

The Propaganda of the Deed: Terrorism, Counterterrorism, and Mobilization -- Ethan Bueno de Mesquita and Eric Dickson

Reputation and Interstate Conflict (Of Friends and Foes) -- Mark Crescenzi

Kant Revisited: Liberalism, Democracy, and Conflict -- Vesna Danilovic and Joe Clare

Sanctions, Signals, and Militarized Conflict -- David Lektzian and Christopher Sprecher

Anonymous said...

JOP IR Feb 2007
What Is the Enemy of My Enemy? Causes and Consequences of Imbalanced International Relations, 1816-2001
Zeev Maoz, University of California, Davis
Lesley G. Terris, Tel Aviv University
Ranan D. Kuperman, University of Haifa
Ilan Talmud, University of Haifa

What Attracts Foreign Investors? An Examination of Human Rights and Foreign Direct Investment
Shannon Lindsey Blanton, University of Memphis
Robert G. Blanton, University of Memphis

Crisis, Charisma, and Consequences: Evidence from the 2004 U.S. Presidential Election
Jennifer L. Merolla, Claremont Graduate University
Jennifer M. Ramos, University of California, Davis
Elizabeth J. Zechmeister, University of California, Davis


Nov 2006
Rethinking Third Party Interventions into Civil Wars: An Actor-Centric Approach
Michael G. Findley, University of Illinois at Urbana-Champaign
Tze Kwang Teo, University of Illinois at Urbana-Champaign

Aug 2006
An Illusionary or Elusive Relationship? The Arab-Israel Conflict and Repression in the Middle East
James H. Lebovic, The George Washington University
William R. Thompson, Indiana University

The Human Rights Peace: How the Respect for Human Rights at Home Leads to Peace Abroad
David Sobek, Louisiana State University
M. Rodwan Abouharb, Louisiana State University
Christopher G. Ingram, Louisiana State University

Making Sense of Issues through Media Frames: Understanding the Kosovo Crisis
Adam J. Berinsky, Massachusetts Institute of Technology
Donald R. Kinder, University of Michigan

May 2006
Credibility in Compliance and Punishment: Leader Specific Punishments and Credibility
Fiona McGillivray, New York University
Alastair Smith, New York University

Explaining Commitment: States and the Convention against Torture
Jay Goodliffe, Brigham Young University
Darren G. Hawkins, Brigham Young University

FantasyIRAdmin said...

We're currently reworking some of the point total systems. The journals and books will probably increase quite a bit.

A question for the group, however: what other things out there should be counted? We feel as if we've got a good cut at a comprehensive scoring system, but what are we missing?

And here's a real topic for debate that might get its own thread... who's the #1 draft pick?

Anonymous said...

Like I'm giving away my number one draft pick. Humph!

Anonymous said...

My vote is for Henry Morgenthau.

Anonymous said...

Some questions about HOW the game works and how to make it work, rather than rules:

1) Cheat sheets? Every other fantasy "sport" ranks "players" via cheat sheet? Is this a consideration for the league? Anyone out there? Could be fun...

2) Is there a program that will run an automated scoring format, will the league's "commish" manage the scoring, or will each player be required to manage their own scoring?

3) How often will scores, standings, and other updates be posted?

I know these are intellectually less stimulating, but worth opening to a broader group if they have not already been established.

Otherwise, and nitpicking on scoring aside, I am totally stoked! Let's play!