Como avaliar pesquisa por Alan Galfend, Peter Green e Vivay Nair

[Renato Martins Assuncao <assuncao@est.ufmg.br>]

Grande coincidencia, o Institute of Mathematical Statistics iniciou em seu ultimo 
numero (http://www.imstat.org/bulletin/Bulletin36_9.pdf)
uma discussao sobre como avaliar a pesquisa estatistica. Isto e' feito tendo
em mente os pesquisadores jovens que estao em busca de  estabilidade (tenure),
que geralmente e' obtida apos alguns anos (de 5 a 7 anos) de trabalho  muito duro
e apos avaliacao de seu desempenho como pesquisadores. 

Sao tres depoimentos, de  Alan  Galfend,  Peter Green e Vivay Nair respondendo a
estas questoes:
1. Should junior faculty members go for quality at the cost of quantity for
publications?
2. Should they refrain from too much interdisciplinary work, which often
leads to publication in non-statistical journals?
3. Should junior faculty write singly authored papers to show independence?
4. Is it risky for an assistant professor to publish only co-authored papers from
interdisciplinary work?

Aqui vai para quem nao consegue acesso as publicacoes do IMS.
Renato
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
IMS Bulletin - Volume 36 - Issue 9 - November 2007
How do we evaluate research?

Web Address: http://www.imstat.org/bulletin/Bulletin36_9.pdf

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Advising junior faculty on research
IMS Bulletin Editor Xuming He writes:

If there is one thing that all statisticians agree on today, 
it is the recognition that statistics has evolved from a 
mathematical sub-field to a cross-cutting scientific discipline.
Along with this exciting evolution come new challenges to a new 
generation of academic statisticians. Junior faculty members in 
the statistics departments or in mathematical sciences departments 
are often unsure how their co-authored papers and/or their interdisciplinary 
research will be evaluated for their promotion. Sometimes they
receive conflicting advice from senior members of our profession 
who have developed their career in a very different environment.

To stimulate discussion and debate that may eventually help build 
consensus in our profession and help our junior faculty choose the 
best course for their career development in today?s competitive 
academic world, the Bulletin sought opinions from senior mentors 
and department chairs on how they would advise junior faculty on the
following questions. 

1. Should junior faculty members go
for quality at the cost of quantity for
publications?

2. Should they refrain from too much
interdisciplinary work, which often
leads to publication in non-statistical
journals?

3. Should junior faculty write singlyauthored
papers to show independence?

4. Is it risky for an assistant professor to
publish only co-authored papers from
interdisciplinary work?

In this issue, you can read candid opinions from three distinguished doyens,
Professors Alan Gelfand (Duke University, USA), Peter Green (University of Bristol,
UK) and Vijay Nair (University of Michigan, USA). Their advice to junior 
faculty is too rich to be summarized into a few statements, so you can 
turn to page 4 to read their articles in full.

However, I am sure that we have not touched upon all the important issues 
facing our junior faculty today, and you may have valuable opinions different 
from those of the courageous trio featured in this issue. I wish to use this 
opportunity to encourage all of you to think about these issues and write to 
us (email bulletin@imstat.org) with questions, comments, or opinions. In the 
coming issues of the Bulletin, we would like to publish what you have to say!

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Alan E. Gelfand: Duke University

I write as Chair of the new Department of
Statistical Science here at Duke University. Indeed,
our name reflects our vision of modern
statistical work ? we are interdisciplinary
scientists. Moreover, it reflects a model
that we have been following here at Duke
almost from the inception of the Institute
of Statistics and Decision Sciences (ISDS),
our former name. Training PhD students
to collaborate with researchers in other
fields is the cornerstone of our program.
Mentoring such research defines life around
here. Regularly, research teams meet to
?brainstorm? challenging problems. Such
teams usually consist of faculty from two or
three departments with some graduate students
and postdocs. Our program attempts
to equip all of our students not only with
a strong theoretical foundation, but also
with comfort in stochastic modeling, with
facility for statistical computing, and with
solid data analysis skills. We view these
tools as essential in training statisticians for
the 21st century. At a university like Duke,
where, institutionally, research collaboration
is strongly encouraged, statistics is the most
natural of interdisciplinary players.

We recognize that there are issues in
all of this that become critical to young
interdisciplinary statistical scientists along
their career paths. How does one judge
interdisciplinary work? How does one
evaluate individual contributions in the
multi-author papers that naturally emerge
from team research? How does one assess
the independence of a junior researcher, the
ability to create an independent and strong
research agenda?

I cannot lay out explicit answers to these
questions. They obviously depend upon the
field of application, the project, the institutions
involved. But just as funding agencies
are putting their money where their mouths
are, in terms of supporting interdisciplinary
work, so must departments, schools, and
universities revise their evaluation criteria
with regard to such work. And, with regard
to giving advice at the individual level, it
is my feeling that the collaborations must
be consequential enough, the problems
must be challenging enough, so that novel
statistical methodology emerges during the
course of the projects. In this way, theoretical
and methodological contributions for
top-notch statistical journals will emerge, as
well as papers employing the methodology
in the motivating application for the subject
matter area: a win-win situation.

In the sort of team research that is
envisioned, the different players contribute
in different ways. So, what becomes critical
is to inform the junior researcher as to how
the various sorts of contributions will be
valued; to assess the young researcher?s contributions
to each multi-authored paper; to
ascertain the ability of the young researcher
to contribute creatively and innovatively to
projects along the foregoing dimensions;
and then, to make decisions regarding
tenure and promotion consistent with these
evaluation criteria. And, of course, not all
work spans the same set of dimensions,
or contributes on all dimensions. There
has to be expectation of and tolerance for
a range of vita profiles at the tenure and
promotion level; a department need not be
homogeneous in this regard; rather, it needs
to accommodate diversity in profiles.

Finally, there are possibly legitimate
concerns regarding the future of our
discipline. Can statistics survive in the
sorts of interdisciplinary environments we
find ourselves in? Who will provide the
methodological breakthroughs? Who will
ensure the rigor? How will we continue to
draw students to our graduate programs?
Actually, on a personal level, I see no
problems. All of the first- and secondtier
statistics journals are receiving more
papers than they can publish, even with
the proliferation of new journals. There
will always be diversity in the way that
researchers make contributions. Individuals
will identify their best way to contribute.
Applications to graduate programs are
not down; in fact, here, we are seeing an
increase in domestic applications, perhaps
an indication of the increasing attractiveness
of our field (apart from the good job
market!). Indeed, statistics is, arguably, a
more exciting field than ever. It now enjoys
a vitality and a relevance that was absent
20, 30, 40 years ago. Moreover, we can not
look back. Modern scientific work requires
that statisticians be incorporated into the
entire enterprise. Statisticians become richer
scientists in the process. There will be fewer
highly specialized theoretical statisticians,
but there will be more statisticians with the
skill set and the passion to pursue the sorts
of work described above. For me, this is
an exciting prognosis. Integrated scientific
work is the track being traveled now and we
must be on board.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Peter Green: University of Bristol, UK

The questions you posed us may be taking a
very US-centric view of career development.
In other parts of the world, procedures may
be different ? for example, I think that here
in the UK, the process is less formulaic,
and more flexible and adaptive, so that it
is easier for us to reward people who have atypical career patterns.
Also, we should not confuse ?co-authored? with ?interdisciplinary?
work.

Quality vs. quantity: I always encourage ambitious and creative
junior researchers to go for quality not quantity. Seek to change
the subject, not just to make incremental improvements. There
is far too much routine work clogging up the journals, that has
no impact on the subject, and little impact on the author?s career!
Very early on in a career, I can see the point of relaxing standards a
little ? you do need to get practised in writing and publishing, and
get some articles into print early on.

Interdisciplinary vs. mainstream: I like to do both, and I
encourage that. It?s important if you do interdisciplinary work not
to get drawn into a ?service? role ? seek to do research that advances
both subjects: in my experience that does get properly recognized.
Statistics has always advanced through cross-fertilisation between
different disciplines, and it?s important to maintain this. Often, a
research project that is genuinely interdisciplinary can lead both to
a joint publication in a substantive science journal and a solo article
in the mainstream statistical literature. But it?s a good idea also to
do separate mainstream theory or methodology: really good applied
work takes a lot of time, and you need to be productive.

Solo vs. joint: joint work with senior (and later, junior) coauthors
is fun, rewarding and productive ? often you get more
than the sum of the two individuals. But it is important early in a
career to publish enough work where your individual contribution
is indisputable. This may either take the form of solo articles, or
interdisciplinary ones in which you are the only statistician. Then
the reader can judge your contribution without ?inside knowledge?
from the other authors
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Vijay Nair: University of Michigan, Ann Arbor

Let me begin with some general remarks
about the promotion and tenure (P&T)
review process. At Michigan, as in most
other places, we look for several key elements:

a) whether the candidate has developed
an independent research program outside of thesis work, 
b) how original, innovative and deep the research is, 
c) is it published in good quality journals, 
d) has the candidate developed a niche for
her/himself and established a reputation, and so on.

Of course, there are no concrete ways (such as n papers in
journals x, y, and z) to measure quality. Some departments used to
assign weights to different journals and multi-authored papers, but
thankfully this is no longer common. You simply have to look at
the entire body of work and make a holistic assessment.

Unfortunately, the subjective nature of this process causes a lot
of anxiety among some junior faculty. As students, we were all clear
about what was needed to succeed. It is frustrating when there are
no longer any concrete guidelines. The lack of consensus among
senior colleagues doesn?t help the situation.

In answer to some of the specific questions raised by the
Bulletin, I offer the following personal opinions.
Quality versus quantity: Quality is clearly the ultimate measure.
However, there must be enough of it (quantity) to make sure the
reviewers can say, yes, the candidate has established an independent
research program (outside of her/his thesis work); shows originality,
innovation, and depth; and is being recognized nationally for her/
his work. It is not likely that three or four papers will be enough to
make this case.

I know of one successful P&T case in Statistics (not from
Michigan) with seven papers. I thought three of these papers were
real gems and three others were very good. If I recall correctly, it
was an early promotion case. At the other extreme, I have seen
mediocre cases with 25 to 30 papers, all of which were just nibbling
around the edges of existing work.

Working on too many topics and jumping around will not help
the cause. It is difficult to assess the impact of one?s work or establish
name recognition. It is important to be selective and strategic.
Interdisciplinary work and publication in non-statistical journals:
I feel the phrase ?interdisciplinary research? is used loosely to
mean many things, ranging from routine consulting and innovative
use of existing methods all the way to very original work.
Routine consulting (designing experiments and/or doing data
analysis) will not count much even if it leads to publications. Papers
based on innovative use of statistical methods alone will also not be
enough.

It is tough to define what makes ?good quality? interdisciplinary
research. From a statistics department?s point of view, I believe it
has to advance both statistics and the scientific discipline. It has to
lead to the development of new statistical methodology or theory,
or make some other contribution. It doesn?t necessarily have to be
published in statistics journals (many such papers are published in
PNAS, Science, Nature and the like, although the statistics part usually
gets short-changed in these outlets).

The bottom line is that there must be statistical innovation.
One can, and should, get tenured based on such work alone.
I?m happy to say that this type of interdisciplinary work is being
highly valued in our profession these days, and we have examples
of faculty who have been tenured, including in our department at
Michigan.

Co-authored papers versus independence: Does a junior faculty
member need to write singly-authored papers to show independence?
My answer is NO. (I know some people feel differently, but
fortunately that group is a minority, and shrinking.) Independence
is indeed important. It is especially important that junior faculty
cut their umbilical cord with their PhD advisor very early. But
there are many ways of establishing independence:
i) working with several different co-authors,
ii) having co-authors who are also junior,
iii) a common theme across several co-authored papers where the
theme is clearly attributable to the candidate,
iv) publishing with graduate students, etc.
On the other hand, publishing a number of papers with the same
senior co-author (even if the person is not your advisor) will raise
questions.

Advice to junior faculty: First and most important, make sure
to enjoy what you are doing. You are in this for the long haul. Of
course, the pressures of getting tenure may be overwhelming at
times, but you can only do good work if you enjoy coming to work
every day.

Second, take charge! Make sure that you do everything you
can to help your own career. This means getting all the information
you need to be successful. Talk to your chair regularly, get to
know all senior faculty members, find out who the key players in
the department are (remember George Orwell?s Animal Farm: all
of us are equal but some are more equal than others), participate
actively in department activities and seminars, ask questions and
make suggestions, make sure that people know you are good, be
assertive (don?t be shy but don?t get a reputation of being aggressive
or an operator), be a good citizen but don?t overload yourself with
committee work. Don?t spend an inordinate amount of time on
teaching. Remember that teaching is important but being an excellent
teacher alone is not going to get you tenure (as we say, it?s a
necessary condition but not sufficient).

Finally, network, network?and network. You need to make
sure that people (on the outside) know of you and your work.
We are all very busy and don?t have enough time to follow other
people?s work. So you need to make sure that senior people in your
area know of you and your work. Send them copies of your preprints
and ask for advice. E-mail them before
a conference and ask if you can find some
time to meet with them. This will pay off in
many ways: getting a sympathetic referee,
getting invited to conferences, getting onto
editorial boards, and most importantly, having
people who know your work when they
are asked to write external review letters.