On 8/7/17 Helen Caines send a summary of the PAC report to star-papers.
This caused a discussion that started on the star-papers list, but was continues on the council list. For the benefit of all in our groups
I have put the complete discussion in a blog.

-----------------------------------------------------------------------------------------------------
8/8/17 Mike Lisa

Dear Helen and colleagues,

Thank you for sending along the PAC report.  It is indeed very positive
about our progress so far and our plans.

However, this first recommendation that you highlight is to me
rather shocking.  I encourage everyone in STAR to read this
recommendation:

• For new results from STAR on global Lambda polarization,
measurements of fluctuations (e.g. net-proton kurtosis) in the search
for a critical point, and on the CME from the isobar run the PAC
recommends that the STAR Collaboration eliminates preliminary
release of the data and proceeds directly to journal publication.
Prior to journal submission or any public release, STAR should
present to BNL's Associate Laboratory Director the proposed
paper and sufficient (confidential) documentation for scientific
feedback on the findings, to which STAR will respond. 

It proposes to entirely override STAR's bylaws and control over
how we present/interpret/release our results.  The persons who
should control STAR presentations, interpretations and publications
are the persons who are on the author list.

If I interpret this correctly, this "recommendation" goes so far as to
identify specific physics results which may not be presented before
publication.  Note to students: NO TALKS for a year or probably two.
(Don't kid yourselves; it takes a long time to generate a publication
through the entire STAR process.)  It further mandates that the ALD
(Berndt) must approve the results and, presumably, the physics
message before publication.

This proposal to micro-manage STAR is quite bold, and I am surprised
at Berndt, John, and the PAC for "suggesting" it.

Since Helen sent this mail to the starpapers list, I am responding
here as well.  However, further discussion should probably continue
in the STAR Council.  I encourage STAR members to speak to
their Council reps.  It is unfortunate that we will not have an in-person
Council meeting until January, I believe.  However, Council should
discuss this very serious issue.

Thanks,
Mike
-----------------------------------------------------------------------------------------------------------------------------
8/8/17
Elke Aschenauer

-----------------------------------------------------------------------------------------------------------------------------

8/8/17
Aihong Tang
Hi, all :

-----------------------------------------------------------------------------------------------------------------------------
8/8/17  Renee Fatima

 
Dear Mike, all,

I am moving this discussion to the council as suggested (wisely) by Mike.

While I agree that the recommendation to remove preliminary release for some analysis is problematic, at least I can understand the motivation. They want to be sure that these high profile analyses are released only after they have been subjected to a review requisite for a publication. Now that we are essentially a one experiment collider we don't have the push back from the competing experiment and so some of the checks and balances have been removed.

In contrast, I think the request to present to the ALD is unprecedented (in my experience) and completely unreasonable. The ALD is not part of the STAR collaboration and is by definition not qualified to determine if we release a result or publish a paper. I fear that our data will not be kept confidential and instead we will be subjected to "comments" from various experiments channeled through our ALD. I think we should fight this to the bitter end. If this recommendation is foisted upon us then I think we should stick to the strict interpretation of the language - that is we "present" to the ALD and we "respond" with words, but we never unwillingly change our intentions to release or publish a result that has passed our review process.

Renee
 ---------------------------------------------------------------------------------------------------------------------------
8/8/17 Scott Wissink
Hi all,

There are some disturbing ideas being suggested here, but as with most such issues, the devil is in the details - in this case, how these ideas are actually implemented.

In general, I think the PAC makes a good point:  the treatment of results with true "discovery potential" should be different than that of  other results.  We already acknowledge this to some extent when we agree to remain silent before a Nature article is released, for example, something we don't do for Phys Rev papers.  If we're honest with ourselves, I suspect we can also all think of times when the preliminary release of a result had the effect of removing some of the urgency of getting to final, publishable results, thereby diluting the impact of the latter.  Just beneath their recommendations, the PAC noted that results on "the global Lambda polarization, kurtoses of event-by-event distributions ... and the CME from the isobar run add discovery potential to the RHIC program" and thus "a more considered and deliberate approach to release of data on these topics was sought."  Taken as a broad directive, I can't say I disagree.

I'm also surprised by the notion that "no preliminary results" translates directly into "no talks for grad students," i.e., your career is down the toilet.  Almost all neutrino oscillation results, just to pick a topic, came from double-blind analyses, which (as Carl noted) necessarily do not allow for preliminary results; yet somehow young people in the field thrived.  I honestly don't think the collaboration suffers - and neither do the careers of  students - if real game-changing results aren't revealed until they are held to a higher standard than we often apply to something with "preliminary" slapped on a figure.

All that said, I completely agree with Renee's final paragraph below.  In the same PAC discussion on topics of high impact, we find the recommendation that "STAR should work with lab management to allow critical evaluation of discovery results prior to any ... release of data."  I believe strongly that lab/BNL management should play no role in the "evaluation" of any STAR results.  When a paper is ready for publication, we make BNL management aware, as a courtesy, and so in principle the ALD (for one) could express concerns.  Were this to happen, it would be up to us to decide if we felt a response was in order.  But any involvement of lab management beyond this would be, I feel, a violation of the well established division of responsibilities between STAR and BNL.

Sorry for bending your ear,
Scott

------------------------------------------------------------------------
8/8/17  Rosi Reed

I would also want to express my misgivings with regards to this point.
Our bylaws describe the manner in which we decide to release a result,
preliminary or otherwise, and no where in this document is there a
statement that says a non-STAR member can interfere.  If we as a
collaboration wish to change our publication policy such that some
results go straight to publication, that is a debate we can have.  I
do not think this change should be imposed from outside.  I also do
not like the precedent.

While it is true that not all results are created equal, I do not see
any reason that these particular results need to have the preliminary
stage removed.  In other collaborations I have belonged to, there were
discussions ad nauseam about how preliminary results allow people to
relax and not get things published, despite all the evidence gathered
that indicate a painful publication policy coupled with the short time
scale of most post-docs and students was the real culprit.

I do believe this would harm the careers of young people who are
working on this topic, our field is very different than neutrinos and
so we can't compare them.  I would agree with Elke, Mike and others in
this respect.  If you take two students with equal abilities, and one
of them has a preliminary result, that student will be able to give
talks and start to gain recognition for her skills.  The other student
will not.....  Or they can speak on other peoples' results, which is
rarely satisfactory and rather difficult for a new student, and this
can sometimes lead to conflict.  I also have a hard time believing
that this student would be the one who is allowed to give talks on the
topic after publication, as many senior people would be eager to
showcase the new (and much, much waited for) result, assuming the
student or postdoc hasn't finished their contract.

I understand the worry that we are the only running experiment right
now.  But the only way to vet results in that case is to have
independent analyses which both yield the same result (ok, we all know
how this can go wrong), I don't think someone who was not involved
with the data from day one can really say much.

Perhaps we should have a council phone meeting?  I'm unsure what the
procedure is in this regard.

------------------------------------------------------------------------
8/10/17 Zhangbu

Dear Mike, Rosi, Carl, Scott, Renee and ALL:

We have carefully read all the comments and suggestions you provided related to

the PAC recommendation on three high-profile publications.

We really appreciate that the council be actively engaged in this matter, 
and constructive in figuring out the best approach toward publication 
in the future. 

We would like to give some background which led to that final recommendation.

During the verbal close-out at the end of PAC meeting in June, it was stated 
(not the exact words): several high-profile results are so important to the 
field and STAR is the only experiment at RHIC that the PAC recommends 

BNL ALD to convene an external panel to evaluate those outcomes

before STAR releases the preliminary results for public presentation.

We felt that this is directly against the core foundation of what a scientific

collaboration is, and will not accept the recommendation.

After a few heated exchanges between co-spokesperson, PAC chair and ALD, 
Helen and I acknowledged that: 

1)       STAR is the only experiment able to obtain those results in any reasonable 
future;

2)       Those outcomes from STAR are vital to the field;

3)       Results with discovery potential should be in the category of “extraordinary claims 
require extraordinary scrutiny” and there is no reason that they should not directly 
go to publication. Imagine what happens if J/Psi, Higgs, Gravity Wave discoveries 
were all preliminary results before their publications. 

4)       STAR has a few preliminary results which are lagging behind for years without 
journal publications (noted by Scott and Carl); there will be concerned that the 
timespan leading to publication is much longer than that required for preliminary results.

(However, those topics chosen have been quite thoroughly studied in the BES-I and 
hopefully we will sail through quickly. )

5)       PAC has a good intent in helping STAR with its credibility and not hostile toward STAR;

6)       The high-profile results mentioned (CME, Kurtosis, and GHP) are the physics arguments 
we used for the next three runs. We do not want to give a “blank check” on everything 
that are important and should be without preliminary release. Therefore, we should be 
very specific on topics;

7)       As mentioned by Carl, a double-blind analysis suggested by PAC, and acknowledged to be 
a reasonable approach STAR should consider, by design, will eliminate the preliminary stage;

8)       We request not to have external panel but provide sufficient material to ALD along with the 
announcement of the paper for feedback and not for decision (as noted by Renee); 
in fact, there have been a few cases when questions about our papers were raised through 
ALD before paper submission, and we had managed to deal with those questions. 

9)       Scientific collaborations should be open and independent in determining how and what 
data should be published even though we were reminded that the fine print we signed 
on User Agreement is that BNL owns the data. 

Helen and I considered that an external panel is directly against STAR publication policy, but 
the proposed approach is a management procedure and not against publication policy.

We therefore agreed to the language in the final PAC recommendation.

We also hope that we will never get to the point of invoking exact language interpretation 
of what it means about “evaluation” and “feedback”.  

Sincerely, 

Zhangbu and Helen 

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
8/13/ From Mike Lisa
Hi Scott,

I disagree with much of your mail, but let me correct just a factual error.
 

We already acknowledge this to some extent when we agree to remain silent before a Nature article is released, for example, something we don't do for Phys Rev papers.  

We do not "remain silent" at scientific conferences, where the results can and should be discussed by the community as part of the scientific process.  Rather, we do not speak to media outlets, at the demand of the journal itself.  The question at hand is about preliminary results at conferences.  The Lambda results that have just been published in Nature were presented several times in preliminary form, and this *greatly* refined and solidified these results.

If we're honest with ourselves, I suspect we can also all think of times when the preliminary release of a result had the effect of removing some of the urgency of getting to final, publishable results,

I'm sure we can think of such cases.  I don't know whether you are thinking of the Lambda paper, but since you have just referred it, I want to make very clear that this is not such a case.  We went from initial observation of a novel phenomenon to being in print in Nature in less than two years.  Indeed, it would have been 18 months if we'd published the initial PRL manuscript.  If anyone is wondering whether this timescale reflects a lack of "urgency to publish" in an experiment like STAR, I'd be happy to talk to them.

Thanks,
Mike