Speak Your Mind: TOP Guidelines Feedback

PAR’s transparency and openness guidelines – asking for your feedback

par-banner-1

“Speak Your Mind” is a PAR webpage feature that allows you to offer insights about big questions in public administration. The responses serve as a community forum for discussion of specific editorial contributions, and the format provides a platform for exchange of different ideas about how we think of public administration as a professional and scholarly enterprise.

DEADLINE EXTENDED TO JULY 10, 2017

Dear Members of the Public Administration Review Community,

Deanna Malatesta and I are excited to share with you the most recent draft of PAR’s transparency and openness guidelines.  With this initiative, supported by both the current and the incoming Editors-in-Chief, Public Administration Review is taking the lead in our field in moving toward more open research.  Over time, this will help enhance the quality of public administration research and improve its potential of informing practice.

In June 2016, Public Administration Review (PAR) Associate Editor Lars Tummers wrote an essay titled “Moving towards an Open Research Culture in Public Administration.”  The essay highlighted the benefits of encouraging more openness about the different stages of research projects.  It appeared on PAR’s “Speak your Mind” web page and led to useful feedback.  You can read the essay and feedback here:

Essay: https://publicadministrationreview.org/speak-your-mind-article/

Feedback: https://publicadministrationreview.org/top-guidelines/

In October 2016, PAR’s Editor-in-Chief Jim Perry signed the Center for Open Science’s Transparency and Openness Promotion Guidelines, which committed PAR to developing its own guidelines within one year’s time.  To assist in the effort, Professor Perry appointed Deanna Malatesta and myself as Transparency and Openness Promotion Editors.  We then began to develop guidelines appropriate to the journal and field.  The attached document incorporates our work to date.

Read Professor Perry’s editorial here: http://onlinelibrary.wiley.com/doi/10.1111/puar.12705/full

We invite you to read the draft guidelines, which you find in the PDF that is linked below.  And we would like to gather your feedback.  If you have any comments or suggestions, please leave a comment below or email Nick.  We value your feedback and will incorporate it in the final version of the guidelines (after discussing it with the current and incoming Editors-in-Chief).

Link to PDF: PAR transparency guidelines

Please post a comment on “Speak Your Mind” or email comments and suggestions to Nick at nicolai.petrovsky@gmail.com (or tweet at me: @pedroniquito ) by Friday, June 30, 2017.  I will acknowledge receipt of every comment.  It may take me a few days as I will be traveling for much of this time.

Deanna and I very much look forward to your input!

Thank you very much for your time.

Best,

Nick Petrovsky

Martin School of Public Policy & Administration, University of Kentucky

Public Administration Review Transparency and Openness Promotion Editor

 

Note: DEADLINE EXTENDED TO JULY 10, 2017

3 responses

  1. As an empirical researcher who has had an article replicated with the same data I used, I am in favor of open data and transparency. I see these as two separate things. To me the most important transparency issues, in order of importance are:
    1. Publication of important (quantitive) findings where the true hypothesis is that the null cannot be rejected. Or, at least, that this is the finding.
    2. Publication of replication, or repeated examination studies.
    3. Open data.

    For the first topic, I recommend that PAR (and other public administration related journals) commit to using 10-25% of their research publication space for articles of this sort. This commitment would encourage researchers to pursue topics where they have a suspicion that some policy, practice, or other potential cause/correlate has no discernable effect/correlation and to pursue publication even when the expected result is not found. Such pursuit would not be limited to the spare time of fully promoted professors.

    For the second topic:
    With respect to replication, clear benefits are: 1. Verifying that the results are as reported. 2. Validating information about the data. 3. Keeping researchers honest.

    With respect to repeated examination, I tire of responding to the question: “What is the new contribution?” There are at least three reasons why every study should be repeated quite a few times: 1. Repeated may be as close as one can get to replication. 2. Human subjects may be notably variable; even with the highest quality research methods, one study is insufficient to settle any matter about them. Further, studies may not use random sampling across the entire universe over which the researcher wishes to imply the results can be generalized. The researcher may use cautious words to account for this lack of generalizability, but in the absence of repeated studies, the results may be treated as more generalizable than they really are. As a notable example, revenue forecasting bias differs in different regions and it differs with respect to jurisdictions that face different fiscal realities. Without repeated studies, this fact may not be discovered. 3. Results may not hold up over time.

    For the third topic, while open data is, in principle, desirable, it is incumbent on its proponents to address the potentially substantial obstacles:
    1. Researchers may not have access to an appropriate platform.
    2. Junior researchers may fear the loss of control over data that is key to their tenure and promotion.
    3. Data may be restricted due to human subjects requirements.
    4. Preparation of data for publication on an open platform may be time-consuming and costly. The social sciences are generally not as well funded as the physical sciences; consequently, this barrier may be particularly hard to overcome.

  2. Thank you very much for your thoughtful comment. Issues 1 and 2 are broader questions that concern the allocation of space in the journal; as such they go beyond the current tasks of the Transparency and Openness Promotion editors. But we will discuss them with the current and incoming editors-in-chief. I certainly agree about the value of publicizing null results and promoting replication. At this point, I’ll respond to the third topic in your comment, which concerns PAR’s draft guidelines:

    1. The platforms recommended in PAR’s draft guidelines (please see page 3 of the draft guidelines) are easy to access to anyone, free of charge, and do not require a particular institutional affiliation.

    2. My personal impression that junior researchers are much more in favor of data and materials sharing. Of course, that may be a selection bias of the people I spend more time with… The loss of control over data that are still being analyzed is a very important one. PAR’s draft guidelines therefore explicitly note the option to embargo data for a set period of time, so researchers can properly complete other analyses of the data they painstakingly collected before sharing the data. The positive incentive for sharing, the data badge, can still be awarded after the embargo has ended. Please see the option page 6 of the draft guidelines, and the second question on page 10 for a fuller explanation.

    3. Human subjects protection supersedes transparency. PAR’s draft guidelines therefore explicitly state that data should only be shared if this does not conflict with the protection of human subjects: “That is, authors should first consider whether sharing data and/or research materials would carry the potential of harm to subjects, violations of their privacy, or breaches of contract (e.g. where authors do not have property rights in data they use)” (cf. page 3 of the draft guidelines). The draft guidelines state that PAR only recommends data/materials sharing in the absence of these constraints.

    4. Data preparation does indeed take some more time. But if this is built into the research process, the added time is marginal compared to the advantages both to the authors and to others. For instance, well-annotated code makes it easier for the authors to return to their analysis years later. Please see FAQ section 2.3) Sharing research materials on page 11 of the draft guidelines.

  3. I appreciate the thoughtful comments from Professor Williams. I wish I had clear resolutions for the three issues with the challenges they pose. I have some general ideas about ways forward, but I also expect that solutions will emerge over time from the collective wisdom of colleagues and new and revamped resources at our disposal. My comments below address the first two items in the list of three. Nick Petrovsky has offered a quality response to the third item.

    With respect to the first item, I can envision PAR and other public administration (PA) journals giving over some space to articles where the null cannot be rejected. I haven’t done a scan of how much attention we now give such articles in PAR, but I know some recent articles (e.g., Jennings and Rubado (2017), Preventing the Use of Deadly Force: The Relationship between Police Agency Policies and Rates of Officer-Involved Gun Deaths) have been, in part, attentive to null findings. Even my own research on public performance pay, some of it published in PAR, has emphasized findings of “no effect” (Pearce and Perry 1983). And my research is only a small portion of studies with similar findings of no effect (Perry, Engbers & Jun 2009) from the application of contingent pay in public organizations.

    I am reasonably certain that journal editors, like me, will be reluctant to give over specified percentages of pages (i.e., 10-25%) to particular categories of articles. We publish 960 pages/year, which offers lots of room for quality articles, regardless of the findings. Each article we publish, however, needs to find its way to our pages based on rigorous peer review. This will continue to be our standard. As I note above, however, I can envision us publishing more articles that fall into the category in your item 1.

    Like studies presenting null findings, I believe replications are more common than we admit. Replication has become more common, in part, because of our need to fill the pages not only of PAR, but the many public administration and policy journals (47 journals total in the latest Web of Science listing of public administration and policy journals) that have proliferated since the 1960s. In one of the areas about which I do research, public service motivation, I believe we have many replications or repeated studies. The replications are a product of a variety of factors, including the large number of scholars globally who have taken an interest in the phenomena and the existence of measurement instruments that facilitate replication.

    I agree that we need to raise fewer barriers to replications (e.g., the oft asked, “What’s new here?” question). We also need to develop standards, some of them implied in your comments, about when replications will be helpful to the advancement of our knowledge about a particular phenomenon. This might encourage more “strategic replications.” My hope is that the TOP process will facilitate development of thoughtful standards for replications. At a minimum, I anticipate the TOP guidelines will make clear to PAR readers that we are open to publishing replications. In addition, the process surrounding review and acceptance of replications will further our understanding of thoughtful standards for replications.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s