Directory of Critical Reviewers

[Philip Strothmann]

A number of members have approached us to inquire, whether a public directory of critical reviewers could not be launched as part of the FSLCI. I’m happy to inform you, that we’ve found a way to make this possible. The directory will go hand in hand with a public relaunch of our members map & profiles, which will be announced soon and result in the option to specify, which sections of your profile should be visible to whom (only you, members, public etc.). When you go now to your profile, you will see already two new tabs, which serve different purposes:

1. Critical Review: This section can be edited under the “profile” section, yet will appear more prominently as an individual navigation tab item which will also be visible for non-logged in visitors of your profile.
2. Review Rating: This menu item is meant to allow for external feedback on the reviews themselves. The idea here is to enable those, who commissioned the critical review, to have the possibility to provide feedback on the quality of the work done.

Now, in order to make this new feature as useful as possible, we would like to ensure that a sufficient amount of information is provided by those interested in doing a critical review. Hence the current fields are placeholders. To this end, we would like to have your feedback on which questions/fields should be included. Current suggestions by @martina include the following:

• Which scientific disciplines does your experience cover?
• Which industrial sectors does your expertise and experience cover?
• Please describe your expertise on environmental, technical and social aspects of production systems.
• Please describe your expertise related to Life Cycle Costing (LCC)
• Please confirm you knowledge and experience with one or several of the following ISO standards (ISO 14040 and ISO 14044, ISO 14045, ISO 14046, ISO / TS 14067)
• Please provide references with whom you have worked with :
• Please describe your experience with LCA methodology and practice, particularly in the context of LCI (including data set generation and data set review, if applicable)
• Please describe your critical review practice
• Which languages can you cover as a reviewer, that may have been used for a LC study ?

Feedback on this list of possible fields/questions is more than welcome to make this as useful as possible.

Thank you very much in advance!

Kind regards,
Philip

[Philippe Osset]

A critical reviewers directory would be great!

As far as the fields are concerned, I would use as much as possible “closed questions” (i.e. a list of check boxes for each field), and leave just one (1) empty space for “open questions” (i.e. additional elements that the reviewer would like to mention).

As far as the content is concerned, I would add ISO/TS 14071 in the list of standards with check boxes, and one empty space to mention additional specific standards (when relevant). I would also add ISO 14025 since PCR critical reviews are relevant, and future ISO 14026 and ISO/TS 14027. And, we could have PEF methodology in the list (even if it is not a standard, and not final yet)!

As far as the review rating is concerned, I am skeptical.

Of course, I listen to my customers (and to my employees), all along the year… and take decisions to make them happier, when possible (it is my interest to do so, of course !). In the case of critical reviews, I reject the fact that the customer may modify my critical review statement after my final delivery, even if he would be happier reading no critic nor limitation… In some cases, I accept to make modifications to the statement before the final delivery, when it makes sense, in others cases… I just keep my statement as it is.

Additionally, this review rating would depend on the level of expertise of the reviewed practitioner/commissioner, who may not be happy of the additional workload / costs he had to do / pay!

Then, it would be a shame if reviewers, in order to get a good review rating in our FSLCI web site, start to be less demanding in their comments!!! When hearing “Hey guy, if you keep this comment in the final statement, then I shall put a bad review rating in the FSLCI interface!”, the (weak) reviewer would answer “ok, I remove/modify my comment”… Even if every critical reviewer will say now “I am never in that situation!” (and maybe it would never occur), a suspicion might be there from third parties which would degrade the benefit of having a list of reviewers on the FSLCI Web site…

What about habilitation? In France, AFNOR has set an habilitation exam for reviewers of EPD done according to NF EN 15804+A1. Of course, a competency evaluation of critical reviewers by FSLCI (by “peers”, not by “unhappy customers”) would be highly welcomed… and difficult to set (it has a cost! Reviewers have to pay more than 1000 euros to be examined)… This recognition would be better than a review rating!

What about a charter? A charter of “critical reviews done according to FSLCI” could also be put in place, with our best practices… and the fact that the critical reviewers registered in the FSLCI directory would accept to follow this charter when they practice critical reviews… The customer can tell if the charter and ISO/TS 14071 requirements have been respected (mainly “closed” questions with indisputable answers). Once again, I would prefer this rating approach to an open review rating.

[Philip Strothmann]

Dear Philippe,

thanks for your comprehensive feedback. As for the questions, let’s have a look at @Martina’s proposal and see, which of those questions could be closed questions:

 

• Which scientific disciplines does your experience cover? -> could be closed, but we would have to come up with a list of scientific disciplines.
• Which industrial sectors does your expertise and experience cover? -> same as above, would need list of industrial sectors.
• Please describe your expertise on environmental, technical and social aspects of production systems. -> would rather be an open one
• Please describe your expertise related to Life Cycle Costing (LCC) -> would rather be an open one
• Please confirm you knowledge and experience with one or several of the following ISO standards (ISO 14040 and ISO 14044, ISO 14045, ISO 14046, ISO / TS 14067) -> this could be a very straight forward list of checkboxes
• Please provide references with whom you have worked with: -> would need to be open
• Please describe your experience with LCA methodology and practice, particularly in the context of LCI (including data set generation and data set review, if applicable) -> would also rather be an open one
• Please describe your critical review practice -> as I’m not too familiar with critical review, are there different ways that could be categorized? Otherwise this would also be open?!
• Which languages can you cover as a reviewer, that may have been used for a LC study? -> Easier to make into a list

So, here I think I would need your guidance in terms of how to make a few of these questions closed ones or whether you feel, that the open ones are not needed?

With regards to the review of reviewers, I totally see your point and have thus for the time being deactivated the review system. Coming up with an actual habilitation exam for people doing critical reviews might be tricky for us to accomplish at this point in time, but I like your idea about a charter. This is something that I can see being implemented easier and could indeed add value.

Let’s see what the others think about this.

Kind regards,
Philip

 

[Anne-Marie Boulay]

Thank you for your good input Philippe! I also agree about the review of reviewer, and I doubt it can be done in a neutral way. I agree to leave it out.

 

 

[Martina Prox]

Thanks to all for the comments and contributions. I like the charter idea also very much.

And I agree to the incresed usability and searchability reached with mostly closed questions, even if we can see, that “mostly closed” is not easy to reach, if we stick to the questions that have been proposed.

For the industrial sectors we could rely on an UN classification of Industry sectors such as http://unstats.un.org/unsd/cr/registry/regcst.asp?Cl=27

We may use the relevant ones of the top level  and go for more specific only in the manufacturing, to cover there a more specfic expertise:

• A – Agriculture, forestry and fishing
• B – Mining and quarrying
• C – Manufacturing
• D – Electricity, gas, steam and air conditioning supply
• E – Water supply; sewerage, waste management and remediation activities
• F – Construction
• G – Wholesale and retail trade; repair of motor vehicles and motorcycles
• H – Transportation and storage
• I – Accommodation and food service activities
• J – Information and communication
• K – Financial and insurance activities
• L – Real estate activities
• M – Professional, scientific and technical activities
• N – Administrative and support service activities
• O – Public administration and defence; compulsory social security
• P – Education
• Q – Human health and social work activities
• R – Arts, entertainment and recreation
• S – Other service activities
• T – Activities of households as employers; undifferentiated goods- and services-producing activities of households for own use
• U – Activities of extraterritorial organizations and bodies

 

Maybe there is a simpler approach…

[Sonia Valdivia]

Dear all,

This is a very good exchange!

Overall I agree with Philip’s and Martina’s suggestions. The idea of a charter was floating few years ago following a demand from the community and I think this would fill in the gap.

I would like to see a first draft which could perhaps be shared with a broader group.

Best regards,

Sonia

[Philip Strothmann]

Dear all,

I think the charter idea is a great one! Maybe we can launch a call for interested members that would like to work on such a charter. Would anyone of you be interested in participating/and or leading this effort?

As for the other questions, I’m happy with the suggestions. With regards to the scientific disciplines, maybe we can come up with a list of the most likely relevant ones and then leave an open field for any other? Otherwise I feel that this would become too complex?!

Kind regards,
Philip

[Marc-Andree Wolf]

Hi all,

I am a bit sorry to say that the idea of the Reviewer Directory is not that new. I had still co-initiated such while at the JRC in 2010, and since not long ago it is online, follwing basically the same logic of what you have brainstormed here: http://eplca.jrc.ec.europa.eu/ResourceDirectory/faces/reviewers/reviewerList.xhtml . For some unclear reasons ISO 14044 is not listed/working, and other schemes could be added (e.g. EN 15804), but otherwise it is already there, is suitable and works. The question is which added benefit we can bring (with our scarce resources) and how far this would help the FSLCI?

Food for thought,

Marc

[Sonia Valdivia]

Hello Marc-Andre,

Thank you for making us aware of the existing EC registry. Just browsed it and could see only SIX reviewers. I would like to know why so few have registered or been accepted in the registry?

And since your name is also there, it would be interesting to know your experience and whether you have been contacted to do any type of review or not.

Has there been any kind of disemination of this registry? As you have noticed, not many know about it, and we at FSLCI aim at building one that is well-known beyond Europe.

Looking forward to hearing from you.

Best regards,

Sonia

[Marc-Andree Wolf]

Hello Sonia,

many thanks for your check and feedback. Why there are so few reviewers registered? I assume because it was so badly announced (JRC was always good at marketing and external communication ;-)). I found it by accident (saw only Andreas registered that time).

My experience to register was easy. I had however initially only a dummy account without relevant info and have only very recently filled it all in. I got then quickly a confirmation mail (they check the entries apparently for vandalism or similar, but there is no other control). I have not been contacted based on this registry since then (at least not that I would know). I do review jobs currently, but not sure how I was identified, possibly via previous contacts.

The scope of the reviewer registry is global as well, so we would not realy add anything/much. Of course can we just ignore it and build essentially the same and announce it widely, but that does not look like a very efficient use of our resources. I think we should be more complementary, especially if an existing activity is open, non-commercial, neutral etc.

The Research group Directory that we offer is more complementary, even though there is the long-term established and much more widely used and know LCA Resources Directory at the JRC that I had developed that time and that is still active (and where the reviewer registry is actually a new subpoint), but that one is about service providers, data and software. At the same time, should we in general aim more at the LCA practitioners in companies, I think, to avoid the wrong external impression that we are geared towards researchers. But this is already a different topic.

Best wishes,

Marc

[Philip Strothmann]

Dear Mark,

thanks for your feedback. I looked at the directory and found it to be a bit hidden and not very appealing from a UI perspective. From a resource investment point of view it would be very easy for us to realize such a directory, as it technically would simply be referenced to your actual user profile – and then you could make elements of your profile visible for either the public, our members etc. Hence we’re talking about literally a day of my time to make this happen.

The other question is of course if we should recreated something that is already out there. While I’m generally in favor of collaborating instead of recreating, in this case I would lean towards doing it anyhow. After all, as Sonia highlights, we want to be the ones doing active outreach and become the place for people to find general information as well as experts and thus I would regard such a directory a valuable addition to what we’re currently offering.

Still, I think you’ve a valuable point and thus I will put the question as to whether or not we should move forward with this idea on the agenda of the next meeting of our Board of Directors later this month.  🙂

Kind regards,

Philip

[Philip Strothmann]

Dear all,

I am happy to inform you that the Board of Directors has decided to move forward with the Critical Review directory, based on the argument that the JRC Directory is probably not going to be promoted widely and might also be less accepted outside of Europe.

hence I will now look into the technical side of things. At the same time it would be great if we could find a few people who would be wiling to help come up with the Critical Review Charter – as suggested by Philippe. Would any of you like to be involved/willing to lead this effort?

Thanks in advance!

Kind regards,

Philip

[Philippe Osset]

Dear Philip, this is great ! As far as the charter is concerned, yes, I would like to be involved in the Critical Review Charter work and I am ready to contribute. Additionaly, if needed and agreed by the other contributors and FSLCI, I could also involve myself in the coordination / agregation of the contributions. I could even post a first list of items for the Charter to gather feedbacks on these items, and contributions. Best regards ! Philippe

[Martina Prox]

Dear Philippe,

Full support from my side for your offer and proposal to become the coordinator for the Charter!
Thanks and all the best, Martina

[Philip Strothmann]

Dear Philippe,

Great! Indeed this would be highly appreciated and I think it would be really good to have a first draft to start a discussion to involve also more members. So yes, if you could come up with some ideas and then coordinate the development of the charter, that would be fantastic! 🙂

Kind regards,

Philip

[Philippe Osset]

Dear all,

A very first draft of the Critical Review Charter is now available at https://fslci.org/docs/preparatory-document/

For those who want to contribute, please provide:

– a general feedback here in this discussion thread, any comment and contribution are welcomed!
– detailed comments and contributions on the items which are in it; this can be done directly in the document (editing it),
– contributions with redacted additional items; this can be done directly in the document (editing it).

I hope that we can all collaborate well, and keep the Charter at a reasonable length, while having something didactic (for CR Practitioners and CR Commissioners) and and clear enough to be applied and used as a reference.

Thank you all in advance !
Philippe

[Philip Strothmann]

Dear all,

I hope I find you well. Somehow this issue has dropped a bit off our agenda over the last few months. Unfortunately, no one commented on Philippe’s great draft charter and I didn’t have much time to advance things on the technical front. Hence, to get things going again, I’ve now introduced all profile fields on your profile. I will move the fields into a separate profile section in going forward, but you can already edit / use the fields to fill out your critical reviewer profile if you click on your profile and then go “edit” / “critical review”. I’ve taken the fields initially suggested in this threat, hence if you would like more / other fields, please let me know!

Also, I would like to encourage you again to take a look at Philippe’s draft charter: https://fslci.org/docs/preparatory-document/

I plan to go public with the directory in the next few weeks and then mark the charter as a draft. That way I hope that we can generate some valuable feedback on the charter before we declare it final 🙂

Have a great weekend!

Kind regards,
Philip

[Author]

Posts