One interesting thing I've noticed from this thread is that the objections often use phrasing like "science should be ...".
This of course raises the question why ought science be done in a certain way. The justification must ultimately be normative, e.g. "so we can reduce human suffering more effectively". I think some people have this idea that science can be cleanly divorced from values, but that doesn't make sense to me; the science that we choose to do and the way we choose to do it is subtly but definitely tethered to our values. Like Hume said, "reason is and ought only to be the slave of the passions". If the carrying out of science is justified normatively - and I accept that for some this is a big "if" - then it makes sense that sometimes those norms will be challenged. Sometimes it will be debatable whether publishing a piece of knowledge is for the human good.
I'm kind of spitballing and playing devil's advocate here as I don't know the answer and haven't been convinced by what anyone has said yet. It's starting to remind me of Yudkowsky banning discussion of Roko's Basilisk. That's someone who clearly values both knowledge and ethics, and was torn over a conflict between the two.
I like Carl Sagan's "science as a candle in the dark." A candle can be used to light a house or to burn it down.
What these guidelines say to me is, "Too many people want to burn the house down right now. Our solution is to stop making candles. We'll make wax figurines instead."
But what makes candles dangerous is also what makes them useful.
To me a better solution (to those of us who use candles for good) would be to help reform the arsonists. Why are there so many of them? Did they get the same education as others? Where do they get their news? What do they believe? What resources do they have or not have access to?
Otherwise, good actors are deprived of tools that would let them, as you said, and I agree, reduce human suffering more effectively.
This of course raises the question why ought science be done in a certain way. The justification must ultimately be normative, e.g. "so we can reduce human suffering more effectively". I think some people have this idea that science can be cleanly divorced from values, but that doesn't make sense to me; the science that we choose to do and the way we choose to do it is subtly but definitely tethered to our values. Like Hume said, "reason is and ought only to be the slave of the passions". If the carrying out of science is justified normatively - and I accept that for some this is a big "if" - then it makes sense that sometimes those norms will be challenged. Sometimes it will be debatable whether publishing a piece of knowledge is for the human good.
I'm kind of spitballing and playing devil's advocate here as I don't know the answer and haven't been convinced by what anyone has said yet. It's starting to remind me of Yudkowsky banning discussion of Roko's Basilisk. That's someone who clearly values both knowledge and ethics, and was torn over a conflict between the two.