The concern is that there is some risk (far from a certainty, but some risk) that AI becomes a much more powerful version of such a person, due to getting something wrong about its very complex training process.

six-month period between November of 52 and April of 53 where we unlocked first the power of the nucleus because we could fuse hydrogen and the other thing we were able to do was uh figure out the threedimensional structure of nucleic acid in the form of the double helix

Are we wise to have built these weapons which now threaten us all?

Could social systems be finally reprogrammed, at long last, ‘as if peoplemattered’ [ 8]?

But none of these projects ever comes anywhere near realization, and none of the three super-states ever gains a significant lead on the others. What is more remarkable is that all three powers already possess, in the atomic bomb, a weapon far more powerful than any that their present researches are likely to discover. Although the Party, according to its habit, claims the invention for itself, atomic bombs first appeared as early as the nineteen-forties, and were first used on a large scale about ten years later. A

But if the technological Singularity can happen, it will. Even if all the governments of the world were to understand the “threat” and be in deadly fear of it, progress toward the goal would continue. The competitive advantage – economic, military, even artistic – of every advance in automation is so compelling that forbidding such things merely assures that someone else will get them first.