The right time to worry about a potentially serious problemfor humanity depends not on when the problem will occur, but on how much time isneeded to devise and implement a solution that avoids the risk.
This is very interesting to me and was worth highlighting.
On September 11, 1933, renowned physicist Ernest Rutherford stated, with utterconfidence: “Anyone who expects a source of power in the transformation of these atomsis talking moonshine.” On September 12, 1933, physicist Leo Szilard invented the neutron-induced nuclear chain reaction.
Save this. Good thing to bring up.
A being’s suffering gives any sufficiently capable agent a pro tanto reason to reduce it, because suffering is a real negative condition for the subject undergoing it. That reason can be outweighed by other reasons, but it is not created by anyone’s approval.
Double
phenomenology
the study of lived experience and conscious, focusing on intentionality, bracketing, and the life-world
blithely
doing something with happy and casual carelessness, often without worrying about the results or noticing problems
pluralistic value alignment
Pluralistic value alignment is an approach in artificial intelligence that shapes models to respect diverse, conflicting human preferences and values rather than forcing a single average consensus
value imposition
Value imposition occurs when a professional directly or indirectly influences a client to adopt their own personal beliefs, attitudes, values, and behaviors.
their hopes and expectations
I think this is talking about the model itself, rather than the person living in poverty. The model will "adapt" to the users preferences. That is how I am understanding this part at least.
However, the question then becomes what data or feedback to train the agent on
Wow this is what I was talking about. Crazy
increasingly prohibits humans from evaluating whether each action is performed in a responsible or ethical manner
This is insane to me how humans can no longer tell (sometimes) how a model is truly acting under the hood. It is quite scary and points to how there are underlying safety measures that need to be put in place. I think a good strategy would be embedding at a lower level, such as in the training data it self. But my understanding with this is very low. As you can most likely tell.
While foresight will not reduce risk if no effectiveaction is available
Yes I fully agree and was saying this to my mother about 5 minutes ago before I got to this sentence
There is a flash of value, followed by perpet-ual dusk or darkness
I believe this is what the world could head down if we suddenly invent AGI. The intelligence explosion would be huge, but we wouldn't have the means to be able to handle it. Reading groups like these are important to get the ball rolling to spark conversation about potential futures, but we must eventually (once we have enough context) be able to put in place safeguards and the infrastructure needed to move away from this path of a flash of value followed by perpetual darkness.
An example of thiskind is a scenario in which machine intelligence replacesbiological intelligence but the machines are constructedin such a way that they lack consciousness (in the senseof phenomenal experience)
This is what is happening now! The way these frontier AI models are built is fundamentally flawed if we are looking to achieve consciousness. I often like referencing Jeff Hawkins. We first must understand the brain before trying to replicate it. These LLMS as we most all know, are just intelligent autofill. Sure they can replace the manual labor of humans tenfold, but they will not and cannot invent the warpdrive.
The permanent destruction of humanity’s opportunityto attain technological maturity is a prima facie enor-mous loss, because the capabilities of a technologicallymature civilisation could be used to produce out-comes that would plausibly be of great value, such asastronomical numbers of extremely long and fulfillinglives
Is the goal of civilization to create long and fulfilling lives? I guess so.
we find that the expected value of reducingexistential risk by a mere one billionth of one billionth ofone percentage point is worth a hundred billion times asmuch as a billion human lives.
What?