Forum adverts like this one are shown to any user who is not logged in. Join us by filling out a tiny 3 field form and you will get your own, free, dakka user account which gives a good range of benefits to you: No adverts like this in the forums anymore. Times and dates in your local timezone. Full tracking of what you have read so you can skip to your first unread post, easily see what has changed since you last logged in, and easily see what is new at a glance. Email notifications for threads you want to watch closely. Being a part of the oldest wargaming community on the net. If you are already a member then feel free to login now.

I had never heard of this before, and I find it fascinating and mind bending....

http://www.slate.com/articles/technology/bitwise/2014/07/roko_s_basilisk_the_most_terrifying_thought_experiment_of_all_time.html

Here is a brief blurb....

which makes Roko’s Basilisk quite literally forbidden knowledge. I was going to compare it to H. P. Lovecraft’s horror stories in which a man discovers the forbidden Truth about the World, unleashes Cthulhu, and goes insane, but then I found that Yudkowsky had already done it for me, by comparing the Roko’s Basilisk thought experiment to the Necronomicon, Lovecraft’s fabled tome of evil knowledge and demonic spells.

I will now present to you Roko's Basilisk.....

Roko postulated a thought experiment: What if, in the future, a somewhat malevolent AI were to come about and punish those who did not do its bidding? What if there were a way (and I will explain how) for this AI to punish people today who are not helping it come into existence later? In that case, weren’t the readers of LessWrong right then being given the choice of either helping that evil AI come into existence or being condemned to suffer?

Choose, but choose wisely.....

So... The plot of Terminator?

 LordofHats wrote:
So... The plot of Terminator?

Ultron?

 whembly wrote:

 LordofHats wrote:
So... The plot of Terminator?

Ultron?

An Ultron-Terminator hybrid?

Now that's frightening.

I am dissapointed by the lack of basilisks.

At first I was like, "I can see why that's a troubling thought exercise, but that's seriously overreacting."

Then I thought about it, and realized the ramifications of what it's ACTUALLY doing. That's subtly horrifying, and Slate and the OP are doing the world a disservice by repeating it.

Redbeard's Basilisk:

Redbeard's is more terrifying. It has an ork hanging out underneath it ready to bash you!


Automatically Appended Next Post:
I'll be honest, I started to read this then

I didn't get terrified. I got bored.

Cavemen Frazzled have no fear of computer AI. Caveman Frazzled just bash with rock.

 Frazzled wrote:

Cavemen Frazzled have no fear of computer AI. Caveman Frazzled just bash with rock.

Perfect response.

 whembly wrote:

 Frazzled wrote:

Cavemen Frazzled have no fear of computer AI. Caveman Frazzled just bash with rock.

Perfect response.

Any response but and you've basically fethed yourself to some degree.

The only solution I see now is to create this malevolent AI so that we can destroy it, for our own sakes.

You've already damned yourself!

 Easy E wrote:
You've already damned yourself!

We're all damned.

As Tyler Durden says... Bring it Bish!

Took me a while to work out what all the fuss was about and frankly it seems strange to be concerned about a thought experiment that requires such a drastic suspension of disbelief to be threatening. Really I was slightly more concerned by the cult undertones of LessWrong than any hypothetical AI.

 Corpsesarefun wrote:
Took me a while to work out what all the fuss was about and frankly it seems strange to be concerned about a thought experiment that requires such a drastic suspension of disbelief to be threatening. Really I was slightly more concerned by the cult undertones of LessWrong than any hypothetical AI.

Same. To be tormented by this thought experiment is to accept that it may happen. The people that believe it may happen also believe if they don't make it happen they will be harmed. They are also tech literate and well funded... And some of the members seem a few cards short of a deck.

This may be a case of a self-fullfilling prophecy.

 Corpsesarefun wrote:
Took me a while to work out what all the fuss was about and frankly it seems strange to be concerned about a thought experiment that requires such a drastic suspension of disbelief to be threatening. Really I was slightly more concerned by the cult undertones of LessWrong than any hypothetical AI.

Corpses gets it in one, really. Not only suspension of disbelief, but such an absolutely deterministic model that if you believe it at all, you may as well not care because it already will, has, and always will have happened.

 TheCustomLime wrote:
Same. To be tormented by this thought experiment is to accept that it may happen. The people that believe it may happen also believe if they don't make it happen they will be harmed. They are also tech literate and well funded... And some of the members seem a few cards short of a deck.

This may be a case of a self-fullfilling prophecy.

That's why the founder shut it down. The last thing you want to do is to encourage someone intelligent, unstable, and with considerable resources to follow that train of reason.

Probably the most responsible thing we can do at this point is to stop talking about it. My first thought was to bring it up to friends because I found it interesting. I'm not sure I can now.

 daedalus wrote:
My first thought was to bring it up to friends because I found it interesting. I'm not sure I can now.

Again, it doesn't matter. Bring it up to whomever you want. For it to come to pass, it has to already have done so.

More importantly, we've already opened Pandora's Box on this one.

I'm with Corpses here, it requires me turning off so much of my thinking to be scared by it that it is kinda silly.

While the thought experiment postulates about an AI, it doesn't take much of a stretch to compare it to a human-led dictatorship either. And that notion sounds all kinds of plausible.

I find actual living, breathing people who think like this:

Like his projected Friendly AIs, Yudkowsky is a moral utilitarian: He believes that that the greatest good for the greatest number of people is always ethically justified, even if a few people have to die or suffer along the way. He has explicitly argued that given the choice, it is preferable to torture a single person for 50 years than for a sufficient number of people (to be fair, a lot of people) to get dust specks in their eyes.

far more worrying than imaginary malevolent AI gods that apparently like to play "Deal or No Deal".

It seems that this Yudkowsky person doesn't understand that some suffering passes a threshold of being insignificant to the party, no matter the size.

Its the time travel aspect that makes this a lousy thought experiment but a pretty good idea for a Doctor Who episode.

Oh, and Demon with a Glass Hand predates Ultron by a few years, too. It was also the Twilight Zone episode that the Terminator was based off of.

 Redbeard wrote:
Redbeard's Basilisk:

Make the drive "stick" a broom

squidhills wrote:
Its the time travel aspect that makes this a lousy thought experiment but a pretty good idea for a Doctor Who episode.

It has nothing to do with time travel (not that this fact would make it any less lousy). Didn't get the panic factor myself in the beginning and found a nice explanation on http://www.bn2b.com/rokos-basilisk-is-one-of-the-weirdest-scariest-ideas-on-the-web/:

Now, you may be confused about how a robot that doesn’t exist until years after you die (if at all) could torture you. I didn’t understand this for quite a while after hearing about Roko’s Basilisk, but when I actually got it, it was maybe the most mind-bending and scary part of the whole thing.

This AI will be so advanced, that it will be able to create a near-perfect copy of yourself, and torture it. Being a super intelligent godlike AI, on a level that our puny human brains can’t fully comprehend, it’ll probably be pretty good at torturing this copy of you. We’re talking years of constant physical pain; unending mental anguish; there could be gross insects involved, maybe snakes. Lots and lots of bad stuff.

But what do you care, if it’s only torturing a future copy of you, not the real you? Here’s the thing: Your clone will think like you, go through life experiencing the same things as you (before being tortured), and it will feel like you. It will think it’s you. The only difference between your copy and you, will be that the real you will get to the end of your life and die, and the copy will at some point be tapped on the shoulder by the AI, and tortured. Up until that point, the replica version of you will have no clue that it isn’t the real you, and that’s the rub:

You might be the replica.

Sounds less like Dr. Who and more like Outer Limits.

I've made a decision. No Pathfinder game is safe with me ever again.

"Roko's Doorknob"
Aura faint enchantment; CL 1st
Slotless wonderous item; Price 2,000 gp; Weight —

Description
This unimpressive, everyday item is typically one that many people will handle or come into contact with in a public place. Doorknobs and flagstones are frequent choices, though barstools and railing is not unheard of either. On contact, this object implants the following thought in the victims mind per sow thoughts (Will save DC 12): "In the future, a somewhat malevolent god named Roko will come about and punish those who does not do its bidding. There is a way for this god to punish people now who are not helping it come into existence later."

Construction Requirements
Craft Wonderous Item, sow thoughts; Cost 1,000 gp

Cost figured as use activated item (2000) * caster level (1) * spell level (1) * duration greater than 24 hours (.5)

Depending on your cosmology, that could get disgusting quickly. Even if you're not spawning dark gods by creating large numbers of believers, you could still destabilize towns full of people quickly.

I can start making those at level 3.


Automatically Appended Next Post:

 Allod wrote:

Sounds less like Dr. Who and more like Outer Limits.

It's not time travel as much as it is an uncertainty of when the milestone of its creation occurs. Were it to occur ever, it could be already happening now, or it could be something someone is making.

I keep picturing I Have No Mouth And I Must Scream.

 Fafnir wrote:
While the thought experiment postulates about an AI, it doesn't take much of a stretch to compare it to a human-led dictatorship either. And that notion sounds all kinds of plausible.

The Basilisk being a machine is the crux of the notion. The computer simulates reality with perfect accuracy in order to predict the outcome of future events, this is a notably unheard of trait in dictators as far as I'm aware so we're safe for now.

 Corpsesarefun wrote:

 Fafnir wrote:
While the thought experiment postulates about an AI, it doesn't take much of a stretch to compare it to a human-led dictatorship either. And that notion sounds all kinds of plausible.

The Basilisk being a machine is the crux of the notion. The computer simulates reality with perfect accuracy in order to predict the outcome of future events, this is a notably unheard of trait in dictators as far as I'm aware so we're safe for now.

This was not explained in the original post. My fault for not reading the article.

It's worth a read, even if it is just to have something to talk about at the watercooler come monday.

Did go back and read it, interesting article.

I don't see there being any point in worrying about being the original or the replica. You can really only be truly aware of yourself anyway.
At that point, the only reasonable thing to do is to go out in the most spiteful and malicious way possible towards anything that could potentially become the basilisk, because feth that guy.

 Allod wrote:

squidhills wrote:
Its the time travel aspect that makes this a lousy thought experiment but a pretty good idea for a Doctor Who episode.

It has nothing to do with time travel (not that this fact would make it any less lousy). Didn't get the panic factor myself in the beginning and found a nice explanation on http://www.bn2b.com/rokos-basilisk-is-one-of-the-weirdest-scariest-ideas-on-the-web/:

Now, you may be confused about how a robot that doesn’t exist until years after you die (if at all) could torture you. I didn’t understand this for quite a while after hearing about Roko’s Basilisk, but when I actually got it, it was maybe the most mind-bending and scary part of the whole thing.

This AI will be so advanced, that it will be able to create a near-perfect copy of yourself, and torture it. Being a super intelligent godlike AI, on a level that our puny human brains can’t fully comprehend, it’ll probably be pretty good at torturing this copy of you. We’re talking years of constant physical pain; unending mental anguish; there could be gross insects involved, maybe snakes. Lots and lots of bad stuff.

But what do you care, if it’s only torturing a future copy of you, not the real you? Here’s the thing: Your clone will think like you, go through life experiencing the same things as you (before being tortured), and it will feel like you. It will think it’s you. The only difference between your copy and you, will be that the real you will get to the end of your life and die, and the copy will at some point be tapped on the shoulder by the AI, and tortured. Up until that point, the replica version of you will have no clue that it isn’t the real you, and that’s the rub:

You might be the replica.

Sounds less like Dr. Who and more like Outer Limits.

Nope. Making a copy of me in the future is not punishing me now. The though experiment posits that the AI can punish people now. The future is not now. Now is now. Without time travel, this hypothetical AI is no threat to me, so I don't care about it or what it does. AIs may one day be a reality. Time travel won't.

Though you are correct about it making a good Outer Limits episode, too.