What scares me about AI is militaries eventually being forced to employ AI at the highest levels of decision making because humans would be too slow to react. While I don't believe in a Skynet scenario, I worry that decisions over the use of weapons of mass destruction would eventually have to be made by machine intelligence having no human consideration.
Such human considerations saved our species when Vasili Arkhipov, a flotilla commander and the second in command aboard a Soviet submarine denied the captain his authorization to use a nuclear torpedo against the U.S. Navy during the Cuban missile crisis in 1962. The use of a nuclear torpedo would've led to a nuclear response and possibly a global nuclear war.
Imagine a scenario where an AI responsible for countering a grave threat posed by some future superweapon, employed because no human being could possibly respond, responds based on a false alarm. That could be the end of all of us.
What excites me about AI is its potential to increase wealth and productivity tremendously and basically end poverty. Also, when you factor in the potential of AI for recursive self-improvement, a technological singularity might be a reality much sooner than we expect.
I'm loving the recent spree of technology and science fiction questions being asked recently!
There is one thing I'm scared off when it comes to A.I! and its not what people usually would think off when talking about A.I, I'm actually scared that it doesn't turn out to be as good as the people developing it have lead us to believe, I'm not scared of A.I taking over our jobs, being smarter than us or anything like that!
I'm scared that it is not a success, I want A.I taking job and being more efficient than us, I want A.I to do research to cure diseases faster than us! I want an A.I assistant when they start to become a thing and when the time comes I want A.I to help us advance to that next step, the next frontier, post-humanism because I believe thats the way this will all lead us too IF we manage to create something that becomes more intelligent that us I hope that it helps us out and doesn't kill us all like Skynet.
Also I want to be alive during this whole process, I can't wait for real A.I to start being integrated as a significant way of our day to day life and not just in search algorithms that gather our data and use it to sell us stuff, I want to see it incorporated in cars, phones, fridges, every utility in the house!
I want A.I in video games, just imagine the possibilities of A.I NPCs, dynamic characters not forced to run on scripts but instead going around the game worlds as if it was real and the player being able to interact with real, alive worlds that feel as if they where real people! This is what crosses my mind right now but there's so much more that A.I would change for us and I can't wait for it!
I am both thrilled, and terrified.
I can think of much more than can go wrong with AI, than what can possibly go right.
Here is my prediction based on what I believe is definitely coming.
In the beginning, AI will drastically improve our life, but as AI progresses into a more advanced entity, which will happen at an exponential rate, it will deduce that we are a threat to the planet, which equates as a threat to it's own existence.
Given the fact that we will have very little to offer to AI with our limited slow evolving imperfect biology, it's likely that our extermination will be imminent.