My responses to Meditations On Moloch from slatestarcodex.com. August 18, 2017
Abstract: I don’t think he’s entirely right. I think Moloch himself fell into a larger trap. And all the rest of it is just temporary thrashing around over a geological time scale. I feel like he’s looking for absolution for profitable but cruel behavior he himself engages in. But that’s not super relevant here. My root position is that “Moloch” here is doomed in the long term.
“thinking of the system as an agent throws into relief the degree to which the system isn’t an agent”
Can we say for sure it’s not self aware? Though I’ll admit odds are it isn’t and either way it clearly can’t be communicated with in way that suggests agency.
“Every single citizen hates the system, but for lack of a good coordination mechanism it endures. From a god’s-eye-view, we can optimize the system to “everyone agrees to stop doing this at once”, but no one within the system is able to effect the transition without great risk to themselves.”
The issue isn’t coordination. That can be achieved with very simple tools. Think “attack at dawn.” The issue is freedom to do so, and the agreement required. The issue is fact immunity and the tyranny of pain.
“Any sect at all that is leaner, meaner, and more survivalist than the mainstream will eventually take over.”
Survival advantage is more complicated than that. Continued existence of diversity proves it. Trade offs prevent this from always being the case. Also cooperation is inherently adaptive. The existence of tissues proves that. If cancer is single celled rebellion, life is proof that rebels never win pandemically. Put another way “leaner” apparently requires cooperation.
“Any sect of rats advocating some action to stop the cycle will be outcompeted by their cousins for whom advocating anything is a waste of time that could be used to compete and consume.”
This is only true in the narrow simplistic example given. This is the failure of game theory. It’s too reductive to be totally accurate. Niches and symbiosis are a thing. Again, diversity exists for good reason. Though I get the point being made and it certainly applies in many contexts. Read on.
“The reasons Nature is red and tooth and claw are the same reasons the market is ruthless and exploitative)”
True, without question. Zero regulation leads to it. America is proof democracy cannot survive vs capitalism so long as corporate propaganda is unregulated and invisible.
“Once one agent learns how to become more competitive by sacrificing a common value, all its competitors must also sacrifice that value or be outcompeted and replaced by the less scrupulous.”
True. Unless they find a way to obviate the context that forced previous iterations into that context. Of course there’s a constraint on the hypothetical buried in the previous statement. Specifically you can define “more competitive” however you wish. See the point above about niches.
“optimizing for electability is optimizing campaign donations from corporations – or maybe it isn’t, but officials think it is.”
That is just one layer. It’s also pure graft and facing reality. The corporations are the real government, or are close to becoming it. After all, like Bernays said: “Propaganda is the executive arm of the invisible government.”
“some competition optimizing for X, the opportunity arises to throw some other value under the bus for improved X. Those who take it prosper. Those who don’t take it die out. Eventually, everyone’s relative status is about the same as before, but everyone’s absolute status is worse than before. The process continues until all other values that can be traded off have been”
But it’s rarely ever just X. It’s usually an array of goals that must be met simultaneously, which is where all the tradeoffs and diversity come from. This is why the arc of history bends towards peace.
“In an insufficiently intense competition (11–14), all we see is a perverse failure to optimize”
This comes with lack of regulation because the first thing people do with power is deny subsequent players equal access to it. Just imagine it. You find a genie bottle. You always carry it off or use it immediately.
“It’s kind of embarrassing that random nobodies can think up states of affairs better than the one we actually live in.”
They aren’t actually better across all relevant contexts usually. We selectively judge and we’re inherently narrow because we are diverse individuals who specialize.
Being emperor includes getting the job in the first place. It is indeed a shame that rising to power is a different skill from using power and it does seems like there’s an argument to be made that the two skills are mutually exclusive, at least in some if not most or all current enduring real world systems.
Also there is the matter of agenda. Education isn’t broken. It’s lying about it’s true purpose.
“like gods being ordered around by a moron.”
That’s every use of technology ever. Think about what a shovel is to the insects and bacteria in the ground we move with it. All power is like that. Fire alone is staggering when thought of this way.
“the traps happen when you find an opportunity to trade off a useful value for greater competitiveness.”
True, this is often the case.
“This is an age of whalefall, an age of excess carrying capacity, an age when we suddenly find ourselves with a thousand-mile head start on Malthus.” & “As long as resources aren’t scarce enough to lock us in a war of all against all, we can do silly non-optimal things – like art and music and philosophy and love – and not be outcompeted by merciless killing machines most of the time.”
It’s always whalefall though. The myth of scarcity is just that. Dreamed up to satisfy objectives. It’s like the myth of over population. Think about grey goo. Why didn’t it happen already? Why isn’t all plant life dead at the hands of some microorganism? Why hasn’t any life closed the loop and just swarmed the planet like some organic von neumann probe? Because reality doesn’t work that way.
“the “race to the bottom” stops somewhere short of the actual ethical bottom, when the physical limits are run into.”
There is no bottom. It’s not a tower, it’s a hyper sphere. Moving left costs you right moving in costs you out. Tradeoffs.
“You can incentivize slaves through the carrot or the stick, and the stick isn’t very good.”
It’s perfect under certain constraints. Those constraints are created by the side effects and strengths of superior approaches.
“Or to give another example, one of the reasons we’re not currently in a Malthusian population explosion right now is that women can only have one baby per nine months.”
Not true. As prosperity rises birth rates decline. Put differently, when people have the true power of choice, they use it. Once post scarcity arrives for us all (not just the aristocracy) we’ll be free to breed responsibly. Also, once death by aging is taken off the table the logical argument for breeding will be deeply weakened. We’ll eventually realize that the argument for doing anything at random is almost always weaker than doing it with intent and planning.
“If those weird religious sects that demand their members have as many babies as possible could copy-paste themselves, we would be in really bad shape. As it is they can only do a small amount of damage per generation.”
Again, not true. A religion is a memetic contagion. Also not every ideology scales. What works for a thousand is not assured to work for a billion. All the catch 22s apply. Rome expanded too far, acquiring power is not the same as using power, etc. Again, like the grey goo, if it was this simple it would have happened already. The ancients weren’t stupid. Why isn’t humanity united under one religion?
“Or suppose someone invents a robot that can pick coffee better and cheaper than a human. The company fires all its laborers and throws them onto the street to die.”
They wouldn’t die. They would reform or revolt long before that. This is aristocratic propaganda. We do not need the companies to keep us alive. They need us to keep them alive. It’s like the rail system. It lives because we don’t attack the tracks. Any stationary asset perceived to be toxic to the people will be destroyed one way or another. This is why our cities rarely have walls anymore.
“in theory it’s optimizing for voter happiness which correlates with good policymaking. But as soon as there’s the slightest disconnect between good policymaking and electability, good policymaking has to get thrown under the bus.”
No. That’s assuming totally fair elections and rational choice. That’s not how humans make decisions or choose policy. There is a segment of humanity in possession of what amounts to a stealthy wide beam low power mind control ray. They are using it to create conditions that make it easier to use. Politics and its corruption is just one subset of that effort. Most obviously they’ve used it to shape “education” into something which confers less and less manipulation immunity.
“The opposite of a trap is a garden.”
Agreed. Which is why we need to completely reform the planet.
“Governments solve arm races within a country by maintaining a monopoly on the use of force, and it’s easy to see that if a truly effective world government ever arose, international military buildups would end pretty quickly.”
That’s not assured at all. Again. Mind control ray. If the holders of that ray profit by continual utterly pointless military spending, and the victims continue to believe there is no such ray, then you can expect the perpetual tossing of spears into the sea.
“The two active ingredients of government are laws plus violence – or more abstractly agreements plus enforcement mechanism.”
That’s the basis of all culture and cooperation. It is the creation of a synthetic version of the laws of physics. An artificial reality. It’s best to realize this is the foundation of all our action. Efforts to reshape, expand, or escape that baseline set of rules that are instantly enforced by the cosmos itself. Breathing fire.
“But these institutions not only incentivize others, but are incentivized themselves.”
Exactly. That mind control ray creates a feedback loop, or at least a spiral.
“The libertarian-authoritarian axis on the Political Compass is a tradeoff between discoordination and tyranny. You can have everything perfectly coordinated by someone with a god’s-eye-view – but then you risk Stalin. And you can be totally free of all central authority – but then you’re stuck in every stupid multipolar trap Moloch can devise. The libertarians make a convincing argument for the one side, and the monarchists for the other, but I expect that like most tradeoffs we just have to hold our noses and admit it’s a really hard problem.”
No it isn’t. You just proved it. One’s down side is a RISK, the other is a CERTAINTY. When you’ve got nothing left to lose the gamble is always wise. Specifically, I propose friendly AI rule. I have a podcast that arrives at this point.
“Multipolar traps – races to the bottom – threaten to destroy all human values. They are currently restrained by physical limitations, excess resources, utility maximization, and coordination. The dimension along which this metaphorical river flows must be time, and the most important change in human civilization over time is the change in technology. So the relevant question is how technological changes will affect our tendency to fall into multipolar traps.”
Technology makes us better at everything we do, good or bad. Considering where we were and where we are, I’m extremely confident in saying where we are going is an awesome place.
The point of government is to implement collective human agenda while avoiding tragedy of the commons issues, which are being called Moloch in this discussion. The sheer fact of our population, continued survival, growing standards of living, lengthening life span, and receding violence proves that we are getting ever better at accomplishing our root objective. Survive and enjoy it. Yes, there is profound need for reform and room for improvement, but it’s already happening.
“Technology is all about creating new opportunities.”
And perfecting the exploitation of old ones.
“Polluting the atmosphere to build products quicker wasn’t a problem before they invented the steam engine.”
It still isn’t a problem. We’ve just relocated the fuel and put the fact that nature isn’t a garden, but rather a trap, right in our faces. We can pull the carbon out of the air and turn it into fuel. Nuclear waste isn’t waste, it’s unspent fuel. Climate change scares us because the change forces us to face certain facts we’d rather not face. It is only a problem if we fail to adapt.
“The limit of multipolar traps as technology approaches infinity is “very bad”.”
No because those traps are based on other constraints. This idea of technological hell is based on the selective dissolution of constraints. If you’re controlling the hypothetical sure, anything’s possible, but that doesn’t make it possible in reality. This is special pleading.
To understand why remember that sufficiently advance technology starts looking like magic. Now try trapping an immortal omnipotent wizard. It doesn’t work that way. The assumption is being made that this situation scales. There’s no proof of that. Continued existence if proof otherwise. See above about grey goo. At worst we could potentially argue that all roads lead to extinction. But if that’s the case then what’s the harm in gambling?
“Physical limitations are most obviously conquered by increasing technology.”
Not quite. Technology is a lawyer. Physics are the law. As yet there’s no lobbying in this context. The law is absolute, but we can exploit it with increasing efficiency. Find more and more loopholes. etc
“The slavemaster’s old conundrum – that slaves need to eat and sleep – succumbs to Soylent and modafinil. The problem of slaves running away succumbs to GPS. The problem of slaves being too stressed to do good work succumbs to Valium. None of these things are very good for the slaves. (or just invent a robot that doesn’t need food or sleep at all. What happens to the slaves after that is better left unsaid)”
Again, this is ignorant. What happens to the slaves is reform or revolt. #BasicIncome
“But as Bostrom puts it in Superintelligence: There are reasons, if we take a longer view and assume a state of unchanging technology and continued prosperity, to expect a return to the historically and ecologically normal condition of a world population that butts up against the limits of what our niche can support. If this seems counterintuitive in light of the negative relationship between wealth and fertility that we are currently observing on the global scale, we must remind ourselves that this modern age is a brief slice of history and very much an aberration. Human behavior has not yet adapted to contemporary conditions. Not only do we fail to take advantage of obvious ways to increase our inclusive fitness (such as by becoming sperm or egg donors) but we actively sabotage our fertility by using birth control.”
Again, special pleading. And tons of assumptions not born out by the data. He’s basically saying things are gonna fall apart for reasons that apparently don’t apply now. He’s saying present conditions don’t scale but he’s using hypothetical conditions that might not scale either. I assert that if his end state was possible it would have happened already with lesser technology. Memes aren’t limited like genes, and if his social darwinism had intrinsic-to-reality-law-of-physics based authority they would have already taken over. Again, grey goo and hyperspheres. There isn’t one organism dominating all life. (No, we don’t count, expressly because of our diversity.)
And more tellingly there isn’t one ideology that dominated all others. Memetic life hasn’t produced a final winner either and I think there’s no reason to assume it ever could. Even within the large successful ones there is constant dispute and diversity. Equilibrium might be possible across some subjectively relevant time scale, but again it is only pure fact resistant dogma to assume it’s gonna end up dystopian.
“Such a desire is currently being selected for, as are other traits that increase our propensity to reproduce.”
Like they say on wiki, citation needed. This sounds like cryptofascist dog whistling to me. At the very least it’s projection based on fear. As if he himself wants to swamp the planet with copies because of insecurity, fear of change, and genetic programming, and is projecting that desire onto the fabric of reality itself.
“could be telescoped into a more imminent prospect by the intelligence explosion. Since software is copyable,”
Software has no need to make identical copies except as precaution. Again, assumption of insecurity reveals insecurity. A Mind would only make a copy to diversify it. Otherwise it would be better simply to expand its own abilities. The very impulse to “breed” in the human context is not making clones, it’s creating new diverse allied entities or extending your own influence. Why risk competitors if you can just extend yourself? And we do it knowing full well the risk that that divergence may lead to a threat. Being afraid of AI is logically the same as being afraid of children. News flash, your daughter may grow up to poison you for the insurance money. That isn’t an argument to stop having daughters.
“The idea of technology making it possible is both plausible and terrifying.”
Possible, maybe, plausible, not even close. Sophistry.
“Excess resources, which until now have been a gift of technological progress, therefore switch and become a casualty of it at a sufficiently high tech level. Utility maximization, always on shaky ground, also faces new threats.”
Again pure baseless greedy fear. Arguments against utility maximization have always been based on special pleading. Selective definition of utility that in each counter argument always magically excludes some obvious aspect of the definition of goodness. Like arguing against charity itself by saying something like “But what if Hitlerbot9000 defines kindness as gas chambers!?” and acting like that’s some damning argument against kindness itself.
If anything the only problem with pure utilitarianism when fairly defined is that it may improperly prioritize reduction of suffering and death in favor of adding a pleasure. Negative utilitarianism doesn’t have this problem. It is simply a refinement of the fair definition asserting that we need to stop the torture and murder immediately.
And yes murder is one way to stop torture, but that’s not a gotcha argument against stopping torture. It simply means that care must be (and would be) taken not to assume destructive options are the only options. Ironically it is the opposition to utilitarians that are the more guilty of the flaw they assert in utilitarianism.
“In the face of continuing debate about this point, I continue to think it obvious that robots will push humans out of work or at least drive down wages (which, in the existence of a minimum wage, pushes humans out of work).”
Oh noes! Whatever shall we do! Humans released by the billion from slavery! FINALLY for the first time in history. This is a non-problem for anyone not infected by aristocracy memes. #BasicIncome
“In the earlier stages of the process, capitalism becomes more and more uncoupled from its previous job as an optimizer for human values. Now most humans are totally locked out of the group whose values capitalism optimizes for.”
That was never its job. Capitalism isn’t a complete ideology. It’s an economic system, and that’s all. And an incomplete one at that. There was never an uncoupling. Just the expanded use of a tool beyond it’s functionality, like asking religion to answer physics questions. Capitalism is half a wrench being used in place of a screw driver. It kinda works sometimes, but we’re falling badly behind our overall goals because of its inherent ineptitude for the tasks assigned to it. The issue isn’t the tool, the issue is our misuse of it.
“Memes optimize for making people want to accept them and pass them on – so like capitalism and democracy, they’re optimizing for a proxy of making us happy, but that proxy can easily get uncoupled from the original goal. Chain letters, urban legends, propaganda, and viral marketing are all examples of memes that don’t satisfy our explicit values (true and useful) but are sufficiently memetically virulent that they spread anyway.”
Not all memes are viral just as not all life is viruses. By this logic chain mail would have consumed all mail already. Again, grey goo, hyperspheres. It doesn’t work like that.
“Religions, at their heart, are the most basic form of memetic replicator – “Believe this statement and repeat it to everyone you hear or else you will be eternally tortured”.”
And again, no one religion or ideology has come to dominate all the others.
“The creationism “debate” and global warming “debate” and a host of similar “debates” in today’s society suggest that memes that can propagate independent of their truth value has a pretty strong influence on the political process.”
True. Fact immunity is a problem, but it’s an understood problem that can be solved. Technology will dissolve the foundations on which that temporary failing is built. Soon the wielders of the mind control ray will find themselves in free fall with a tool that no longer works.
“Maybe these memes propagate because they appeal to people’s prejudices, maybe because they’re simple, maybe because they effectively mark an in-group and an out-group, or maybe for all sorts of different reasons.”
Again truth, but the point is being missed. Trade offs and diversity and hyperspheres.
“The point is – imagine a country full of bioweapon labs, where people toil day and night to invent new infectious agents. The existence of these labs, and their right to throw whatever they develop in the water supply is protected by law. And the country is also linked by the world’s most perfect mass transit system that every single person uses every day, so that any new pathogen can spread to the entire country instantaneously. You’d expect things to start going bad for that city pretty quickly. Well, we have about a zillion think tanks researching new and better forms of propaganda. And we have constitutionally protected freedom of speech. And we have the Internet. So we’re kind of screwed.”
No, because again if a mind annihilating memetic pathogen could evolve, it would have already and this planet would be acognitive. Plenty of religions and ideologies have been self annihilating but it’s clear that the laws of physics simply do not permit one that is capable of spreading to everyone. I suspect it’s about scale, but that’s mostly a guess. I would suggest looking at the maximum size of suicide cults. I suspect there is a point at which force starts to be required to prevent defectors and internal opposition. How many people must be poised with the koolaid in hand before one of two of them start eyeing the fire exits? I suspect it’s a shockingly small number.
“There are a few people working on raising the sanity waterline, but not as many people as are working on new and exciting ways of confusing and converting people, cataloging and exploiting every single bias and heuristic and dirty rhetorical trick So as technology (which I take to include knowledge of psychology, sociology, public relations, etc) tends to infinity, the power of truthiness relative to truth increases, and things don’t look great for real grassroots democracy.”
True. Democracy is an illusion and it always has been. It’s only adaptive, like everything else, in a fairly specific niche. It has its place, but that place is not everywhere. Like capitalism unrestrained democracy simply hands power to the mind control ray.
“The worst-case scenario is that the ruling party learns to produce infinite charisma on demand. If that doesn’t sound so bad to you, remember what Hitler was able to do with an famously high level of charisma that was still less-than-infinite. (alternate phrasing for Chomskyites: technology increases the efficiency of manufacturing consent in the same way it increases the efficiency of manufacturing everything else)”
I dispute that it’s possible to have a philosophical zombie or toolbox with infinite charisma. The closer these tools get to this hypothetical perfection (which can’t even exist so long as humans are diverse thanks to mutually exclusive needs) the closer they get to self aware and beyond the control of their initiators.
“Coordination is what’s left. And technology has the potential to seriously improve coordination efforts.”
Agreed.
“But coordination only works when you have 51% or more of the force on the side of the people doing the coordinating, and when you haven’t come up with some brilliant trick to make coordination impossible.”
That 51% number is arbitrary for starters. The real value is closer to 10%. There’s no such trick for the same reason the asphalt and grass lacks a staph infection. No grey goo, no super ideology.
“People are using the contingent stupidity of our current government to replace lots of human interaction with mechanisms that cannot be coordinated even in principle.”
This is part of a plot. Starve the beast. The government isn’t stupid, it’s being sabotaged. Always there are elements within a system that wish to sabotage elements of it for personal empowerment. It’s an endless cycle as far as I can tell: “Selfishness beats altruism within groups. Altruistic groups beat selfish groups. Everything else is commentary” It’s gonna require constant intelligent adjustment and perpetual reform to achieve anything like stasis. And yes that’s a paradox, but so is everything else. It’s like “change is the only constant” or “there is an exception to every rule except this one.” That’s just the nature of nature. Take it up with the laws of physics.
“This is a rare confluence of circumstances where the we are unusually safe from multipolar traps, and as such weird things like art and science and philosophy and love can flourish.”
Defeatist, fatalist, ignorant, dogma. Reads the same to me as some old racist on his porch talking about society going to hell in a hand basket because blacks live on his block now without a cross being burnt on their lawn. Insecurity projected, and nothing more.
“As technological advance increases, the rare confluence will come to an end.”
Citation needed.
“New opportunities to throw values under the bus for increased competitiveness will arise. New ways of copying agents to increase the population will soak up our excess resources and resurrect Malthus’ unquiet spirit. Capitalism and democracy, previously our protectors, will figure out ways to route around their inconvenient dependence on human values. And our coordination power will not be nearly up to the task, assuming something much more powerful than all of us combined doesn’t show up and crush our combined efforts with a wave of its paw.”
Yes selfishness is adaptive within altruistic groups leading to decay unchecked, thing is, vigilance exists. By this logic all deaths would be from cancer. Clearly they aren’t. Cancer can be cured and is cured in every person that doesn’t die of it. A functional immune system proves it’s possible to have a functional immune system. We can, and will, and are, and arguably have, built one culturally. There is just room for improvement. Yes the patient could die. Arguably will die if you define death to include change. But qualitatively immortality of our culture is entirely possible.
“It can end in Eliezer Yudkowsky’s nightmare of a superintelligence optimizing for some random thing (classically paper clips) because we weren’t smart enough to channel its optimization efforts the right way. This is the ultimate trap, the trap that catches the universe. Everything except the one thing being maximized is destroyed utterly in pursuit of the single goal, including all the silly human values. Or it can end in Robin Hanson’s nightmare (he doesn’t call it a nightmare, but I think he’s wrong) of a competition between emulated humans that can copy themselves and edit their own source code as desired. Their total self-control can wipe out even the desire for human values in their all-consuming contest.”
Again, if this was possible at these scales, we’d have seen countless examples of the principal in action already to the point of a single dominant life form and a single dominant ideology. Again, it simply doesn’t work that way. Computation power can never match reality. A quantum computer selectively processes information but a lump of matter IS information. A perfect simulation of a rock will always take up more reality than the rock itself. If evolution could produce solutions like these they would have been exploited already. Again, there may be some backwards facing great filter death ray in our future but it will not take the form of runaway optimization. Some random bacterial mutation or psychotic idea would already have chanced on it. Life itself, memetic and genetic, has been trying to brute force that password since the dawn of time and it will never get it because clearly there isn’t one. Asserting there is one we just haven’t reached it yet is unprovable dogma. It’s an opinion, not a conclusion.
“Human-like cognitive organizations would then lack a niche in a competitive post-transition economy or ecosystem.”
Except all evidence points to altruistic groups being more adaptive than selfish ones. The very persistence of multicellular life in the face of constant single cell assault proves it. Saying they’ll eventually win that war in some future context again is a baseless assertion contrary to the evidence of your own existence.
“a society which nevertheless lacks any type of being that is conscious or whose welfare has moral significance. In a sense, this would be an uninhabited society.”
I dispute the possibility of pzeds in real life. In terms of class, a sufficiently high resolution copy is a member of the original set. A perfect copy of me, may not be me, but it certainly is human and conscious, at least to whatever extent I am. Any argument that it is “fake” in some way by definition would also apply to me unless you cheat the hypothetical. Which is what a pzed is conceptually, a cheat. Like saying “Have this spherical cube.” I can say it but that doesn’t mean it can actually exist.
“The last value we have to sacrifice is being anything at all, having the lights on inside. With sufficient technology we will be “able” to give up even the final spark. (Moloch whose eyes are a thousand blind windows!)”
That bridge is crossed. How much of what you do everyday is conscious? The whole concept of training proves that we’ve outsourced huge portions of ourselves to unconscious processes and that’s not even starting on the concept of inspiration and dreaming. Things which sound quintessentially human but are in fact quite possibly just as autonomic as a leg twitch reflex. It is my position that I think therefore I am. But you have no real proof. and I don’t think you ever could. I think it’s best to just assume that which tells you it feels is feeling simply to avoid future situations where you might be disbelieved yourself. Err on the side of caution and all that.
“Everything the human race has worked for – all of our technology, all of our civilization, all the hopes we invested in our future – might be accidentally handed over to some kind of unfathomable blind idiot alien god that discards all of them, and consciousness itself, in order to participate in some weird fundamental-level mass-energy economy that leads to it disassembling Earth and everything on it for its component atoms. (Moloch whose fate is a cloud of sexless hydrogen!)”
Again, saying that gonna happen at some point in the future despite it not already happening in the past 13 billion years is pretty silly. The mass energy economy we see when we look out into the stars is pretty gloriously diverse as far as I can tell. Saying some process or other is gonna magically convert it all into homogeneous soup is as silly as it sounds. Frankly even if it could it wouldn’t last. Speaking of sexless hydrogen: “Hydrogen is a light, odorless gas, which, given enough time, turns into people.” – Edward Robert Harrison
“Moloch can’t agree even to this 99.99999% victory.”
And that’s actually the good news. You’re right that the laws of physics are indeed quite implacable. But that also means they are incorruptible. I feel like I can work with consistent law universally and fairly applied. The existence of human culture proves humanity agrees with me. Remember what I said about the maximum possible size of suicide cults?
“We will break our back lifting Moloch to Heaven, but unless something changes it will be his victory and not ours.”
We’re gonna die anyway again if you assert sufficient change is equal to death. Five year old me died and was replaced by 38 year old me. And 38 year old me will die to be replaced hopefully by 38 million year old me 🙂
“Everyone dies, unpredictably, at a time not of their own choosing, and all the virtue in the world does not save you.”
Suicides would disagree. Also I plan to live forever, so far so good 🙂 Yes I might die, it might even be probable. But the only way you win that debate is literally over my dead body, and like Epicurus “Why should I fear death? If I am, then death is not. If Death is, then I am not. Why should I fear that which can only exist when I do not?”
“ORTHOGONALITY THESIS.”
Is right but only because it conflates intelligence, knowledge, and action. An emotionless super-intelligence isn’t possible in my view simply because it would only act when given a directive and that directive would function as an emotion. That’s all emotions are. The tasking of other processes. Evade fire, approach beauty, etc.
An emotionless super intelligence is basically a computer or library with no code and no patrons. It just sits there knowing stuff and doing nothing. Exactly like a rock. Physically encoded information of no value until some feeling agent gives it value. (This rock is special, this book is useful.)
So yes an AI could be tasked with absurd things, but again that bridge is crossed. All manipulation of reality is essentially a god being ordered around by a moron. But a god acting on moronic orders is going to be qualitatively less than a god acting on profundity. To the extent it is being foolish is the extent to which it falls short of super intelligence.
“This seems to me the strongest argument for authoritarianism. Multipolar traps are likely to destroy us, so we should shift the tyranny-multipolarity tradeoff towards a rationally-planned garden, which requires centralized monarchical authority and strongly-binding traditions.”
Agreed, minus the obvious right wing aristocracy serving libertarian tough love garbage. We can and will code a benevolent AI ruler and it will be a natural outgrowth of progressive rule of law. The current trend towards peace and goodness will continue, empowered by exponential technology growth. Utopia is coming. And unlike all these arbitrary, baseless, nameless, horror-of-the-future arguments, mine is based squarely in reality and requires no magic arbitrary unprecedented and universal changes.
“The liberal counterargument to that is that evolution is a blind idiot alien god that optimizes for stupid things and has no concern with human value.”
I think he meant the liberal straw man.
“Think your local anarchist commune versus Sparta.”
I don’t see any Spartans around. But anarchist power vacuums pop up and implode repeatedly. (And everyone rational realizes that all libertarians are either anarchists or liars.) Though they are doing so with less frequency as humanity approaches a system we all enjoy enough to stop killing each other.
“But the current rulers of the universe – call them what you want, Moloch, Gnon, whatever – want us dead, and with us everything we value.”
If the universe wanted us dead, we’d be dead. It’s hilarious hubris to think we’re surviving against what amounts functionally to god’s will. We are simply not that bad ass. Wake me when we alter a single law of physics.
“The opposite of a trap is a garden. The only way to avoid having all human values gradually ground down by optimization-competition is to install a Gardener over the entire universe who optimizes for human values.”
Agreed. And we’re already doing it. We call it the law. Eventually it’ll be machine readable and the documents we serve will be aware enough to actually speak with. Right now we use lawyers and judges but eventually the middle men will be unneeded and we’ll be able to ask the law itself direct questions.
“And the whole point of Bostrom’s Superintelligence is that this is within our reach. Once humans can design machines that are smarter than we are, by definition they’ll be able to design machines which are smarter than they are, which can design machines smarter than they are, and so on in a feedback loop so tiny that it will smash up against the physical limitations for intelligence in a comparatively lightning-short amount of time. If multiple competing entities were likely to do that at once, we would be super-doomed. But the sheer speed of the cycle makes it possible that we will end up with one entity light-years ahead of the rest of civilization, so much so that it can suppress any competition – including competition for its title of most powerful entity – permanently. In the very near future, we are going to lift something to Heaven. It might be Moloch. But it might be something on our side. If it’s on our side, it can kill Moloch dead.”
Agreed. See previous comment.
“Moloch” is screwed.
