I'm also wary of the additional attack surface. As with all new technologies, security is an afterthought and I guess it won't be long before some organizations find way to poison the training data, do prompt injection on a massive scale or find another way to have it discreetly inject malware in the code or compromise the computer it's working on.
I thought the coding agents were trained on practically all public code out there in the net, and then arguably some private code some people may not realize it's training them or something.
Do you think those evil organizations are going to find an even more poisonous corpus ? That'd be a big achievement in its own, if only as an existence proof.
I guess the bad actors (I mean those outside the relevant corporate boards) may find ways to trick the agents to misbehave more or more predictably, but not sure it'd be through training.
About the risks of people stopping learning. I heard a story the other day about a teacher who added some chatbot bait in an assignment. Some white text on white background telling the
composition should include some weirdly unrelated concepts. And then he found almost all students had submitted text following the bait, and he wondered they hadn't even at least read what
they sent, if they were'nt ready to write it themselves.
Then I thought: Well, I use text based mail user agents and browsers sometimes. So what if I was the student and I read the bait and humanly wrote a piece closely following the teachers instructions?
Why should I be required to disregard, or even care that some text was white on white ? Because a teacher assumes everyone does things like himself?
So the harm form LLMs is not only what they do, but what humans feel forced to do in order to adapt to widespread LLMs.
And just like with self driving accidents. The problem is not the number or the severity but what do you do with them? Even if claude wrote always good code, it wouldn't be maintainable just because
the humans left at the place wouldn't have made the initial effort to understand the problem that coding manually requires. But then that was already out the window with offshoring, staff rotations
and general mismanagement, so will it really make it any worse ?
All these companies borrowing hundreds of billions to fund data centers worries me because it seems inconceivable that they will be able to repay the loans.
Yeah, it seems inconceivable they even find ROI. It seems inconceivable they get environmental permits for they data centers, it feels inconceivable their suppliers get away with buying rare earths from warlods,
it seems inconceivable societies and governments accept collectively paying infrastructure for these companies private benefits, it seems inconceivable that they win enough lawsuits about copyright, server load,
psychological distress caused, unlicensed professional advice given, crime helping and abetting, etc.
But people keep betting on it. So I interpret it more like a bet on a change from a rules based society to a more feudal society where those with deep pockets, be they humans or things, can do what they please.
Businesses can offset hefty ai costs by hiring fewer people, but I don't see how private individuals (the biggest market by far) will want to spend that much.
Individuals will be coerced, as you say yourself.
Companies maybe can't fire so many people.
Most work is not there to produce any good, it's just there to keep people busy so they don't work around the powers that be.
So if companies stop hiring people they lose their power. Even if they could keep their production or increase benefits, that's just money.
Companies without jobs get less competitive (in the limit if they manage to offload everything to an AI and fire all employees except the CEO,
then anyone can build a competing company by just offloading to the AI, and then there's the
AI-CEO joke).
Companies without jobs get less political influence, fewer people to harrass and extort, fewer places to have offices in, less dogfeeding
(a la mythical Ford wanting employees to own a car).
How much are email summaries worth to you? I use ai mainly as a better google and suspect must people do too. But if the cost weren't negligible (free) i'd just use duck duck go.
Exactly. But not only to you. How much are email summaries worth to anyone ? How much is your work worth to others ? (specially those paying it?).
How much more than the AI costs is your work worth if 90% of it is done by an AI bot ?
Most work wasn't really worth much already before the AI bubble. If you get the same low quality and little purpose with higher costs, then how do you make ends meet ?
So AI is unreliable, expensive, and reckless, and people are unreliable, expensive and reckless. AI can't probably get much better, only more expensive, because
in the end it learns from humans. And even if it started learning autonomously, humans wouldn't be able to judge it's any better, just like I can't tell whether an olive tree is
more clever than another olive tree, or any more clever than I am.
But most people (rightly) dislike genocide, so you have to
keep the people alive, and you'll have to incur some costs for that. You don't have to keep any AI alive and don't have to incur costs for them.
So you could at least keep employing people.
The AI doesn't have any reason to want to take over to keep itself alive either. It's just some AI owner that wants it to exist to do something for him.
But this all depends on the AI owner not having to pay all the costs of his AI, as always, so in the end is about getting others to pay for your toys and vices,
so abuse, resistence, overthrow, like so often in history...
My fear is that big tech will start using extortion to get individuals to shell out for ai. Enshitification. I'm really scared actually.
Yes. I think they're already doing it.
The smallest part of it is that you go to a search engine and get AI slop without asking for it.
That will probably happen indeed. But we can always worry more about being killed in a upcoming war I guess, hopefully we'll be back on our game in 10 years or so after the bubble has burst.
That'd be nice but they thought of it and entshitificated it too. You'll be killed where the current world war gets to you, but it will be by an autonomous weapon, using AI to
navigate, read orders, select targets, aim, etc... And you and I are most likely training it right now by writing on the nets (specially I, with my walls of text

, you just wrote a line).
Just remember humans can also love, build works of art, assemble Pyras, etc.