I think the issue is LLMs can’t do a humans job, only be a tool for a human to use. And the tasks it’s good at weren’t large bottlenecks to begin with.
the tasks it’s good at weren’t large bottlenecks to begin with.
Disagree. There are plenty of tasks it is being tried with which it’s not good at, but to an extent you don’t know that until you’ve tried.
I use LLMs for code review, they help - a lot. Like so many new technology applications, we just didn’t do code review at anything like the level that LLMs enable doing it at, with LLMs we’re doing more of what we “always wished we had the time / attention span for” but never actually did, in practice. LLMs are lowering the total cost of finding issues in code, making the reward of doing them in depth worth the cost. In the past we’d do a much higher level review, and miss a lot of detail issues, but the cost of finding those detail issues without LLMs was very high and for the most part we deemed the detail issues “not worth the cost to find.” Using LLMs for code review is actually making more work than we used to put into code reviews, but the returns are there - worthwhile.
You might think of it like finite element analysis for mechanical design. Sure, that was possible with paper drawings and slide rules, but it wasn’t worth the cost using those tools. Now, there are areas of mechanical design where FEA modeling is the standard practice - if you don’t do it you’re considered to be slacking, not applying state of the art tools to the job.
There are some things that I find LLMs to be horrible at - like 3D drawings in Blender. However, they are actually really good at making specialized Blender usage tutorials to make it easier / faster for human artists to learn how to use the tool themselves - and once in awhile they can whip up a very helpful python script to automate something that would have been tedious / time consuming to do over and over but not quite worth learning python and the APIs to automate.
That’s a lot of words just to reiterate what OC said. None of your examples disagree with the fact that these companies are using AI for a ton of stuff they’re not good at. And the stuff it is good at are not bottlenecks. If you removed LLMs from the face of the planet today, code quality will not suffer significantly. Sure, doing code review en masse is good. But that is not what was holding back computer programming. It’s existence is not progressing software in any significant way either. It’s a nice to have, not a must have. Code was fine for four decades without LLMs. Indeed it makes me wonder, if a machine learning model was purpose made to do code review, instead of general purpose LLMs doing it, how much better it could be, if we actually leveraged what they’re good at.
Disagree. Code review done right is a virtually impossible bottleneck for most companies to handle, so in the past they didn’t do it and we have the shitshow of security and other bugs that we are experiencing today.
If you removed LLMs from the face of the planet today, code quality will not suffer significantly.
But LLMs aren’t going anywhere, and they are already being used to find vulnerabilities by hats both white and black. They are also finding functional bugs that affect life safety and financial stability.
Code was fine for four decades
The way that municipal water was fine for centuries before chlorination. Cholera outbreaks were just one of those unavoidable realities, like mass school shooting events in the US.
if a machine learning model was purpose made to do code review, instead of general purpose LLMs doing it, how much better it could be, if we actually leveraged what they’re good at.
From what I have seen over the past year, this type of specialization is happening, and the progress is real and significant.
Comparing code bugs to cholera. Nice way to be utterly disconnected from reality and what really matters in life. Let’s burn the planet down and waste all of our clean water because of code review.
Some businesses already have. Take Ford for example.
Edit: I guess they’re not being hired back because they’re “cheaper”, but rather because AI couldn’t effectively do their job. I suppose that turns out to be cheaper in the long run.
I think Ford and the rest are using AI as an excuse to do a rank and yank - everybody is let go, then some are invited back. This isn’t anything new - Florida Power and Light (which is about 1/10th the size of Ford in terms of engineers) did this “fire everybody and make them reapply for jobs” thing back in the early 1990s. It’s a way of “cleaning house” without singling out the bad apples.
I realize the subscription rates are running at a loss for the LLM providers, but… costs are also dropping. Anyway, the Claude $200 per month subscription rate is more tokens than I could practically use doing heavy software engineering 40 hours a week. If I saturated it with jobs around the clock I could use up all the tokens in the $200/month subscription in about a week, but that’s 168 hours. Working a 40 hour schedule, even at 4.33 weeks per month that’s only 173 hours… I’d hope people spend at least 5 hours a month doing something besides babysitting their LLM console.
Now, one way to burn tokens at an epic rate is to launch multiple projects in parallel, three or even four sessions open at one time working on different things, or possibly different aspects of the same thing. That’s also a good way to completely lose the picture of what the LLM is building and have no idea how a thing works when it gets done. I’m sure there are plenty of software development houses trying to “optimize” their workforce this way - I suspect their risks are… significant.
The vetting for those offshore programmers ain’t worth it tbh.
How many will you go through till you get a competent one? I worked at a company with an office in India and… Maybe 20% of the staff there were gifted and hardworking, 80% were a net detriment to the product we were building.
The office was closed. I hope they kept on the people who were actually good as remote employees but I have no idea.
Sad reality, I’m using LLMs to review the work of our offshore programmers, and as you say: some are good, some are not - and sadly, the ones who are not are also not learning to leverage LLMs to improve their work products. Without LLM review, I’d be advocating to find some of our offshore programmers “more productive ways to apply their skills.” With four rounds of LLM review, I’m effectively rewriting (the bad ones’) code for them… every… single… pull request.
My best offshore coder figured out how to use the LLMs to review his own code, we have architecture discussions about the best things to do and never have issues with how they are done.
My worst offshore coder “doesn’t believe in using AI” and continues to submit code for merge to master branch with obvious race conditions, panic crashes, etc.
Pretty soon they’ll find that humans are less expensive for a given amount of work.
Dogs will do it for belly rubs and $20 in kibble
I think the issue is LLMs can’t do a humans job, only be a tool for a human to use. And the tasks it’s good at weren’t large bottlenecks to begin with.
Agree
Disagree. There are plenty of tasks it is being tried with which it’s not good at, but to an extent you don’t know that until you’ve tried.
I use LLMs for code review, they help - a lot. Like so many new technology applications, we just didn’t do code review at anything like the level that LLMs enable doing it at, with LLMs we’re doing more of what we “always wished we had the time / attention span for” but never actually did, in practice. LLMs are lowering the total cost of finding issues in code, making the reward of doing them in depth worth the cost. In the past we’d do a much higher level review, and miss a lot of detail issues, but the cost of finding those detail issues without LLMs was very high and for the most part we deemed the detail issues “not worth the cost to find.” Using LLMs for code review is actually making more work than we used to put into code reviews, but the returns are there - worthwhile.
You might think of it like finite element analysis for mechanical design. Sure, that was possible with paper drawings and slide rules, but it wasn’t worth the cost using those tools. Now, there are areas of mechanical design where FEA modeling is the standard practice - if you don’t do it you’re considered to be slacking, not applying state of the art tools to the job.
There are some things that I find LLMs to be horrible at - like 3D drawings in Blender. However, they are actually really good at making specialized Blender usage tutorials to make it easier / faster for human artists to learn how to use the tool themselves - and once in awhile they can whip up a very helpful python script to automate something that would have been tedious / time consuming to do over and over but not quite worth learning python and the APIs to automate.
That’s a lot of words just to reiterate what OC said. None of your examples disagree with the fact that these companies are using AI for a ton of stuff they’re not good at. And the stuff it is good at are not bottlenecks. If you removed LLMs from the face of the planet today, code quality will not suffer significantly. Sure, doing code review en masse is good. But that is not what was holding back computer programming. It’s existence is not progressing software in any significant way either. It’s a nice to have, not a must have. Code was fine for four decades without LLMs. Indeed it makes me wonder, if a machine learning model was purpose made to do code review, instead of general purpose LLMs doing it, how much better it could be, if we actually leveraged what they’re good at.
Disagree. Code review done right is a virtually impossible bottleneck for most companies to handle, so in the past they didn’t do it and we have the shitshow of security and other bugs that we are experiencing today.
But LLMs aren’t going anywhere, and they are already being used to find vulnerabilities by hats both white and black. They are also finding functional bugs that affect life safety and financial stability.
The way that municipal water was fine for centuries before chlorination. Cholera outbreaks were just one of those unavoidable realities, like mass school shooting events in the US.
From what I have seen over the past year, this type of specialization is happening, and the progress is real and significant.
Comparing code bugs to cholera. Nice way to be utterly disconnected from reality and what really matters in life. Let’s burn the planet down and waste all of our clean water because of code review.
Some businesses already have. Take Ford for example.
Edit: I guess they’re not being hired back because they’re “cheaper”, but rather because AI couldn’t effectively do their job. I suppose that turns out to be cheaper in the long run.
I think Ford and the rest are using AI as an excuse to do a rank and yank - everybody is let go, then some are invited back. This isn’t anything new - Florida Power and Light (which is about 1/10th the size of Ford in terms of engineers) did this “fire everybody and make them reapply for jobs” thing back in the early 1990s. It’s a way of “cleaning house” without singling out the bad apples.
We are getting there - with 4,33 weeks a month that’s $866 a month in AI costs. You may get a programmer for that in some offshore countries
I realize the subscription rates are running at a loss for the LLM providers, but… costs are also dropping. Anyway, the Claude $200 per month subscription rate is more tokens than I could practically use doing heavy software engineering 40 hours a week. If I saturated it with jobs around the clock I could use up all the tokens in the $200/month subscription in about a week, but that’s 168 hours. Working a 40 hour schedule, even at 4.33 weeks per month that’s only 173 hours… I’d hope people spend at least 5 hours a month doing something besides babysitting their LLM console.
Now, one way to burn tokens at an epic rate is to launch multiple projects in parallel, three or even four sessions open at one time working on different things, or possibly different aspects of the same thing. That’s also a good way to completely lose the picture of what the LLM is building and have no idea how a thing works when it gets done. I’m sure there are plenty of software development houses trying to “optimize” their workforce this way - I suspect their risks are… significant.
The vetting for those offshore programmers ain’t worth it tbh.
How many will you go through till you get a competent one? I worked at a company with an office in India and… Maybe 20% of the staff there were gifted and hardworking, 80% were a net detriment to the product we were building.
The office was closed. I hope they kept on the people who were actually good as remote employees but I have no idea.
Sad reality, I’m using LLMs to review the work of our offshore programmers, and as you say: some are good, some are not - and sadly, the ones who are not are also not learning to leverage LLMs to improve their work products. Without LLM review, I’d be advocating to find some of our offshore programmers “more productive ways to apply their skills.” With four rounds of LLM review, I’m effectively rewriting (the bad ones’) code for them… every… single… pull request.
My best offshore coder figured out how to use the LLMs to review his own code, we have architecture discussions about the best things to do and never have issues with how they are done.
My worst offshore coder “doesn’t believe in using AI” and continues to submit code for merge to master branch with obvious race conditions, panic crashes, etc.
deleted by creator