Legacy COBOL code is largely used in critical systems like those of banks and airlines. What could go wrong with having that code rewritten by stochastic parrots who get programming answers wrong half of the time?
LLMs produce code that is functionally error prone while looking reasonable (in the same way that it produces answers that are grammatically correct, correctly spelled, but factually incorrect).
As we all know, fixing bugs in someone else’s code is generally more difficult than writing the code correctly in the 1st place , and that’s going to apply to a LLMs code output just as much as a humans, if not more.
I’m aware they’re not using a generic model, but that’s not much better. Current custom-made models still fuck up significantly more than humans, and in less predictable ways.
Even if their custom model is slightly incorrect 1% of the time, that’s still a major problem in critical systems like those.
It’s good to see that they aren’t just piping through GPT-whatever, but in my experience, the vast majority of people who tout AI, even at large corporations, generally have zero idea how it works. I’m still not convinced that this is a good idea.
I mostly use A.I. to translate. ChatGPT gets that done it gets it done pretty good, especially when you say “translate this mandarin text into English. I don’t care if it is somewhat inaccurate, just do it as best as you can.“
It’s cool for small and easily testable functions like sorting, but to refactor large amounts of code? No thanks. Would be great if it could leave comments on my pull request though.
I thought it would leave comments on individual lines of code with feedback and code quality, but seems like it just summarizes what the pull request changes
the summary stuff would be better if it was per file instead of overall
I suppose I shouldn’t be surprised at the negative response here, but personally this seems like the perfect application of LLMs. Yeah, it’ll need to be verified by humans, but so would human-translated code. Using an appropriately trained LLM to do the first pass translation has the potential to eliminate a lot of toil.
Legacy COBOL code is largely used in critical systems like those of banks and airlines. What could go wrong with having that code rewritten by stochastic parrots who get programming answers wrong half of the time?
LLMs produce code that is functionally error prone while looking reasonable (in the same way that it produces answers that are grammatically correct, correctly spelled, but factually incorrect).
As we all know, fixing bugs in someone else’s code is generally more difficult than writing the code correctly in the 1st place , and that’s going to apply to a LLMs code output just as much as a humans, if not more.
That’s assuming they’re using one of the generic models like ChatGPT and not something custom they’ve created specifically to do this.
Edit: they are in fact using their own as per the article
I’m aware they’re not using a generic model, but that’s not much better. Current custom-made models still fuck up significantly more than humans, and in less predictable ways.
Even if their custom model is slightly incorrect 1% of the time, that’s still a major problem in critical systems like those.
Which models are those?
It’s good to see that they aren’t just piping through GPT-whatever, but in my experience, the vast majority of people who tout AI, even at large corporations, generally have zero idea how it works. I’m still not convinced that this is a good idea.
I mostly use A.I. to translate. ChatGPT gets that done it gets it done pretty good, especially when you say “translate this mandarin text into English. I don’t care if it is somewhat inaccurate, just do it as best as you can.“
The AI would likely be trained or fine tuned specifically for COBOL. In these very narrow use cases AI can find some things that humans can miss.
Google did this recently on a sorting algorithm and was able to speed it up by 70%: More info here
It’s cool for small and easily testable functions like sorting, but to refactor large amounts of code? No thanks. Would be great if it could leave comments on my pull request though.
Try PR-Codex
I thought it would leave comments on individual lines of code with feedback and code quality, but seems like it just summarizes what the pull request changes
the summary stuff would be better if it was per file instead of overall
Hm don’t think I can help you with that unfortunately.
It’s nice for quickly seeing what a PR is about, not much more
I suppose I shouldn’t be surprised at the negative response here, but personally this seems like the perfect application of LLMs. Yeah, it’ll need to be verified by humans, but so would human-translated code. Using an appropriately trained LLM to do the first pass translation has the potential to eliminate a lot of toil.