The hottest new programming language is English! Or maybe not.
We recorded 4 claims from this post. 3 were checked against evidence. 1 can’t be checked. 4 more are held. Read the post at its source.
Claims from this post (4)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: SupportedSaid
“But the truth is it doesn’t work that well yet.”
What we tested: Programming in English with LLMs does not yet work very well.
What happened: On CodeContests validation problems, direct prompting solved 19% with five attempts; a structured testing and revision process reached 44%. This supports substantial reliability limitations while also documenting useful capability and improvement. (arXiv; Tal Ridnik, Dedy Kredo and Itamar Friedman, )
Assessment: UnresolvedSaid
“If the system really is being “programmed in English” it is not always doing what is being asked to do:”
What we tested: If ChatGPT/GPT-4 really is being "programmed in English" through its allegedly leaked system prompt, it is not always doing what it is asked to do.
Assessment: UnresolvedSaid
“Much of what Reid Southen, Katie Conrad, and I recently described is in flagrant violation of those directives.”
What we tested: Much of what Marcus, Reid Southen and Katie Conrad recently described is in flagrant violation of the directives in ChatGPT's allegedly leaked system prompt.
Assessment: Can’t be checkedSaid
“In the new regime, caveat emptor.”
What we tested: People programming via LLMs in English should beware of the risks.
Held from this post (4)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- Attribution is uncertain
- Attribution is uncertain
- The criteria are ambiguous. Our reading, not assessed: We have apparently just entered a new, different and frightening era in which debugging involves not only fixing our own mistakes but also coming to terms with machines that may or may not uphold their end of the bargain.
- The two reviews read the claim differently