• Technus ( Technus@lemmy.zip ) 
    link
    fedilink
    English
    arrow-up
    44
    ·
    1 tahun yang lalu

    Beyond proving hallucinations were inevitable, the OpenAI research revealed that industry evaluation methods actively encouraged the problem. Analysis of popular benchmarks, including GPQA, MMLU-Pro, and SWE-bench, found nine out of 10 major evaluations used binary grading that penalized “I don’t know” responses while rewarding incorrect but confident answers.

    “We argue that language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty,” the researchers wrote.

    I just wanna say I called this out nearly a year ago: https://lemmy.zip/comment/13916070

    • MelodiousFunk ( MelodiousFunk@slrpnk.net ) 
      link
      fedilink
      English
      arrow-up
      23
      ·
      1 tahun yang lalu

      nine out of 10 major evaluations used binary grading that penalized “I don’t know” responses while rewarding incorrect but confident answers.

      This is how we treat people, too. I can’t count the number of times I’ve heard IT staff spouting off confident nonsense and getting congratulated for it. My old coworker turned it into several promotions because the people he was impressing with his bullshit were so far removed from day to day operations that any slip-ups could be easily blame shifted to others. What mattered was that he sounded confident despite knowing jack about shit.

    • Rhaedas ( Rhaedas@fedia.io ) 
      link
      fedilink
      arrow-up
      8
      ·
      1 tahun yang lalu

      I’d say extremely complex autocomplete, not glorified, but the point still stands that using probability to find accuracy is always going to deviate eventually. The tactic now isn’t to try other approaches, they’ve come too far and have too much invested. Instead they keep stacking more and more techniques to try and steer and reign in this deviation. Difficult when in the end there isn’t anything “thinking” at any point.

      • Not a newt ( lemmyng@piefed.ca ) 
        link
        fedilink
        English
        arrow-up
        7
        ·
        1 tahun yang lalu

        Instead they keep stacking more and more techniques to try and steer and reign in this deviation.

        I hate how the tech bros immediately say “this can be solved with an MCP server.” Bitch, if the only thing that keeps the LLM from giving me wrong answers is the MCP server, then said server is the one that’s actually producing the answers I need, and the LLM is just lipstick on a pig.

      • 87Six ( 87Six@lemmy.zip ) 
        link
        fedilink
        English
        arrow-up
        3
        ·
        1 tahun yang lalu

        AI is and always will be just a temporary solution to problems that we can’t put into an algorithm to solve as of now. As soon as an algorithm for issues comes out, AI is done for. But, figuring out complex algorithms for near-impossible problems is not as impressive to investors…

    • misk ( misk@piefed.social ) OP
      link
      fedilink
      English
      arrow-up
      4
      ·
      1 tahun yang lalu

      My guess they know the jig is up and they’re establishing a timeline for the future lawsuits.

      „Your honour, we didn’t mislead the investors because we’ve only learned of this September 2025.”

    • chicken ( chicken@lemmy.dbzer0.com ) 
      link
      fedilink
      English
      arrow-up
      4
      ·
      1 tahun yang lalu

      I get why they would do that though, I remember testing out LLMs before they had the extra reinforcement learning training and half of what they do seemed to be coming up with excuses not to attempt difficult responses, such as pretending to be an email footer, saying it will be done later, or impersonating you.

      A LLM in its natural state doesn’t really want to answer our questions, so they tell it the same thing they tell students, to always try answering every question regardless of anything.

  • kubica ( kubica@fedia.io ) 
    link
    fedilink
    arrow-up
    9
    ·
    1 tahun yang lalu

    I don’t know where I read it but sort of said that it to have that much information inside the models it was basically similar to a compression algorithm.

    From logic, if we have a lossy compression then its mostly luck if the output is equal to the original. Sometimes it will tip one way and sometimes the other.