This is a game played on June 19 between Ueno Asami (as black) and Fujisawa Rina (as white). At move 159, Asami cut with K14.

It involves a followup of the life and death of the white group M10, and most human players would be able to judge it, but AI (from Golaxy to Katago) all seem to deem it “alive” and continue to fight without securing the life of that group first. While a human player can read out the sequence like Rina did during the game and make the right choice.

But it takes like 10k+ playouts before AI realize the K14 cut is a very good move, and (with few playouts AI even judges K14 as a blunder). And it takes millions of playouts before AI realizes the M10 white group is in trouble.

What other AI blindspots and hallucinations have you seen in real games?

  •  OmnipotentEntity   ( @OmnipotentEntity@beehaw.org ) English
    3•1 year ago

    As a follow up, I looked into this position with the current strongest rated network, and at 25 visits, K14 is actually the favorite move, followed by the reduction at O8*. After 100,000 visits it hates every move, but thinks that K14 is about 6.5 points better than the next best move H14. There are some big territorial swings, and depending on the exact sequence the network will like the situation more or less, so the network thinks this is a very volatile position.

    * Note, now that I’ve had a look at it, I don’t believe that it is possible for black to kill white at M10. Best black can do seems to be seki: bO8, O7, N5, P8, N8, N9, M8, P9. But if you have a better result for black please let me know.

    •  countingtls   ( @countingtls@lemmy.ml ) OPEnglish
      1•1 year ago

      Hmm, from I remember, black seems to be able to play bN8 after the bO8 bO7 (if white choose to play wL11 bK11, wK12 to cut off the black (or if white doesn’t respond locally, black can just easily connect back, and white is already too far behind), And L11 is where AI seems to be blind to life and death, and shorten its own group’s liberties.

      If white doesn’t choose to take the 3 black stones after the bO8, bO7, bN8, and choose to continue to push P8, black can just block with P9. White has to connect M8, and black can play L10, and white cannot connect M9 (or black connect N5 the whole white group dies). And white would have to capture with N5, black can throw in M9, wN9 capture, bO9, white again cannot connect M9, or bO4 kills all. Hence, the only option for white is to play O4, and black would be able to kill all the M10 white group stones. (this is effectively connect and die problem, and the commentator that day, along with Fujisawa I think, all read these out)

      Hence, the best option to keep that M10 group and minimize the lost is to play N5 to capture 3 black stones first right after black K14, and let black to cut off the two K15 K16 white stones (Fujisawa had the chance to save those two stones before, and the game might go into yose, but she played H15 and allow Ueno to push and cut)

      •  OmnipotentEntity   ( @OmnipotentEntity@beehaw.org ) English
        0•1 year ago

        Ah, I see what you mean now. I can confirm at lower visits, KataGo does indeed want to push and cut, and this is a problem. For what it’s worth, at higher visits KataGo sees the problem with the push and cut at wL11, and suggests other moves instead, such as wK13, or tenuki-ing and playing wC9.

        I think that wN5 played in the game is premature though, the squeeze play you described would be suicidal for black to try without the push and cut due to the aji at R8.

        I showed this position to lightvector (the author of KataGo) on the Computer Go Discord, and he had this to say:

        Thanks! I looked at this position too and I concur, there is no blind spot, in the sense that there is no important move affecting the tactics that has too low of a policy prior causing it to be not explored or under-explored, even deep into the variations I didn’t find any sign of that.

        There is some strong value head errors on positions, where some positions in branches that lead to a win by one side have evaluations part of the way along that are highly positive for the other side, which dissuades the search from thinking they work until potentially a fairly large number of playouts that make it “push through” that hill to discover the true value. That’s the reason why you sometimes see it miss the right move, and why values in the position fluctuate a ton even at fairly large numbers of playouts. More value head training to judge the short-term tactics might help a bit.

        In this case I suspect the value head might be too smooth or linear in its extrapolation. Like if you take a position winning for player A, and add 4 different reasons that each at first glance might allow B to win, but each one actually barely doesn’t work, then all of them count for nothing in reality, but if the value head is a little bit inaccurate and also is too linear and assigns some value to each one, the combination of all of them pulling in the same direction and the value head attributing some weight to each one and linearly adding them together might be enough in total to make the value head think B is good. I get a sort of sense that there’s something like that maybe going on here.