Powderhorn ( @Powderhorn@beehaw.org ) to TechnologyEnglish · 1 年前Anthropic publicly releases AI tool that can take over the user’s mouse cursorarstechnica.comexternal-linkmessage-square6linkfedilinkarrow-up141file-text
arrow-up141external-linkAnthropic publicly releases AI tool that can take over the user’s mouse cursorarstechnica.com Powderhorn ( @Powderhorn@beehaw.org ) to TechnologyEnglish · 1 年前message-square6linkfedilinkfile-text
minus-square Kissaki ( @Kissaki@beehaw.org ) linkfedilinkEnglisharrow-up6·1 年前IIRC Windows has an accessibility feature where the cursor jumps to the primary default action in opening dialogs. Doing it screenshot based seems inefficient if y du could iterate through windows and controls.
minus-square flashgnash ( @flashgnash@lemm.ee ) linkfedilinkarrow-up3·1 年前Makes it work universally, even if the gui isn’t made with a standard toolkit Also it’s ai they don’t care about efficiency
minus-square Toribor ( @Toribor@corndog.social ) linkfedilinkEnglisharrow-up2·1 年前Yeah this is one of those things where accessibility settings can probably get you 90% there but screenshots and machine learning can probably close the gap somewhat reliably (even if it’s much less efficient).
IIRC Windows has an accessibility feature where the cursor jumps to the primary default action in opening dialogs.
Doing it screenshot based seems inefficient if y du could iterate through windows and controls.
Makes it work universally, even if the gui isn’t made with a standard toolkit
Also it’s ai they don’t care about efficiency
Yeah this is one of those things where accessibility settings can probably get you 90% there but screenshots and machine learning can probably close the gap somewhat reliably (even if it’s much less efficient).