I know people are gonna freak out about the AI part in this.
But as a person with hearing difficulties this would be revolutionary. So much shit I usually just can’t watch because open subtitles doesn’t have any subtitles for it.
Yeah, transcription is one of the only good uses for LLMs imo. Of course they can still produce nonsense, but bad subtitles are better none at all.
Et tu, Brute?
VLC automatic subtitles generation and translation based on local and open source AI models running on your machine working offline, and supporting numerous languages!
Oh, so it’s basically like YouTube’s auto-generatedd subtitles. Never mind.
Hopefully better than YouTube’s, those are often pretty bad, especially for non-English videos.
They’re awful for English videos too, IMO. Anyone with any kind of accent(read literally anyone except those with similar accents to the team that developed the auto-caption) it makes egregious errors, it’s exceptionally bad with Australian, New Zealand, English, Irish, Scottish, Southern US, and North Eastern US. I’m my experience “using” it i find it nigh unusable.
ELEVUHN
ELEVUHN
They are terrible.
Youtube’s removal of community captions was the first time I really started to hate youtube’s management, they removed an accessibility feature for no good reason, making my experience with it significantly worse. I still haven’t found a replacement for it (at least, one that actually works)
and if you are forced to use the auto-generated ones remember no [__] swearing either! as we all know disabled people are small children who need to be coddled!
Same here. It kick-started my hatred of YouTube, and they continued to make poor decision after poor decision.
I’ve been working on something similar-ish on and off.
There are three (good) solutions involving open-source models that I came across:
- KenLM/STT
- DeepSpeech
- Vosk
Vosk has the best models. But they are large. You can’t use the gigaspeech model for example (which is useful even with non-US english) to live-generate subs on many devices, because of the memory requirements. So my guess would be, whatever VLC will provide will probably suck to an extent, because it will have to be fast/lightweight enough.
What also sets vosk-api apart is that you can ask it to provide multiple alternatives (10 is usually used).
One core idea in my tool is to combine all alternatives into one text. So suppose the model predicts text to be either “… still he …” or “… silly …”. My tool can give you “… (still he|silly) …” instead of 50/50 chancing it.
@neme @TheImpressiveX [Music]
In my experiments, local Whisper models I can run locally are comparable to YouTube’s — which is to say, not production-quality but certainly better then nothing.
I’ve also had some success cleaning up the output with a modest LLM. I suspect the VLC folks could do a good job with this, though I’m put off by the mention of cloud services. Depends on how they implement it.
Yeah I’ve used local whisper and LLMs to automatically summarize Youtube-videos and podcasts to text with good results.
Cool, thanks for sharing!
I see you prompt it to “Make sure to only use knowledge found in the following audio transcription”. Have you found that sufficient to eliminate hallucination and going off track?
Yes I have been impressed with the quality of summaries keeping to the content. I have seen, rare, attribution errors though, where who said what got mixed up in unfortunate ways.
All hail the peak humanity levels of VLC devs.
FOSS FTW
accessibility is honestly the first good use of ai. i hope they can find a way to make them better than youtube’s automatic captions though.
There are other good uses of AI. Medicine. Genetics. Research, even into humanities like history.
The problem always was the grifters who insist calling any program more complicated than adding two numbers AI in the first place, trying to shove random technologies into random products just to further their cancerous sales shell game.
The problem is mostly CEOs and salespeople thinking they are software engineers and scientists.
The app Be My Eyes pivoted from crowd sourced assistance to the blind, to using AI and it’s just fantastic. AI is truly helping lots of people in certain applications.
Spoiler: they won’t
Spoiler, they will! I use FUTO keyboard on android, it’s speech to text uses an ai model and it is amazing how great it works. The model it uses is absolutely tiny compared to what a PC could run so VLC’s implementation will likely be even better.
My experience with generated subtitles is that they’re awful. Hopefully these are better, but I wish human beings with brains would make them.
subtitling by hand takes sooooo fucking long :( people who do it really are heroes. i did community subs on youtube when that was a thing and subtitling + timing a 20 minute video took me six or seven hours, even with tools that suggested text and helped align it to sound. your brain instantly notices something is off if the subs are unaligned.
Oh shit, I knew it was tedious but it sounds like I seriously underestimated how long it takes. Good to know, and thanks for all you’ve done.
Sounds to me like big YouTubers should pay subtitlers, but that’s still a small fraction of audio/video content in existence. So yeah, I guess a better wish would be for the tech to improve. Hopefully it’s on the right track.
i just did it for one video :P it really is tedious and thankless though so it would be a great application of ml.
I did this for a couple videos too. It’s actually still a thing, it was just so time consuming for no pay that almost nobody did it, so creators don’t check the box to allow people to contribute subs
You can use tools like whishper to pre generate the subtitles. You will have pretty accurate su titles at the right times. Then you can edit the errors and maybe adjust the timings.
But I guess this workflow will work with VLC in the future as well
Jup. That should always be paid work. It takes forever. I tried to subtitle the first Always Sunny Episode. I got very nice results. Especially when they talked over another. But to get the perfect timing when one line was about to get hidden and the other appears was tedious af. All in all the 25 minutes cost me about the same number of hours. It’s just not feasible.
It’s nice to see a good application of ai. I hope my low end stuff will be able to run it.
I don’t mind the idea, but I would be curious where the training data comes from. You can’t just train them off of the user’s (unsubtitled) videos, because you need subtitles to know if the output is right or wrong. I checked their twitter post, but it didn’t seem to help.
subtitles aren’t a unique dataset it’s just audio to text
They may have to give it some special training to be able to understand audio mixed by the Chris Nolan school of wtf are they saying.
No, if you have a center track you can just use that. Volume isn’t a problem for a computer listening to it since they don’t use the physical speakers.
I took the other comment as a joke but this is accurate and interesting additional information!
I was part of that!
Fuck no. Leave the subtitles alone. Make people learn something, like searching and applying subtitles files or actually make them write their own and give back, for a change.
actually make them write their own and give back
Yeah just cure your deafness and write the subtitles yourself, idiot
I am fortunate to not be deaf (yet) but I have, in fact, writen the subtitles for various titles and submited it to Open Subtitles, both in English and my own native language.
Is it contribution enough for you?
this is great news.
MPV is possible?
Existed for a while without much fanfare. VLC doesn’t innovate. And it’s basically dead on Linux.
VLC doesn’t innovate. And it’s basically dead on Linux.
Afaik it’s false. It had a major update recently and it’s installed on a lot of Linux systems.
installed on a lot of Linux systems.
Fake info. It would be fake too if you made the opposite claim. Because such info is simply not available.
VLC being an MPlayer clone with better branding has been a running half-joke for decades.
The latest released version of VLC is not compatible with ffmpeg versions > 4.4 🤗. Some distros have actually considered dropping the package for that reason. Maybe some did, I don’t know. But if the situation doesn’t change, some definitely will.
And VLC 4, which those who still care for some reason have been waiting for it to be released for years, is centered around libplacebo, a library that was factored out of mpv 😎 .
I’m not emotionally charged against VLC or anything. In fact, I occasionally use it on Android. But what’s stated above is just facts.
If vlc switches its back end to use what MPV uses, I’d be more likely to use vlc. Sounds like a good move.
Rendering only.
I mean the only thing I dont like about vlc is the rendering. What else does MPV have on vlc?
This is great timing considering the recent Open Subtitles fiasco.
Huh?
Open Subtitles now only allows 5 downloads per 24 hours per IP. You have to pay for more.
Kind of annoying when searching for the exact sub file for the movie file you have.
Especially when half those subtitle files appear to be AI generated anyway, or have weird Asian gambling ads shoved in.
Glad MKV seems to be the standard now, and include subs from the original sources.
Oof. Well, they have to make money somehow. And probably there were people abusing the site. It wouldn’t surprise me for example if many did not cache the subtitles but had them on demand for videos.
Use https://opensubtitles.com/, rather than https://opensubtitles.org/.
This what be news good is for now
Perhaps we could also get a built-in AI tool for automatic subtitle synchronization?