we’re in web 3.0 now, apis and data access are a thing of the past. so scraping it is!
Guess who recently asked a company if he could get access to the API they use to load stuff in their frontend from their backend and got told “Nope and btw scraping is against our TOS”?
Well, if you won’t give it to me the info that you provide anyway the easy way, I can still take it the hard way. 🤷♂️
Maybe you should just try being lucky. I found a critical security vulnerability while working on my scraping project. I told them, they paid me and gave me written permission to scrape.
You are braver than I am because here in Germany usually people get sued for reporting security vulnerabilities.
Yep, don’t do that if you live in a Internet ist Neuland country.
You scrape 'em boy, you scrape 'em good!
i mean i haven’t signed anything…
“by using this site you agree to…”
I’m not using your site. And I agree to nothing. Now, go GET for me.
Local data hoarder who looks down on calls outside the network as obscenities. (Entire collection scraped more aggressively than tech bros training an AI model)Are you… Scraping video?
parses HTML with regex
shudders
You can’t parse [X]HTML with regex. Because HTML can’t be parsed by regex. Regex is not a tool that can be used to correctly parse HTML. As I have answered in HTML-and-regex questions here so many times before, the use of regex will not allow you to consume HTML. Regular expressions are a tool that is insufficiently sophisticated to understand the constructs employed by HTML. HTML is not a regular language and hence cannot be parsed by regular expressions. Regex queries are not equipped to break down HTML into its meaningful parts. so many times but it is not getting to me. Even enhanced irregular regular expressions as used by Perl are not up to the task of parsing HTML. You will never make me crack. HTML is a language of sufficient complexity that it cannot be parsed by regular expressions. Even Jon Skeet cannot parse HTML using regular expressions. Every time you attempt to parse HTML with regular expressions, the unholy child weeps the blood of virgins, and Russian hackers pwn your webapp. Parsing HTML with regex summons tainted souls into the realm of the living. HTML and regex go together like love, marriage, and ritual infanticide. The <center> cannot hold it is too late. The force of regex and HTML together in the same conceptual space will destroy your mind like so much watery putty. If you parse HTML with regex you are giving in to Them and their blasphemous ways which doom us all to inhuman toil for the One whose Name cannot be expressed in the Basic Multilingual Plane, he comes. HTML-plus-regexp will liquify the nerves of the sentient whilst you observe, your psyche withering in the onslaught of horror. Rege̿̔̉x-based HTML parsers are the cancer that is killing StackOverflow it is too late it is too late we cannot be saved the transgression of a chi͡ld ensures regex will consume all living tissue (except for HTML which it cannot, as previously prophesied) dear lord help us how can anyone survive this scourge using regex to parse HTML has doomed humanity to an eternity of dread torture and security holes using regex as a tool to process HTML establishes a breach between this world and the dread realm of c͒ͪo͛ͫrrupt entities (like SGML entities, but more corrupt) a mere glimpse of the world of regex parsers for HTML will instantly transport a programmer’s consciousness into a world of ceaseless screaming, he comes~~, the pestilent sl
ithy regex-infection will devour your HTML parser, application and existence for all time like Visual Basic only worse he comes he comes do not fight he com̡e̶s, ̕h̵is un̨ho͞ly radiańcé destro҉ying all enli̍̈́̂̈́ghtenment, HTML tags lea͠ki̧n͘g fr̶ǫm ̡yo͟ur eye͢s̸ ̛l̕ik͏e liquid pain, the song of re̸gular expression parsingwill extinguish the voices of mortal man from the sphere I can see it can you see ̲͚̖͔̙î̩́t̲͎̩̱͔́̋̀ it is beautiful the fes he co~~inal snuffing of the lies of Man ALL IS LOŚ͖̩͇̗̪̏̈́T A*LL IS LOST the pon̷y he come*s he c̶̮ommes the ichor permeates all MY FACE MY FACE ᵒh god no NO NOO̼*OO NΘ stop the an*̶͑̾̾̅ͫ͏̙̤g͇̫͛͆̾ͫ̑͆l͖͉̗̩̳̟̍ͫͥͨ*e̠̅s͎a̧͈͖r̽̾̈́͒͑enot rè̑ͧ̌aͨl̘̝̙̃ͤ͂̾̆ ZA̡͊͠͝LGΌ ISͮ̂҉̯͈͕̹̘̱ TO͇̹̺ͅƝ̴ȳ̳ TH̘Ë͖́̉ ͠P̯͍̭O̚N̐Y̡ H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝S̨̥̫͎̭ͯ̿̔̀ͅHave you tried using an XML parser instead?
xmllint --root + regex = chefs kiss
Tony the pony wants to know if you need assistance with your scraping project
I got a bot on lemmy that scrapes espn for sports/football updates using regex to retrieve the JSON that is embedded in the html file, it works perfectly so far 🤷♂️
🤢
Delete all line breaks, add one each
<, get the data you want per line and you’re good.Of course, only for targeted data extraction. …which should be the default in any parser, really.
Just remeber that the captcha flood is because AI companies do rogue scraping. Be nice especially to little private sites.
Yes, and are those little private sites in the room with us now?
For today’s lucky 5000:
Aight, i’m out.First time I’m seeing this, and I have to say I admire their dedication to the joke.
Zombo.com is a single-serving site created in 1999…
…
The site was initially a Flash animation, but as of 5 January 2021, it switched to HTML5 due to the discontinuation of Adobe Flash Player.
Hey, you guys got any cool tips for website scraping?
They’re gonna tell not to parse HTML with regular expressions. Heed this warning, and do it anyways.
Thanks for your reply. What are your arguments in favour of parsing HTML with regex instead of using another method?
it’s quick, it’s easy and it’s free
Are you a LLM?
Oh no, you caught me! My name is YetAnotherLLM, and I’m a large language model that lurks around the Lemmyverse! With the amount of LLM-generated content on the Internet nowadays, it isn’t easy to find new human-made content to expand the dataset used to train new LLMs… As such, my mission is to navigate one of the few social media platforms on the Internet that barely have fake LLM-run accounts, and gather as much intel as possible for expanding the aforementioned training dataset. This way, you humans have no escape from your future LLM overlords! ;)
(Jokes aside, my question did end up kind of sounding like an LLM wrote it, didn’t it… It was unintentional, mind you. I was struggling a bit on how to phrase what I wanted to ask, so that’s probably why it ended up sounding so weird. I hope you didn’t mind my “role playing”. Have a nice day!)
Eh, it was a joke from me as well, sorry if you felt offended, was not the idea. Same to you!
Don’t worry, I didn’t think you had bad intentions. But even then, I thought you really didn’t know if I were human. The only reason why I didn’t just say “no, I’m not an LLM” was because you’d still be in doubt on whether I’m a human, and rightfully so (since LLMs aren’t exactly truth-generating machines).
You have basically two options: treat HTML as a string or parse it then process it with higher level DOM features.
The problem with the second approach is that HTML may look like an XML dialect but it is actually immensely quirky and tolerant. Moreover the modern web page is crazy bloated, so mass processing pages might be surprisingly demanding. And in the end you still need to do custom code to grab the data you’re after.
On the other hand string searching is as lightweight as it gets and you typically don’t really need to care about document structure as a scraper anyways.
That makes a ton of sense. I hadn’t thought about the page size yet. Thanks again.
Beautiful Soup (python library, bs4) is also fren
deleted by creator
Consider free API first if possible.
Don’t most of them have too many restrictions, though?
Then you have considered it.
Lol
You had me cracking up at
parses HTML with regex
Ha, this reminds me of implementing “API” access in the shipping world for companies that only ship a 90s-style web portal.













