13 years ago my god. I wonder what Jon Skeet is doing these days.
I remember when he passed me in the reputation ranking back in the early days and thinking that I needed to be a little bit more active on the site to catch him lol.
In practice, if your target is very limited and consistent, it's probably fine. But as a general statement about someone's behavior, it really sounds like someone is wasting a lot of time and regularly getting sub-par results.
Much less beholden to arbitrary rules also. Way too many times companies will just up and lift their API access or push through restrictions. No ty, I'll just access it myself then
Outdated and unsupported and hasn't been replaced yet but is the standard way to use the service.
Lots of authorization tokens.
The example in the docs doesn't work (if there is one).
You have no idea where the online tutorial got the information because it doesn't have links to resources and the docs have barely anything even though its giant.
Uses asynchronous programming to make it faster but its still much much slower then scrapping without asynchronous programming.
I created a shitty script (with ChatGPT's help) that uses Selenium and can dump a Confluence page from work, all its subpages and all linked Google Drive documents.
When a customer needs a part replaced, they send in shipping data. This data has to be entered into 3-4 different web forms and an email. This allows me to automate it all from a single form that has built in error checking so human mistakes are limited.
Company could probably automate this all in the backend but they won’t :shrug:
Websites and services create APIs for programmers to use them. So Spotify has code that let's you build a program that can use its features. But you need a token they give you after you sign up. The token can be revoked and used to monitor how much of their service you're using. That way they can restrict if its too much.
Scraping is raw dogging the web slut you met at the cougar ranch who went home with you because you reminded her of her dog
'Scraping' is the process of anonymously and programmatically collecting data from a webpage(s), often without the website's permission and only limited to the content made publicly available. This is in contrast to using an API provided by the database owner which is limited by tokens, access volume, available end points etc.
There's a ton of money to be made from scraping, consolidating, and organizing publicly accessible data. A company I worked for did it with health insurance policy data because every insurance company has a different website with a different data format and data that updates every day. People will pay da big bux for someone to wrap all that messiness into a neat, consistent package. Many sites even gave us explicit permission to scrape because they didn't want to set up an api or find some way to send us files.
Right now, gathering machine learning data is hot, cause you need a lot of it to train a model. Companies may specialize in getting, say, social media posts from all kinds of sites and putting them together in a consistent format.
Most clear example is these apps that get you the best deals on hotels or flights, they compare prices by web scrapping. Obviously they take a cut in these transactions.
I really hope Libreddit switches to scraping, the "Error: Too many request" thing is so annoying, I have to click the redirect button in Libredirect like 20 times until I can actually see a post.
Still a better experience than Reddits official site tho.
Sorry, I'm ignorant in this matter. Why exactly would you want to scrape websites aside from collecting data for ML? What kind of irreplaceable API are you using? Someone please educate me here.
API might cost a lot of money for the amount of requests you want to send. API may not include some fields in the data you want. API is rate limited, scraping might not be. API requires agreement to usage terms, scraping does not (though the recent LinkedIn scraping case might weaken that argument.)
This kinda reminds me of pirating vs paying.
Using api = you know it will always be the same structure and you will get the data you asked for. Otherwise you will be notified unless they version their api. There is usual good documentation. You can always ask for help.
Scraping = you need to scout the whole website yourself. you need to keep up to date with the the websites structure and to make sure they haven't added ways to block bots (scraping). Error handling is a lot more intense on your end, like missing content, hidden content, query for data. the website may not follow the same standards/structuree throughout the website so you need to have checks for when to use x to get y. The data may need multiple request because they do not show for example all the user settings on one page but in an api call they would or it is a ajax page and you need to run Javascript scripts and click on buttons that may change id, class or text info and they may load data when you do x with Javascript so you need to emulate the webpage.
So my guess is that scraping is used most often when you only need to fetch simple data structures and you are fine with cleaning up the data afterwards. Like all the text/images on a page, checking if a page has been updated or just save the whole page like wayback machine.
My understanding is that the result of the LinkedIn case is that you can scrape data that you have permission to view but not to access data that you were not intended to. The end result that ClickWrap agreements are unenforceable.
So uh...as someone who's currently trying to scrape the web for email addresses to add to my potential client list ... where do I start researching this?
Start looking into selenium, probably in Python. It's one of the easier to understand forms of scraping. It's mainly used to web testing, though you can definitely use it for less... nice purposes.
Step one will be learning to code in any language.
Step two is using a library to help with it. HtmlAgilityPack has always been there for me. Don't use regex.