265 posts tagged with "web scraping"

As I have started writing this article, I didn’t expect it to end this way. Weeks after creating my first draft, Russian forces entered its western neighbor’s border and war raged in Ukraine.

Many questions have been raised. People around the world kept their eyes glued to their screens, waiting for more news about the invasion and looking for answers. I was no exception. I’ve seen the steady stream of content, talking about the different sides of the crisis, its ramifications, and its ripple effect.

Technology brings new opportunities. And every few years, there comes a new thing, which is hyped up so much that it is believed that it will change everything. After cryptocurrency, we got a new thing. Something that many thought to be an unstable bubble – about to pop any second; while others think of it as the future. Say hello to NFT.

HR or Human Resources is a department as important as any other in a business or a corporation. It helps manage the workforce so that workers are happy and a healthy environment is created, which helps achieve the organization's targets.

In a world where we have employed computers, AI, and the internet to better everything, why should HR lag behind? After all, if the employees are loyal and happy where they work, they are more likely to give their all while doing their jobs, all of which ultimately leads to growth. In order to do that, many big companies have made use of a newer approach, using public web data in order to improve human resource processes.

A flood of information and data runs on the web. In this global age, people use the internet to achieve almost everything. Everything they click on, things they search for, and the websites they spend the most time on translates to user behavior and tells us about what they like to see. Such information is invaluable and is waiting right there in front of us.

Collection of such data, ensuring proper data processing with the help of data collection can help your eCommerce business grow in ways unimaginable.

Today's internet is expanding at an unimagined rate, and the data roaming about the servers worldwide are extensively diverse and can be used to gather valuable insights, but how? The answer is Web Scraping! But what exactly is web scraping, and how to achieve your goals with data extraction?

Web scraping specialists are dealing with using proxy servers to overcome various anti-bot defenses every day. One of those protections is IP rate limiting, a primary anti-scraping mechanism.

Let's learn more about this protection method and the most effective ways of bypassing it.

Using a quality proxy server is the key to a successful web scraper. A variety of IPs along with their quality make it possible to collect data from various web sites without worrying about being blocked.

Still, many websites provide free proxy lists, so can the process of getting IP addresses from them be automated? Are free proxies good enough for web scraping? Let's check it out.

Making POST, PUT, and DELETE requests is a crucial web scraping and web testing technique. Still, this functionality is not included in Puppeteer's API as a separate function.

Let's check out the workaround for this situation and create a helper function to fix this out.

For the readable text of an already rendered page, start with page.$eval('body', body => body.innerText). Use textContent when you explicitly want descendant text that can include hidden content and script/style source. If the HTML already contains the data, a converter such as html-to-text can work without a browser.

Those methods produce different outputs. Below, the browser sees three dynamically populated products, while the raw HTTP response contains empty product placeholders. The examples show how that difference affects text extraction, and when to keep structured records instead of flattening the page.

Working with images in NodeJS extends your web scraping capabilities, from downloading the image with an URL to retrieving photo attributes like EXIF. How to achieve the image download and obtain the data?

This article is a part of the series on image downloading with different programming languages. Check out the other articles in the series:

• How to download images with Python?
• How to download images with Node.js?
• How to download images with PHP?
• How to download images with Go?
• How to download images with C#?
• How to download images with Java?
• How to download images with cURL?