When scraping websites for all important data, developers are looking for, above all, privacy and security. Using proxies is the best and most effective way to do so without the risk of exposing your IP address, making it less likely that those sites will ban your address in future visits.
Axios is a popular JavaScript Node.js library frequently used for website scraping. It’s a popular tool used widely due to its fast and accurate downloading of website content.
Why Use Axios?
Axios is popular for very good reasons. For one, it’s a promise-based HTTP client for Node.js known for its adaptability and simplicity of use.
For example, Axios can be used both in a browser and Node.js. It’s a simple and intuitive tool, allowing you to easily configure it to your specific needs and send the correct requests. You can also set up default headers and other options that can be applied to all requests, while also configuring for different response types.
As a promise-based tool, it provides flexibility, particularly with asynchronous requests, and will return a promise which is then resolved when the response data is sent.
Why Use Proxies with Web Scraping or Data Fetching?
Proxies, often called proxy servers, act as intermediaries between a client (such as a web browser or a data scraper) and a target server or website. Their primary function is to facilitate data exchange between these two entities.
Here's how it works:
When a client makes a request, the proxy server takes and forwards it to the target server. The target server processes the request and sends a response back to the proxy server. Finally, the proxy server relays this response to the client.
This interaction might seem complex, but it serves several important purposes:
Anonymity: Proxies can hide the client's IP address from the target server, making it more challenging for the server to identify and block the client.
Data Scraping: Some clients use proxies to scrape data from websites, which can be against the website's terms of service. By routing their requests through a proxy, they aim to avoid detection and potential IP blocking.
Multiple Proxies: Some users employ multiple proxies in sequence, a practice known as proxy chaining, to further obfuscate their identity and reduce the risk of being detected or blocked.
Rotating IP Addresses: Certain proxy providers offer services with rotating IP addresses. This means that the proxy's IP address changes periodically, making it even harder for target servers to identify and block scraping activities.
Setting Up Node.js and Axios
Using Node.js with Axios requires a few critical steps to ensure the programs work smoothly together.
Prerequisites
You’ll need to install both Node.js and npm (Node Package Manager). You can download and install both from the Node.js website.
Initializing a Node.js Project
Once you have that in place, create a project within Node.js. This will be a new folder to store the code you’re about to write. To do that, use this code:
mkdir axios-proxy
cd axios-proxy
npm init -y
You’ll also want to have Axios installed to be able to make web requests. Then, create a app.js file (this will hold your project code):
npm install axios
touch app.js
You should now see various new files and folders within the "axios-proxy" directory. To finish up, you'll want to establish a start command by modifying the package.json file like this:
{
"name": "axios-proxy",
"version": "1.0.0",
"description": "",
"main": "app.js",
"scripts": {
"start": "node app.js"
},
"author": "",
"license": "ISC",
"dependencies": {
"axios": "^1.1.3"
}
}
And just like that, your new project is ready to go. Check if everything is in order by running this simple npm start command:
npm start
> axios-proxy@1.0.0 start
> node app.js
If it is, the code will run correctly. If not, you’ll need to go back and check your work.
Proxy + Axios: How to make it work
There are lots of public proxies you can use for this so you shouldn’t have a hard time finding one. For example, you can check ScrapingAnt's free proxies list.
To make it easier thought, you can run this proxy with Axios:
const axios = require('axios');
const proxy = {
host: '<proxy_host>',
port: <proxy_port>,
};
axios.get('https://api.ipify.org/?format=json', {
proxy: proxy,
})
.then(function (response) {
console.log(response.data);
})
.catch(function (error) {
console.log(error);
});
You should get a response like this:
{
"ip": "<your_proxy_ip>"
}
Note that free proxies, while affordable, may often be slow, unreliable, and unsafe. Using a reliable, professional proxy provides you with a secured endpoint while also rotating IP addresses by default.
Configuring and Managing Proxies
Expect to use authenticated proxies too. These are proxies that require a username and password before access. It's a more often used option for paid proxies, but it's also possible to find free proxies that require authentication.
Axios makes it easy to set up this authentication. You only need to provide your credentials using the "auth" property, and Axios will automatically incorporate them when you make a request. This way, the proxy knows who you are and can let you through.
It's handy for getting through networks or websites with restrictions or for making sure your online activities are secure.
Run this code to set it up:
const axios = require('axios');
axios.get('https://api.ipify.org/?format=json', {
proxy: {
host: '<proxy_host>',
port: <proxy_port>,
auth: {
username: '<proxy_username>',
password: '<proxy_password>',
},
},
})
.then(function (response) {
console.log(response.data);
})
.catch(function (error) {
console.log(error);
});
If you get a response with IP address of the proxy used, then you’re good to go.
Testing Proxies
To understand why and how to test proxies, it’s important to understand how proxies work.
Data scraping requires your site to send multiple requests to the target site. Those frequent queries can raise red flags and cause the target site to block your IP address. Other target sites use other tools, such as throttling requests and captcha tools, to prevent web scraping.
Using proxies allows you to circumvent these preventative measures safely. By using multiple IP addresses, your likelihood of being blocked diminishes greatly. When spreading out your requests over multiple IP addresses, you’re less likely to trigger the anti-scraping measures on these target sites.
So, testing web proxies is absolutely critical to ensure that your requests are delivering quickly and that the proxy service is reliable. Here are the three main things to look for in your proxy testing:
Speed
Slow proxies can adversely affect how successful you are at web scraping. If the proxy is too slow, you can end up with request timeouts, delayed and failed requests.
By using proxy testing tools, you can test the proxy and get a score and load time for each service tested. These measures can help you understand how quickly each proxy operates and if they will be able to meet your demands.
Security
Security is the main reason you’re using a proxy. You need a proxy that protects your privacy every time. There are various tools that will test the security of a proxy’s SSL certificate and help provide you with a security score.
Reliability
You do not want your proxy going offline when you need it. It’s best to choose a proxy that has a high rate of availability but has low instances of downtime.
Rotating Proxies
For best results, use a proxy service that uses rotating proxies. A rotating proxy process uses a separate IP connection whenever it connects to a target website. This process makes it look like the request comes from a different device or location each time.
Rotating proxies are used not only for web scraping but also for load testing, doing market intelligence research and making multiple social media accounts. So, yes, they are rather efficient.
Setting Up Proxy Logic
Creating a proxy rotator is something that can be done with some basic programming knowledge. The basic approach is to write a script that uses a variable that pulls a proxy from a list of options. The script then assigns a random IP address to each request of the target site.
Here’s an example:
const axios = require('axios');
const proxies = [
{
host: '<proxy_host_1>',
port: <proxy_port_1>,
auth: {
username: '<proxy_username_1>',
password: '<proxy_password_1>',
},
},
{
host: '<proxy_host_2>',
port: <proxy_port_2>,
auth: {
username: '<proxy_username_2>',
password: '<proxy_password_2>',
},
},
{
host: '<proxy_host_3>',
port: <proxy_port_3>,
auth: {
username: '<proxy_username_3>',
password: '<proxy_password_3>',
},
},
];
const proxy = proxies[Math.floor(Math.random() * proxies.length)];
axios.get('https://api.ipify.org/?format=json', {
proxy: proxy,
})
.then(function (response) {
console.log(response.data);
})
.catch(function (error) {
console.log(error);
});
Of course, you may have different needs, so you’ll want to tweak the code. That’s fine.
For example, you can keep track of the proxy availability, latency and other metrics. You can also use a proxy rotator service to handle the proxy rotation for you.
Troubleshooting Proxy Issues
Despite all this, there are still many things that can happen once you use a proxy to ping a website. Here are four of the most common issues:
Too Many IP Bans - If too many of your proxy IP addresses are getting banned quickly, you may need to switch to a different proxy type or one that is much more secure.
Failed Proxy Provider - If your proxy provider is failing or can’t connect to the desired site, check it on different browsers or separate devices. With free proxies, you may just need to wait, otherwise, consider switching to a more reliable provider to help with your scraping.
Poor Configuration - If you get an error message, your network or operating system is not configured properly.
Proxies Not Connecting - If you’re getting a proxy server error, then data is getting lost. Error messages can help find the source, but if not then you may need help from your proxy provider.
Proxy vs Web Scraping API
While proxies are a great way to protect your privacy and security, they are not the only option. Web scraping APIs are another option that can help you get the data you need without the hassle of proxies.
ScrapingAnt is a reliable web scraping API with a 99.99 percent uptime and an 85.5 percent anti-scraping avoidance rate. It offers multiple subscription plan options (as well as a free one) and low-latency rotating proxy tools.
With reliability, security, and speed, ScrapingAnt is ideal for any general web scraping and free for your own personal use.
Using Axios and Node.js is an effective, reliable, and powerful way to scrape websites for business intelligence and research. Coupling this with data extraction services like ScrapingAnt or rotating proxy providers is a smart way to protect yourself from IP blockers and bans that can hinder your precious research.
Last Words
Axios is a versatile and powerful tool for making HTTP requests in Node.js and browser-based applications. Its simplicity, flexibility, and support for various features, including authenticated proxies, make it a popular choice among developers.
Whether scraping data, accessing restricted resources, or simply making HTTP requests, Axios simplifies the process and enhances your ability to handle web interactions effectively. Here comes the caveat.
When using proxies and scraping websites, it's essential to respect the websites' terms of service and legal regulations to maintain ethical and legal practices. Web scraping without permission or violating a website's terms of service can disrupt the normal operation of a website, strain its resources, and potentially harm its users' experience.
Respecting the rules set by website owners is a matter of ethical conduct in the digital space.
Happy Web Scraping, and don't forget to use trial options or a free plan before you buy a subscription 💰