Skip to main content

Web Scraping REST API with Node, Express and Puppeteer

Web Scraping REST API with Node, Express and Puppeteer

Step 1. Create a Node project

Create a folder. Open the folder using a terminal.
Type: npm init and press enter
Answer or leave unanswered the questions asked by the program.
Your Node Project is ready.

Step 2. Install Puppeteer and Express 

Run npm install --save express in the terminal.
Run npm install --save puppeteer in the terminal.
This installs puppeteer as well as an instance of browser.

Step 3. Create Web Scraping Program

Create a file named app.js
Add the following lines to app.js
const express = require('express')

const scraper = require('./utils/scraper')

const app = express();

app.get('/reviews', (req, res) => {
scraper.extractReviews(req.query.url)
.then(data => {
res.status(200).json({ message: "success", data: data })
}).catch(err => res.status(500).json({ message: "Something went wrong. Could not fetch result." }))
});

app.listen(process.env.PORT || 3000, () =>
console.log('Example app listening on port!'),
);
Add a folder named utils
Add a file in utils named scraper.js
Add the following code in scraper.js
const puppeteer = require('puppeteer'); // import puppeteer

extractReviews = async (url) => {
const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.goto(url, {waitUntil: 'networkidle2'});
const reviewCount = await page.evaluate(() => document.querySelector('span[itemprop="reviewCount"]').getAttribute('content'));
let reviewArray = [];
if (reviewCount > 0) {
url = url+"&pagenumber=0&RSort=1&csid=ITD&recordsPerPage="+reviewCount+"&body=REVIEWS#CustomerReviewsBlock"
await page.goto(url, { waitUntil: 'load' });
reviewArray = await page.evaluate(() => Array.from(document.querySelectorAll('.review')).map(review => ({ reviewTitle: review.querySelector('.rightCol blockquote h6').textContent, reviewComment: review.querySelector('.rightCol blockquote p').textContent, reviewRating: +review.querySelector('.leftCol .itemReview dd .itemRating strong').textContent, reviewDate: review.querySelector('.leftCol .reviewer dd:nth-of-type(2)').textContent, reviewer: review.querySelector('.leftCol .reviewer dd:nth-of-type(1)').textContent })));
}
await browser.close();
return { reviewCount: +reviewCount, reviewArray: reviewArray, url: url };
};

module.exports.extractReviews = extractReviews
Run node app.js in terminal to start the server on localhost.
In browser open the following URL to test your Scraping API: http://localhost:3000/reviews/?url=http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=3415697
Tadaaaaa!
If it did not work. Let me know in the comments.

Comments

Post a Comment

Popular posts from this blog

Difference between .exec() and .execPopulate() in Mongoose?

Here I answer what is the difference between .exec() and .execPopulate() in Mongoose? .exec() is used with a query while .execPopulate() is used with a document Syntax for .exec() is as follows: Model.query() . populate ( 'field' ) . exec () // returns promise . then ( function ( document ) { console . log ( document ); }); Syntax for .execPopulate() is as follows: fetchedDocument . populate ( 'field' ) . execPopulate () // returns promise . then ( function ( document ) { console . log ( document ); }); When working with individual document use .execPopulate(), for model query use .exec(). Both returns a promise. One can do without .exec() or .execPopulate() but then has to pass a callback in populate.
1 comment
Read more

Two Ways of rendering in React Examples

Two Ways of rendering in React Examples 1 - Render twice HTML < div id = "p1" ></ div > < div id = "p2" ></ div > CSS . person { display : inline - block ; margin : 10px ; border : 1px solid # eee ; box - shadow : 0 2px 2px # ccc ; } JAVASCRIPT (Babel) function Person ( props ) { return ( < div className = ' person "> < h1 >{ props . name }</ h1 > < p > Age :{ props . age }</ p > </ div > ); } ReactDOM . render (< Person name = "VK" age = "29" />, document . querySselector ( '#p1' )); ReactDOM . render (< Person name = "HK" age = "28" />, document . querySselector ( '#p2' )); 2 Render all at once HTML < div id = "app" ></ div > CSS . person { display : inline - block ; margin : 1...
Post a Comment
Read more

Resolve: Uncaught TypeError: firebase.database is not a function

If you are getting the error: Uncaught TypeError: firebase.database is not a function Resolve it by including firebase-database.js in your html page as follows: <!-- The core Firebase JS SDK is always required and must be listed first --> <script defer src = "https://www.gstatic.com/firebasejs/6.2.4/firebase-app.js" ></script> <script defer src = "https://www.gstatic.com/firebasejs/3.1.0/firebase-database.js" ></script> That is it. Let me know if this was helpful.
Post a Comment
Read more