Scraper - v2.0.3

Scraper

Badges

Scraper
Getting Started

About

This tool uses its own parsing of the HTML into JSON, then this package scrolls through it to find the desired information.

Read this for more information. Or visit this website for the API types

License

This tool is licensed under the Apache 2.0 License.
See the LICENSE file for more information.

Getting Started

Basic information

The package breaks down the HTML into JSON nodes, following the concept that an HTML tag is represented as a JSON object/node.

Each node represents an HTML tag with it's tag name, attributes and children nodes. The children nodes may contain text as string values

Note that the parser completely ignores HTML comments ()

Usage

First and most importantly, we need to have our HTML string ready in a variable.

If you're going to use the same HTML string for multiple uses, it's better to parse it yourself and then pass the JSON output to the functions (so the HTML string will be parsed only once). The following example shows how to do this:

import { parse } from '@disqada/scraper'

// Use another package to fetch the HTML from the web
const html = `
  <html>
    <head>
      <title>Test Page</title>
    </head>
    <body>
      <h1 id="title">Hello, world!</h1>
      <p class="content">This is a test paragraph.</p>
    </body>
  </html>
`
const nodes = parse(html)

// Rest of the code ...

A full node

import { findNode } from '@disqada/scraper'

const node = findNode(nodes, {
  tag: 'h1',
  attr: {
    key: 'id',
    value: 'title'
  }
})

// node = {
//   type: 'element',
//   tagName: 'h1'
//   attributes: [{key: 'id', value: 'title' }],
//   children: []
// }

A text value

import { grabText } from '@disqada/scraper'

const text = grabText(nodes, {
  tag: 'p'
})

// text = 'This is a test paragraph.'

TextOptions

The function grabText can be given a TextOptions object that specifies some configurations for the search process. Note that it's optional.

Click on the blue highlighted TextOptions to read more.

An attribute value

import { grabAttr } from '@disqada/scraper'

const attr = grabAttr(nodes, { tag: 'p' }, 'class')

// attr = 'content'

CLI commands

download

You can download an HTML file and its parsed JSON file under the scrap folder in the root path of your project outside runtime by calling the download command CLI.

Arguments

Note that either --url or --path must be given

Arg name	required	Description
`--file`	true	Name of the downloaded HTML and parsed JSON file
`--url`	false	Link of the web page
`--path`	false	Path of a local HTML file (the HTML file will be copied to the `scrap` folder)

Examples

scraper --url='https://example.com/sample' --file='sample'

scraper --path='./samples/v1/index.html' --file='sample1'