Scraper URLs

Scraper URLs are the scraping targets of our scrapers in Watchdog. On this page, we'll dive into the different url endpoints you can use to manage urls programmatically.

The url model

The url model contains all the information about the url that our scraper will scrape.

Properties

Name
id
Type
integer
Description
Unique identifier for the url.
Name
name
Type
string
Description
User specified name for the url, used to identify the url in the dashboard.
Name
url
Type
string
Description
The url that our scraper will scrape.

GET/v1/scraper/

List all urls

This endpoint allows you to retrieve a list of all your urls.

Request

GET

/v1/scraper/

curl -G https://watchdog-api.zeroruka.moe/v1/scraper/ \
  -H "Authorization: Bearer {token}" \

Response

{
  "1": {
    "name": "Test",
    "url": "https://www.ebay.com.sg/sch/i.html?_from=R40&_nkw=test&_sacat=0&_sop=10&_ipg=120",
    "added_by": "[{2: {'username': 'test1', 'scraper_status': False, 'admin': False, 'chat_id': 818971977}}]"
  },
  "2", {
    // ...
  }
}

PUT/v1/scraper/

Add a url

This endpoint allows you to add a url to your list of urls for our scraper to scrape.

Required attributes

Name
url
Type
string
Description
The search phrase for the listings you want to scrape. It will be automatically converted to a url.
Name
name
Type
string
Description
User specified name for the url, used to identify the url in the dashboard.

Request

POST

/v1/scraper/

curl https://watchdog-api.zeroruka.moe/v1/scraper \
  -H "Authorization: Bearer {token}" \
  -d url="malaya notes" \
  -d name="malaya notes"

Response

{
  "message": "Successfully added Url(id = 4 name = malaya notes, url = https://www.ebay.com.sg/sch/i.html?_from=R40&_nkw=malaya+notes&_sacat=0&_sop=10&_ipg=120)"
}

DELETE/v1/scraper/

Delete a url

This endpoint allows you to delete a url from your list of urls on your account.

Required attributes

Name
url
Type
string
Description
The url that you want to delete.

Request

DELETE

/v1/scraper/

curl -X DELETE https://watchdog-api.zeroruka.moe/v1/scraper/ \
  -H "Authorization: Bearer {token}"
  -d url="https://www.ebay.com.sg/sch/i.html?_from=R40&_nkw=malaya+notes&_sacat=0&_sop=10&_ipg=120"

Response

{
  "message": "Successfully deleted Url(id = 3 name = Test, url = https://www.ebay.com.sg/sch/i.html?_from=R40&_nkw=test&_sacat=0&_sop=10&_ipg=120)"
}