Introduction

I did this project because I wanted to track changes to a website. It helped me a lot, so I wanted to share my code with others.

How it works?

Currently, web-watcher only supports static HTML pages, it analyzes the structure of the page and the tags and defines a ratio which determines whether the page has changed or not. Thus, text changes are not detected, only changes in the structure of the page itself are detected.

Example

Not detected:

first.html
....
<h1>Hello world!</h1>
....

second.html
....
<h1>Goodbye world!</h1>
....

Detected:

first.html
....
<h1>Hello world!</h1>
....

second.html
....
<h2>Goodbye world!</h2>
....

Requirements

First of all, you have to create a new discord bot, generate a token and add the bot to the server of your choice.

If you don’t know how to so, here are some few steps:

Go to the discord developer portal and create a new application with the name of your choice:

App creation

Click on the Bot tab on the left and create a new bot:

Bot creation

Then go to the OAuth2 tab, check the Bot scope and the Send messages permission:

Bot permissions

Copy the generated URL in the middle of the screen, open it and add the bot to the server of your choice.

Congrats, you added the bot to your discord server!

Now all you have to do is to obtain your secret token. This is the token that will be used by web-watcher to connect to discord with the --token option or DISCORD_TOKEN environment variable. For more details check the usage page.

Return to the Bot tab and click on Click to Reveal Token:

Bot token

Copy it for the next steps and make sure to keep this token secret!

Install

You can install the pre-compiled binary (in several different ways), use Docker or compile from source.

Here are the steps for each of them:

Install the pre-compiled binary

Download the pre-compiled binaries from the releases page and copy to the desired location.

Running with Docker

You can also run it within a Docker container. Here as follows an example command:

export DISCORD_TOKEN=XXXXXXXXX....

docker run -d                                               \
    -e DISCORD_TOKEN                                        \
    -v web-watcher-data:/app                                \
    --name web-watcher                                      \
    docker.pkg.github.com/shellbear/web-watcher/web-watcher

The container is based on latest Go docker image.

Compiling from source

If you feel adventurous you can compile the code from source:

git clone https://github.com/shellbear/web-watcher.git
cd web-watcher

# get dependencies using go modules (needs go 1.11+)
go get ./...

# build
go build -o web-watcher .

# check it works
./web-watcher

Usage

You can pass customize some options.

> web-watcher --help
Web-watcher discord Bot.

Options:
  -interval int
        The watcher interval in minutes (default 60)
  -prefix string
        The discord commands prefix (default "!")
  -ratio float
        Changes detection ratio (default 1)
  -token string
        Discord token

Arguments

--interval

The watcher interval in minutes (default 60).

The watcher will check for website changes at this given interval.

--prefix

The discord commands prefix (default !).

--ratio

The web page changes ratio. Must be between 0.0 and 1.0.

Every x minutes the watcher will fetch the website page and compares it with the previous version. It will check changes and convert these changes to a ratio. If page are identical, this ratio is equals to 1.0, and it will decrease for every detected change.

--token

The discord bot token. The token can also be passed with the DISCORD_TOKEN environment variable.

If you don’t know how to generate one, a quick tutorial describes all the steps in the requirements page.

Commands

!watch [URL]

Add a URL to the watchlist.

!unwatch [URL]

Remove a URL from the watchlist.

!watchlist

Get the complete watchlist.

Note: you can customize the command prefix (!) with the --prefix parameter, for information, check the usage page.

Credits

This bot is built with these awesome dependencies:

go-difflib - Partial port of Python difflib package to Go
xxhash - Go implementation of the 64-bit xxHash algorithm (XXH64)
Gorm - The fantastic ORM library for Golang
DiscordGo - Go bindings for Discord