Web app to detect changes in content in a website.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 
 
Go to file
dgtlmoon 8aae4197d7
UI - Make tabs hoverable
2 years ago
.github Playwright extension added back to Dockerfile to resolve conditional fix Alpine (musl) based systems (#1087) 2 years ago
changedetectionio UI - Make tabs hoverable 2 years ago
docs Bumping VisualSelector example animation 2 years ago
.dockerignore Adding dockerignore 4 years ago
.gitignore Configurable "Browser Steps" when Playwright/Chrome is configured (enter text, scroll, wait for text, click button etc) (#478) 2 years ago
CONTRIBUTING.md update path for validation in the CONTRIBUTING.md (#1046) 2 years ago
Dockerfile Configurable "Browser Steps" when Playwright/Chrome is configured (enter text, scroll, wait for text, click button etc) (#478) 2 years ago
LICENSE Create LICENSE 4 years ago
MANIFEST.in Configurable "Browser Steps" when Playwright/Chrome is configured (enter text, scroll, wait for text, click button etc) (#478) 2 years ago
Procfile Tweaking heroku Procfile 3 years ago
README-pip.md Adds support for jq JSON path querying engine (#1001) 2 years ago
README.md Update README.md 2 years ago
app.json Adding heroku app.json app 3 years ago
changedetection.py Disable SIGCHLD Handler for now - keeping SIGTERM for DB writes 2 years ago
docker-compose.yml Option to Hide the Referer header from monitored websites. (#996) 2 years ago
heroku.yml Try simpler heroku recipe 3 years ago
requirements-dev.txt Installation via pip (#186) 3 years ago
requirements.txt BrowserSteps - More work on cleaner shutdowns of browser session 2 years ago
runtime.txt Tweak runtime Heroku version 3 years ago
setup.py Microsoft Windows installation fixes (#492) 3 years ago

README.md

Web Site Change Detection, Monitoring and Notification.

Live your data-life pro-actively, Detect website changes and perform meaningful actions, trigger notifications via Discord, Email, Slack, Telegram, API calls and many more.

Self-hosted web page change monitoring

Release Version Docker Pulls License

changedetection.io

Don't have time? Let us host it for you! try our $6.99/month subscription - use our proxies and support! , half the price of other website change monitoring services and comes with unlimited watches & checks!

  • Chrome browser included.
  • Super fast, no registration needed setup.
  • Get started watching and receiving website change notifications straight away.

Target specific parts of the webpage using the Visual Selector tool.

Available when connected to a playwright content fetcher (included as part of our subscription service)

Self-hosted web page change monitoring context difference

Easily see what changed, examine by word, line, or individual character.

Self-hosted web page change monitoring context difference

Perform interactive browser steps

Fill in text boxes, click buttons and more, setup your changedetection scenario.

Using the Browser Steps configuration, add basic steps before performing change detection, such as logging into websites, adding a product to a cart, accept cookie logins, entering dates and refining searches.

Self-hosted web page change monitoring context difference

After Browser Steps have been run, then visit the Visual Selector tab to refine the content you're interested in. Requires Playwright to be enabled.

Example use cases

  • Products and services have a change in pricing
  • Out of stock notification and Back In stock notification
  • Governmental department updates (changes are often only on their websites)
  • New software releases, security advisories when you're not on their mailing list.
  • Festivals with changes
  • Realestate listing changes
  • Know when your favourite whiskey is on sale, or other special deals are announced before anyone else
  • COVID related news from government websites
  • University/organisation news from their website
  • Detect and monitor changes in JSON API responses
  • JSON API monitoring and alerting
  • Changes in legal and other documents
  • Trigger API calls via notifications when text appears on a website
  • Glue together APIs using the JSON filter and JSON notifications
  • Create RSS feeds based on changes in web content
  • Monitor HTML source code for unexpected changes, strengthen your PCI compliance
  • You have a very sensitive list of URLs to watch and you do not want to use the paid alternatives. (Remember, you are the product)
  • Get notified when certain keywords appear in Twitter search results
  • Proactively search for jobs, get notified when companies update their careers page, search job portals for keywords.

Need an actual Chrome runner with Javascript support? We support fetching via WebDriver and Playwright!

Key Features

  • Lots of trigger filters, such as "Trigger on text", "Remove text by selector", "Ignore text", "Extract text", also using regular-expressions!
  • Target elements with xPath and CSS Selectors, Easily monitor complex JSON with JSONPath or jq
  • Switch between fast non-JS and Chrome JS based "fetchers"
  • Easily specify how often a site should be checked
  • Execute JS before extracting text (Good for logging in, see examples in the UI!)
  • Override Request Headers, Specify POST or GET and other methods
  • Use the "Visual Selector" to help target specific elements
  • Configurable proxy per watch
  • Send a screenshot with the notification when a change is detected in the web page

We recommend and use Bright Data global proxy services, Bright Data will match any first deposit up to $100 using our signup link.

Please star this project and help it grow! https://github.com/dgtlmoon/changedetection.io/

Installation

Docker

With Docker composer, just clone this repository and..

$ docker-compose up -d

Docker standalone

$ docker run -d --restart always -p "127.0.0.1:5000:5000" -v datastore-volume:/datastore --name changedetection.io dgtlmoon/changedetection.io

:latest tag is our latest stable release, :dev tag is our bleeding edge master branch.

Windows

See the install instructions at the wiki https://github.com/dgtlmoon/changedetection.io/wiki/Microsoft-Windows

Python Pip

Check out our pypi page https://pypi.org/project/changedetection.io/

$ pip3 install changedetection.io
$ changedetection.io -d /path/to/empty/data/dir -p 5000

Then visit http://127.0.0.1:5000 , You should now be able to access the UI.

Now with per-site configurable support for using a fast built in HTTP fetcher or use a Chrome based fetcher for monitoring of JavaScript websites!

Updating changedetection.io

Docker

docker pull dgtlmoon/changedetection.io
docker kill $(docker ps -a -f name=changedetection.io -q)
docker rm $(docker ps -a -f name=changedetection.io -q)
docker run -d --restart always -p "127.0.0.1:5000:5000" -v datastore-volume:/datastore --name changedetection.io dgtlmoon/changedetection.io

docker-compose

docker-compose pull && docker-compose up -d

See the wiki for more information https://github.com/dgtlmoon/changedetection.io/wiki

Filters

XPath, JSONPath, jq, and CSS support comes baked in! You can be as specific as you need, use XPath exported from various XPath element query creation tools. (We support LXML re:test, re:math and re:replace.)

Notifications

ChangeDetection.io supports a massive amount of notifications (including email, office365, custom APIs, etc) when a web-page has a change detected thanks to the apprise library. Simply set one or more notification URL's in the [edit] tab of that watch.

Just some examples

discord://webhook_id/webhook_token
flock://app_token/g:channel_id
gitter://token/room
gchat://workspace/key/token
msteams://TokenA/TokenB/TokenC/
o365://TenantID:AccountEmail/ClientID/ClientSecret/TargetEmail
rocket://user:password@hostname/#Channel
mailto://user:pass@example.com?to=receivingAddress@example.com
json://someserver.com/custom-api
syslog://

And everything else in this list!

Self-hosted web page change monitoring notifications

Now you can also customise your notification content!

JSON API Monitoring

Detect changes and monitor data in JSON API's by using either JSONPath or jq to filter, parse, and restructure JSON as needed.

image

This will re-parse the JSON and apply formatting to the text, making it super easy to monitor and detect changes in JSON API results

image

JSONPath or jq?

For more complex parsing, filtering, and modifying of JSON data, jq is recommended due to the built-in operators and functions. Refer to the documentation for more specifc information on jq.

One big advantage of jq is that you can use logic in your JSON filter, such as filters to only show items that have a value greater than/less than etc.

See the wiki https://github.com/dgtlmoon/changedetection.io/wiki/JSON-Selector-Filter-help for more information and examples

Parse JSON embedded in HTML!

When you enable a json: or jq: filter, you can even automatically extract and parse embedded JSON inside a HTML page! Amazingly handy for sites that build content based on JSON, such as many e-commerce websites.

<html>
...
<script type="application/ld+json">
  {"@context":"http://schema.org","@type":"Product","name":"Nan Optipro Stage 1 Baby Formula  800g","price": 23.50 }
</script>

json:$.price or jq:.price would give 23.50, or you can extract the whole structure

Proxy Configuration

See the wiki https://github.com/dgtlmoon/changedetection.io/wiki/Proxy-configuration , we also support using BrightData proxy services where possible

Raspberry Pi support?

Raspberry Pi and linux/arm/v6 linux/arm/v7 arm64 devices are supported! See the wiki for details

Support us

Do you use changedetection.io to make money? does it save you time or money? Does it make your life easier? less stressful? Remember, we write this software when we should be doing actual paid work, we have to buy food and pay rent just like you.

Firstly, consider taking out a change detection monthly subscription - unlimited checks and watches , even if you don't use it, you still get the warm fuzzy feeling of helping out the project. (And who knows, you might just use it!)

Or directly donate an amount PayPal Donate

Or BTC 1PLFN327GyUarpJd7nVe7Reqg9qHx5frNn

Support us!

Commercial Support

I offer commercial support, this software is depended on by network security, aerospace , data-science and data-journalist professionals just to name a few, please reach out at dgtlmoon@gmail.com for any enquiries, I am more than glad to work with your organisation to further the possibilities of what can be done with changedetection.io