Dec 30, 2015

Monitor URL Content for Changes: Endgame Family Netflix

Update1: evading Netflix geoblock just got more involved

I make use of some americanizing ssh tunneling for netflix and witty dialoge in Comedians In Cars Getting Coffee. So I need a cheap american host to relay the traffic and a rate of $6/year for a mini vps seems very price-conscious. Those offers are usually quickly gone, so I need a mechanism to notify me when it is available again.

The basic plumbing to do a keyword check on a webpage is simple: curl, grep and an if-else-construct. Cutycapt is optional and introduces dependencies, but helps with quickly debugging. If you don't run a postfix, a "transactional email services" comes in handy. Of the ones that I've been checking, mailgun from rackspace has a free basic plan and compared to others, a quick onboarding process. Transactional email: Let's do it. Register at mailgun and enter API keys.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
#!/usr/bin/env bash
#
# simple urlwatcher: check if term is present at site and send email in case it changes
# if you go for the cutycapt part, enable in cron with: @hourly xvfb-run --server-args="-screen 0, 1024x768x24" ~/urlwatch/urlwatch.sh >> ~/urlwatch/urlwatch.log 2>&1

timestamp=$(date --rfc-3339=seconds)
checkurl="https://subnetlabs.com/billing/cart.php?a=add&pid=50&systpl=impactvps53responsive"
searchterm="Out of Stock"

# make no error here, or it fails silently, you're welcome :)
screenshotdir="/home/km/urlwatch/"

# mg_sandbox="sandboxxyzcc.mailgun.org"
mg_apikey="api:key-xyz"
mg_hostname="external.jify.de"

curl -s "$checkurl" | tac | tac |\
if grep -q "$searchterm"; then

    echo $timestamp '- searchterm found, site unchanged.';

else
    echo $timestamp '- searchterm not found, possible change!'
    echo "Saving Captured HTML"
    curl -s --insecure "$checkurl" -o "$screenshotdir/$timestamp.html"

    echo "Capturing Screenshot"
    cutycapt --url="$checkurl" --out="$screenshotdir/$timestamp.png" \
             --min-width=1024 --min-height=768;

    # wait for cutycapt
    echo "Waiting for Cutycapt"
    sleep 3;

    echo "Sending Mail"
    curl -s --user "$mg_apikey" \
        https://api.mailgun.net/v3/$mg_hostname/messages \
        -F from='notjify urlwatcher <postmaster@'$mg_hostname'>' \
        -F to='me <me@example.com>'\
        -F subject='urlwatch for '"$searchterm"'' \
        -F text='item available again, order now at '"$checkurl"'' \
        -F attachment=@"$screenshotdir/$timestamp".png \
        -F attachment=@"$screenshotdir/$timestamp".html;
    echo "Mail sent"

    echo "Cleanup files"
    rm "$screenshotdir/$timestamp".html "$screenshotdir/$timestamp".png;
fi

Add it to your crontab and wait for that actionable transactional email trickling in

@hourly ~/bin/urlwatcher.sh >> ~/bin/urlwatcher.log 2>&1

Have VPS, want US Netflix

For a basic VPS setup, I run a basic Ansible script to flip some ssh options and add users with pubkeys to the machine.

To simplify the login procedure, add this to ~/.ssh/config:

Host netflix
  User hello
  Port 22
  IdentitiesOnly yes
  IdentityFile ~/.ssh/hello
  HostName proxy.example.com
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no

And open up the ssh tunnel via

ssh -D 8080 netflix

And Chrome (because EME, Flash) with

google-chrome --proxy-server="socks5://127.0.0.1:8080" --host-resolver-rule="MAP * 0.0.0.0, EXCLUDE 127.0.0.1:8080"

or with macOS

open -a /Applications/Google Chrome.app --args --proxy-server="socks5://127.0.0.1:8080" --host-resolver-rule="MAP * 0.0.0.0, EXCLUDE 127.0.0.1:8080"

  1. the proxying (either by ssh/vpn) for Netflix stopped working sometime in 2018 - they now check the originating netblock and if it's declared datacenter, the infamous proxy-error will be shown. Check the usage-type in ip2location.com/demo or ipinfo.io for quick lookup. If you want to query locally, the unsigned-int IP csv column in this file can be used with SQLs inet_ntoa(). Though in the lite version of the database "DCH" (Datacenter) or other types will not be listed, only public "PUB" ones. Residential IP classifiction: yet another niche I didn't realize to have been monetized (social media "automation") with ensuing arms-race. Anyway, the motorized coffee-fetching and laughing series showed up on european Netflix roughly the same time, so I ride on with Jerry and no withdrawl symptoms.