This is partly because browser technology is constantly evolving which forces web scraping libraries to change with them.
Therefore many articles written about the topic reference deprecated libraries like PhantomJS and dryscrape which makes it difficult to find information that is up-to-date.
In this article we will show you how to scrape dynamic content with Python and Selenium in headless mode.
To make things more exciting we will do so by providing an example that has a real life use case.
Namely sending a notification to your Android or iOS device when certain TeamSpeak users enter or leave a given TeamSpeak server.
First make sure to install Selenium and the Simplepush library.
Then we need to make sure to have the ChromeDriver installed.
On Ubuntu or Raspbian:
Now we can start coding.
Did you notice how we use the multiprocessing library to start Selenium in its own process?
This is because otherwise our program could run out of memory since Python has difficulties collecting unused WebDriver instances.
By running them inside their own processes we make sure that all memory is released back to the OS once a process finishes.
Now if you run our little program, it will check tsviewer.com every five seconds to see if one of our friends joined or left the server (as defined by TSVIEWER_URL and TSVIEWER_ID).
If that was the case, it will send out a notification to the Simplepush key defined by SIMPLEPUSH_KEY.
The new version of our iPhone/iPad app adds support for Apple’s Critical Alerts.
Critical Alerts can bypass Do not Disturb mode and your mute switch.
If you want to use Critical Alerts on your iOS device you can edit or create an event and activate Critical Alerts there.
Also make sure to give Simplepush your permission for Critical Alerts if you want to use them.
The Raspberry Pi Zero W is an impressively small device with a single core 1GHz CPU and 512MB of RAM.
You can do all kind of great stuff with it.
For me the most interesting use cases happen in a headless setup where you do not connect your Raspberry to any peripherals.
See this great explanation on how to set up your Raspberry Pi Zero W without any monitor or keyboard.
When running your Raspberry Pi headlessly, you probably find yourself in the position where you want to get information from your Pi to yourself.
This is where Simplepush comes into play.
With Simplepush a few lines of bash will make it possible to send push notifications to your Android device.
The following example will send the amount of available memory on your Pi as a push notification to the user with the Simplepush key HuxgBB (replace this key with your own. You get your key by installing the Simplepush app - no registration required).
With Simplepush this also works with end-to-end encryption:
You can set your password in the encryption section of the app. This is also where you find your salt.
We also provide libraries for sending push notifications (normal and end-to-end encrypted) from within programming languages.
The ESP8266 is a low-cost system-on-a-chip (SoC) which comes with integrated WiFi and a full TCP/IP stack.
It is the perfect match for prototyping IoT projects or for just tinkering around.
With Simplepush you can send push notifications to your smartphone.
In this blog post you will learn how to combine these two components in a secure way in which nobody can intercept your data. Neither at your local network nor at nodes (e.g. Google) on the way from your ESP8266 to your smartphone.
For our task of sending encrypted notifications from the ESP8266 to a smartphone, we choose the Arduino environment.
Demo with a NodeMcu development board.
With Arduino libraries it is easy to do HTTPS requests from your ESP8266.
However at the time of writing, doing SSL on the ESP8266 still comes with certain problems attached.
For example you have to hard code the SHA1 fingerprint of the SSL certificate of the domain you want to connect to. This is a problem since certificates change and thereby their fingerprints change too.
When it comes to sending push notifications from the ESP8266 to your smartphone, we can sidestep this problem by using Simplepush’s encryption feature.
This means sending encrypted notifications (AES-CBC-128) via HTTP.
Just download our library and import the ZIP file in the Arduino IDE (as described here).
Then sending notifications (both encrypted and unencrypted) is as easy as shown in the following code snippet.
For password and salt fill in the password you chose and the provided salt (can be found in the encryption section of the app).