Category Archives: Computer

Extracting GPS data from the GoPro 7

While extracting the telemetry data from the GoPro is reasonably well documented I have found some gaps for getting the extracting utilities installed and when extracting and combining data from multiple files. These notes are for a Debian/Ubuntu installation in a BASH Shell.

Installing the gopro-utils

As I couldn’t find any straightforward instructions for installation, I’ll be going through everything I needed to do to get it working, you may have some of these packages installed already.

Now to get the gopro-utils and install them, I’m placing the source files into my Downloads directory, as well as the GPS data extractor we’ll be adding the other telemetry tools too, this is all a bit long winded.

Extracting the Data

You will need to find which stream in the video recording the data has been saved to, to find this use ffprobe to examine the recording and look for the stream that contains GoPro MET, for example:

You can see that what we are wanting is on stream 3, as far as I can tell this stays the same every time, I don’t know if it is different for other GoPro models.

This bash script extracts the GPS data in GPX format from all the GoPro GX recordings in the directory, other options have been commented out, if you are using Garmin VIRB edit there is also an option for use with that. The script creates two files, one that contains the raw data and another with the desired GPS data, the GPS output file has the same name as the recording, but in lowercase with a .gpx extension.

Merging GPX files

As the GoPro splits recordings into 4GB blocks, when extracting you will get a single GPX file for each recording. Many pages found by Google say that to create a single track from these all you need to do is append the files into one big file. This is wrong, what you end up with is a single file with many short tracks, when what you are after is one long track covering the entire journey. This bash script uses gpsbabel to create single merged file from the extracted GPX data, it creates a file called “gpsoutput.gpx”.

The next stage will be to write a script that combines all these and completes the job in one easy process.

Links and Sources

Using Inkscape for CNC designs

Inkscape is a free vector graphics editor for all major platforms, generally it is aimed at art and design users but it does have an option for generating G-Code for use in your favourite CNC software. While Inkscape doesn’t have many of the functions of proper CAD/CAM software it is an relatively easy place to start for creating basic designs, I have been using it to make boxes out of 3.5mm plywood.

These notes are based around my cheap CNC machine sold as an CNC3018 by a variety of Chinese manufacturers on Amazon and eBay, the included controller is a Woodpecker CNC board (Ardunio clone) I have upgraded to GRBL v1.1 and I am using version 0.92.4 (April 2019) of Inkscape with the included Gcodetools.

CNC Wood Cutting

This post focuses on setting up Inkscape for the CNC machine and producing the g-code from your drawing, it is not intended to be an Inkscape tutorial.

Document Setup

With a new drawing set your Document Size, this should be the same as your CNC bed, in my case this is 300 x 180mm. From the Inkscape menu go to File > Properties and in the Page Tab set the Display Units (millimeters in my case), the Orientation to Landscape and Page Size width: 300 and height: 180. In the Grids Tab set the Grid Units to mm and the Spacing X and Spacing Y to 1.0. Back on your main page, turn the page grid on with: View > Page Grid.

Inkscape Document Properties

By default Inkscape scales the stroke/line width when you resize a shape, to prevent this click the the fourth box from the right in the top icon bar “when scaling objects, scale the stroke width by the same proportion”

Turn Line Scaling off to prevent the line width changing when resizing

You can save this as a template, such as: CNC3018.svg or as the document default with: default.svg by saving the file to your templates directory:
On Linux and OS X: ~/.config/inkscape/templates/
On Windows: C:\Users\<username>\AppData\Roaming\inkscape\templates

The lines you draw will need to be the same width as the bit you are using in the CNC machine. Draw a rectangle, Right mouse click on the rectangle and select Fill and Stroke…. In the Fill Tab click the X – no paint box and on the Stroke Style tab set the width to that of the bit you are using – 1.5mm, subsequent rectangles will be in the same style, other shapes will need to be setup this way too. The colour of your lines should be black, there is some functionality for different colours to represent different depths but I have not yet worked out how to do this.

Setting the line width

Layout Tips for G-Code Routing

Remember to check the dimensions of the cuts, with an outside cut such as the width and height of a box side you need to measure for the inside of your rectangle, for holes in your box measure to the outside edge, Inkscape sets distances to the outside edge.

For positioning holes for switches and the like, I add thin lines 0.1mm thick as guides and make use of the width/height settings as well as the Object > Align and Distribute options. A pair of digital vernier calipers are a great aid to discovering the required sizes. Remember to delete these before G-Code encoding.

Using guidelines for accurate layout

When generating the G-Code each shape will be seen as an individual object, so lets say you want to have two sides of your box cut from a single sheet of plywood, this would be two rectangles abutting each other with a side to be cut overlapping. As it takes four passes to cut each shape 1mm at a time, this means it’ll take six passes down the centre. To fix this select both rectangles and then Path > Combine followed by Path > Difference to make a single object.

Outputting to G-Code

Now that you have completed your drawing, save your work then convert your objects to paths by selecting all objects then Path > Object to Path. You may also want to place your drawing near the bottom left of the document, as this is where the CNC router starts. Now using Gcodetools there are three things you need to do to produce the G-Code file. None of the Gcodetools windows close automatically when apply is clicked, you will need to do that yourself. From the Inkscape menu:

1. Extensions > Gcodetools > Tools Libary…
Select Tools Type: cylinder and click apply In the overlarge green box that appears you will need to set the tool diameter and feed speed.

Gcodetools Tools Library

This can be a bit fiddly as the text can become detached from the box and the settings lost, what seems to work most reliably for me is to change to Text Objects (F8) click on the numbers you want to change and once done go back to Select and Transform (F1). Resize the box afterwards to check that it is still working – if the green box moves but the text does not then Ctrl-Z a few times and try again.

Gcodetools Tools Library – Green background is detached from the text – this won’t work.
setting default used description
Diameter 10 1.5 tool bit diameter in mm
Feed 400 300 speed while cutting through the material in mm/second
Penetration angle 90 90
Penetration feed 100 100 Plunge speed in the material in mm/second
Depth Step 1 1 Depth of cut on each pass in mm

2. Extensions > Gcodetools > Orientation Points
This tells the g-code where to start, normally bottom-left on the CNC Set the following:
– Orientation type: 2-points mode
– Z Surface: 0mm – this is the top of your surface
– Z Depth: -3.4mm – this is the thickness of material to cut, a negative number

With the Orientation Points added.

3. Extensions > Gcodetools > Path to Gcode
This creates the G-code file, in the Preferences Tab set the following:
– File: output filename 
– Directory: output directory
– Z safe height: 5mm – height above the work surface when moving between cuts
The filename once set doesn’t change, an incremental number is appended to the output filename. Click the Path to Gcode Tab before clicking apply (this appears to be a bug).

Your image will be updated to show the g-code routing, give this a visual check to ensure that all objects have been coded and that it looks right, the path to be taken should be in colour and contain arrows showing the direction of the router.

If there are too many arrows or if a line has arrows pointing in different directions then there may be an object underneath, check on your original artwork, in the image with the three circles below you see that A has not been converted to a path with Path > Object to Path, B has a duplicate object underneath and C is correct.

Gcode check; A and B have problems, C is correct.

The generated G-Code does not appear to include the Spindle Motor Start command – So remember to start the spindle manually in your CNC software before running the G-Code – its interesting how easily these bits break with a sideways load. Remember if you are cutting trough rather than engraving, don’t forget to put a layer to sacrifice between whatever you are making and the CNC’s bed, I use 5mm MDF/Fibreboard.

Links and Sources

FFMPEG for Video Conversion

FFmpeg is a command line program to manipulate, convert, record and stream video and audio, it is available for Mac, Linux and Windows. Here is a handy list of commands for reference, these have been tested with version 3.1.12 in a Debian Linux environment. I expect this list to grow over time as needs arise.

• Rescale a 4K video to 1080p

• Convert to H.264 (AVC) codec for use on uploading to YouTube, Vimeo, etc:

Using this codec reduces the time it takes for the video to be available after upload, however YouTube converts the file again to the VP9 codec and unless you have a popular channel, 100 subscribers or more, then this can take a few days or weeks and in the meantime your video can appear quite poor and blocky even when watching at 1080p, especially when there is a lot of movement like in a car dash-cam video. You can use FFmpeg to encode to VP9 webm format with this bash script:

This script is based on the encoding method shown in the WebM Wiki on my computer it is very slow and takes a quite a few hours to encode just nine minutes of video and the eventual results are so poor you’ll be wondering why you bothered.

• Convert to MP4 for use in Vegas Studio:

If you have a particularly old/odd video and get lots of pts has no value errors, then try this:

The -fflags +genpts option adds a Presentation Timestamp (PTS) to the frames, this must be before the -i as shown to work. Source.

• Set the video playback speed, this method adjusts the Presentation Timestamp (PTS) on each frame which may not work with older software. To slow down video divide the PTS by your required speed, this example slows the action by two times setpts=PTS/2.0. You can also reduce the number of dropped frames by increasing the frame-rate -r 50, in this case I went from 25fps to 50fps, but depending in the chosen speed frames may still be dropped.

Speed up your video by multiplying the PTS, in this case two times faster: setpts=PTS*2.0

• Convert file or extract audio from file into an MP3, the output is set to 128K constant bitrate

• Concatenate Video Files
This combines two video files, when using formats such as MP4 or MKV you will need to create intermediate files, otherwise only the first file will be included in the output:

• The opus not found error
When converting a file and you see an error like Could not find tag for codec opus in stream #1… you will need to state the output format

Links and Sources

Extracting MP3 audio from video files

Here is a small Bash script that converts any supported ffmpeg video format; such as .MKV, .MP4 or .MOV and extracts the audio to an .MP3 file, It will also split that MP3 file into chunks and put them in a convenient directory. You will need to install ffmpeg and mp3splt for your particular platform.

Example Usage:

This uses ffmpeg to convert “big fat file.mkv” to “big fat file.mp3” and then uses mp3splt to create a directory “big fat file” containing the files 01 – big fat file.mp3, 02 – big fat file.mp3, etc. The MP3 files will be encoded at 128k Constant Bit Rate and each file will be around 50 minutes in length. To install in Debian/Ubuntu use: sudo apt-get install ffmpeg mp3splt

mp3splt can find the audio in a quiet region near where the split is desired rather than midway through a word, this should make for much cleaner playback across tracks.

Alternative Method

This script gives the same results but uses ffmpeg to split the large MP3 file and then adds track numbering metadata using id3v2. To install in Debian/Ubuntu use: sudo apt-get install ffmpeg id3v2

Creating an Audiobook

Taking this further, I was thinking that it would be nice to have these converted into the M4B Audiobook format for use on my elderly iPod. The script below assumes that you have processed the files as above and have added metadata tags using a tool like mp3tag (yes I know this is for Windows).

To complete this we need to: Combine the multiple MP3 files into one big file, or read the original big file then convert that to M4B format at 96K bit and add chapter marks every ten minutes. For this I have used ffmpeg v3.2.12 and libmp4v2 (for the mp4chaps utility), to install in Debian/Ubuntu use: sudo apt-get install libmp4v2-dev mp4v2-utils ffmpeg

This script works best from a single MP3 file rather than from those that have been re-combined back into a single file, recombining the files caused ffmpeg to exclaim “invalid packet size” and “invalid data” errors. It is able to tell the difference between a directory and a single MP3 and processes the file accordingly, don’t forget to add metadata tags and cover art before you run the script.

When encoding to the M4B using a re-combined file I saw a few of these errors from ffmpeg:

These appear to be caused by the mp3splt program from when the original MP3 file was being split into 50 minute chunks, but I can’t hear any effect on the output.

Lots of information about the file can be gotten using mediainfo, to install in Debian/Ubuntu use: sudo apt-get install mediainfo, example use:

Links and References

Bluetooth Low Energy (BLE) on the Raspberry Pi

Bluetooth Low Energy – BLE – Bluetooth 4.0 is an industry-standard wireless protocol built for the Internet of Things – IoT, it is designed to provide connectivity for devices operating from low capacity power sources such as coin cell batteries.

Raspberry Pi2 with ASUS USB-BT400 Bluetooth 4.0 Dongle

In this introduction to BLE I’ll be configuring a Raspberry Pi2 computer to talk to a smart watch. We will be installing the latest version of BlueZ from source, enabling BLE support. This is not a tutorial on decoding the data from the watch I am just using it as an example, although I may write about decoding it in a future posting.

I am using a ASUS USB-BT400 Bluetooth 4.0 Dongle on a Raspberry Pi2 but this will work on any computer with a Debian based distribution. Your dongle must be BLE/Bluetooth 4.0 capable otherwise this won’t work. I am using an ID107HR activity tracker with pedometer and heart rate monitor, randomly chosen from the list of cheap ones available on Amazon. While using the Pi to talk to the the watch make sure Bluetooth on the phone is off as it can only connect to one device at a time.

The current distribution of Raspbian – jessie on the Raspberry Pi comes with version 5.23 of the BlueZ Bluetooth stack that’s rather old, dating from September 2014 which lacks many of the features we will be needing. The current version 5.44 of the BlueZ has many changes in the package with many familiar components such as hcitool and gatttool being depreciated, so I will be ignoring those and using the available commands, bluetoothctl, on the terminal.

Installing BlueZ

With Raspbian – jessie installed we will need to update the Pi make sure some packages are installed and then installing the latest version of BlueZ. But first, remove the installed version 5.23 of BlueZ:

Next, perform the traditional housekeeping updates then install the build tools and USB libraries. Those parts that are installed already will be automatically skipped.

Now, download the source code, at time of writing the current stable release is version 5.44, check the BlueZ site for the latest version.

Inside the BlueZ directory, configure, make (this takes a while), and install. The experimental option adds BLE support and enabling the library allows for python use later on:

Configuring and Starting BlueZ

At this stage we will need to check that the installation worked and that we can see your bluetooth dongle. With your bluetooth dongle in a USB port you should see it on your list of USB devices, here you see mine as device ID: 0b05:17cb ASUSTek Computer, Inc.:

You will also need to enable the experimental services, edit the file /lib/systemd/system/bluetooth.service and in the [Service] section change the ExecStart line to end with –experimental:

Start the bluetooth service, and while we are at it enable it to load on boot:

Once started, check the status of the bluetooth daemon with:

Should you need to, the service can be stopped and prevented from loading on boot with:

Finally, you may want to enable auto-power on for the device, to do so create this bluetooth config file:

and add these two lines:

You should restart the Pi at this point and check that the daemon has loaded properly with sudo systemctl status bluetooth

Testing BlueZ

Start the bluetooth controller, you should see your dongles MAC address and alias:

for first time use, try scanning to find your watch, if it doesn’t appear it is out of range, its battery is flat, or your dongle does not support BLE, here you can see it as ID107 HR:

bluetoothctl remembers your devices, so when you next use the program the watch appears on the list at the start. The controller has a number of options, these can be seen with help command. You can use show to view the status of your dongle:

The list of UUID’s show the services supported by the Dongle. Now we can power the dongle on, set the agent – this manages the connection, and then connect to the watch on which the bluetooth symbol will appear. Once connected there will be a pause then you will see a list of attributes supported by the watch, it is advertising the services available:

and now that we have connected we can ask for some info:

These UUID’s are used to describe the sevices available on the device, some are pre-defined and can be found in the a href=”https://www.bluetooth.com/specifications/gatt/characteristics” target=”_blank” rel=”noopener noreferrer”>GATT schema, others are vendor specific and unless they publicly release these, decoding can become rather difficult. There are four types of attribute:

  • Services – collections of characteristics and relationships to other services that encapsulate the behavior of part of a device
  • Characteristics – attribute types that contain a single logical value
  • Descriptors – defined attributes that describe a characteristic value
  • Declarations – defined GATT profile attribute types

Each attribute is identified by a 128 bit ID, for example, one of the characteristics from the list above: 00002902-0000-1000-8000-00805f9b34fb, the first eight bits are used as an unique identifier: 00002902 and are shown as UUID’s: 0x2902. Data is contained in services, each service has a number of characteristics that may contain further descriptions depending on the requirement of the characteristic. You can see how the data is mapped out in this chart:

Service Containers

A spreadsheet with the watch data reformatted and tastefully coloured to illustrates this. Observe the Service URL column, it looks a lot like a directory structure:

Here we see two services /service0008 and /service000c looking further into the second service: /service000c we see that it has four characteristics, and to of those have descriptors. We can interrogate the characteristics and descriptors to glean further information by selecting the attribute and reading, like so:

Which is all very nice, but not particularly helpful as the manufacturer has chosen to use custom, proprietary, UUID’s for the watch. We don’t know the instructions to send to have the watch realease its data.

Those Scripting BlueZ

Inevitably, you’ll be wanting to automate connections. This becomes easy with the automation scripting language expect. Install, then make a script file:

In this example the script forgets the watch, finds the watch, connects to the watch, gets some info and then disconnects:

in the script, send sends a command, don’t forget to add the carriage return – \r and expect is used to wait for a response within the timeout period, here it is set to 10 seconds. expect -re is using regex when looking for a reply, otherwise it uses a literal string. So much more can be done with expect and there are many tutorials, such as this one written by FluidBank.

More Bluetooth Data

For analysing bluetooth data a couple of very useful tools are available, Wireshark and Android data logging. I will go through the installation but not look at the data in any detail, this posting is getting a bit long. This Section is in two parts, installing Wireshark and Android Debug Bridge.

Sniffing with the Shark
Wireshark is a network and bluetooth packet sniffer, it shows you network and bluetooth traffic occurring on your Pi. Here is a quick installation method for a reasonably new version of Wireshark (v2.2.4) from the backports, answer yes to the question “Should non-superusers be able to capture packets?”:

and if you get a message about permissions, reconfigure the package and answer yes:

Start Wireshark and double click your bluetooth device on the list, in my case bluetooth0. There is not much to see as Wireshark will only see traffic between the watch and the Pi:

Wireshark Data Capture

Android Debug Bridge – ADB
For Anroid 4.2.2 and above, activate developer mode on the phone, go to Settings, tap About Phone and at the bottom of the list tap Build Number three times. Back in the main settings page Developer Options has appeared, tap developer and turn USB debugging On. With the phone plugged into a USB port a little Android head should appear in the information bar at the top-left of the screen. To begin we will need to install some udev rules written by Nicolas Bernaerts:

Install the android tools, confirm that you have at least version 1.0.31, and start ADB

At this point on the phone an allow USB debugging dialog will appear, give permission and always trust to authorise it. ADB will now show the device as a device:

If the device list is empty, with everything plugged in good and proper and the phone setup in developer mode, start your diagnosis by checking udev; open another terminal window and view logging with udevadm monitor –environment and reload with sudo udevadm control –reload I’m not entirely sure what I did to get it from ‘not working’ to ‘working’. If all else fails elevate yourself to root.

Data Capture
With ADB now setup we can capture the Bluetooth data being exchanged. With bluetooth off, in the Developer Settings find Enable Bluetooth HCI snoop log and turn it On. In the smartwatch app synchronise with your watch, once complete turn Bluetooth off manually – this is to minimise the amount of captured data. Don’t forget to turn logging off on the phone when done. To find where the log file has been stored and copy the file from the phone to the Pi use:

We can now use Wireshark to read the log file…

Wireshark reading the Android Bluetooth Log

This wasn’t quite the posting I originally had in mind, I wanted to decode the data from the watch for my own use, making something more useful, impressive graphs and charts, than that provided by the Android App VeryFit 2.0 but as the manufacturer has chosen to use proprietary GATT codes it makes the job that much harder. It may be much simpler to just buy an expensive FitBit and download the data from them. But with writing this I now know a few things that were previously unknown, and I hope that this has provided some light to your BlueZ (a pun!, right at the end!).

Links and Sources

Fixing the Arduino incoming network connections error on the mac

On the Apple Mac if you use the Teensy micro-controller with the Arduino IDE you may have come across a persistent firewall error message when starting the IDE, I have seen this error for quite a while over a range of system and software upgrades. I have applied this fix to:

  • OS X 10.10 Yosemite and above / macOS 10.12 Sierra
  • Arduino IDE 1.6.13 – all versions, at least 1.5 and above.
  • Teensydunio 1.33 – and older versions

The Arduino IDE is installed in the default applications folder, as is the Teensyduino. Some knowledge of using the terminal is required.

Symptoms

On your Apple Mac, you installed the Teensyduino software for the Teensy and now when you start the Arduino IDE this error message appears:

Do you want the application “Arduino.app” to accept incoming network connections?
Clicking Deny may limit the application’s behaviour. This setting can be changed in the Firewall pane of Security & Privacy preferences.

Incoming Network Connections Error

Cause

When the Arduino IDE is installed it includes a certificate to assure the system that everything is correct, the Teensyduino installation makes changes to the IDE configuration and this causes a mismatch and the signature in the certificate does not match the installation.

You can verify the failed certificate in the terminal with the spctl command:

Without the Teensyduino software installed, the certificate shows correctly:

Another check is to use codesign

Fix

To fix this, first we need to create a self-signed certificate. In finder Keychain Access can be found in Applications > Utilities > Keychain Access

Keychain Access certificates

From the menu choose: Keychain Access > Certificate Assistant > Create a Certificate… and set the following:

  • Name: anything useful, without spaces. You will be using this name later to apply the certificate – I used ‘arduino’
  • Identity Type: Self Signed Root
  • Certificate Type: Code Signing
  • Check the box “Let me override defaults”, this is important
Certificate Creation

click continue, and continue again past the security warning, then over the next few pages:

  • Serial Number: 1 – The serial and certificate name combination must be unique
  • Validity Period: 3650 – this will give you ten years
  • Email, name, etc: anything you like, or leave blank
  • Key pair info: set to RSA, 2048 bits
  • On the next four screens, from “Key usage extension” to “Subject Alternate Name Extension” accept the defaults
  • Location: login keychain.

Once created and back in the list, choose your certificate and from the menu go to File > Get Info (Cmd-i). In the Trust section at the top change: When using this certificate to Always Trust.

Trusted Certificate

Now that the certificate has been created, you need to apply it to the application, in terminal use the codesign command, this takes a few moments:

You’ll be asked to verify this, click ‘always allow’. To verify that your certificate has worked, check with codesign, using spctl will not work as this is a self-signed certificate.

Now, when you start the IDE on first run it will give you the allow/deny message again, click Allow and on subsequent use it will open as expected.

Links and Sources

Upgrading the Python Oracle Client

This is a follow up to one of my previous postings: Python and the Oracle Client. The main databases here are being upgraded to Oracle 12 and I’ve taken the opportunity to update the client used by my Python scripts, also its good practice to install new clients when old versions go out of support.

Current Setup

The system I am upgrading here has the following configuration, but this should work with any RPM based distribution, such as CentOS and SUSE :

  • Red Hat Enterprise Linux Server release 6.6 (Santiago)
  • Python 2.6.6
  • python connector: cx_Oracle – 5.1.2
  • oracle-instantclient11.2-basic-11.2.0.4.0-1.x86_64
  • oracle-instantclient11.2-devel-11.2.0.4.0-1.x86_64

To find the versions of your currently installed software:
$ python
Python 2.6.6 (r266:84292, Nov 21 2013, 10:50:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
>>> print cx_Oracle.version
5.1.2

and the Oracle client:
$ rpm -qa | grep oracle
oracle-instantclient11.2-basic-11.2.0.4.0-1.x86_64
oracle-instantclient11.2-devel-11.2.0.4.0-1.x86_64

Preparing

If you have the old versions installed you will need to do some tidying up by removing the client and python connector, version 11 of the client despite being RPM packaged had some non-standard elements. Use rpm to delete the old version of instant client, remove devel first:
$ sudo su
# rpm -ev oracle-instantclient11.2-devel-11.2.0.4.0-1.x86_64
# rpm -ev oracle-instantclient11.2-basic-11.2.0.4.0-1.x86_64

you may also need to remove the library reference from a previous installation:
# rm /etc/ld.so.conf.d/oracle.conf
# ldconfig

to remove the Python oracle connector, there are two methods. Manually, by finding the previously installed package deleting the files and editing the package list:
# find / -name cx_Oracle.py -print
/usr/lib/python2.6/site-packages/cx_Oracle-5.1.2-py2.6-linux-x86_64.egg/cx_Oracle.py
# cd /usr/lib/python2.6/site-packages
# rm -rf cx_Oracle-5.1.2-py2.6-linux-x86_64.egg

now edit the easy-install.pth file
# nano /usr/lib/python2.6/site-packages/easy-install.pth
and remove the line:
./cx_Oracle-5.1.2-py2.6-linux-x86_64.egg

Or do it the easy way, if you have pip installed:
# sudo pip uninstall cx_Oracle
easy_install does not have an uninstall option.

Installing

Download and install version 12 of the Instant Client and SDK (devel), these can be gotten from: http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html For Linux choose the correct flavour for your installed operating system: x86 or x86-64 for 64bit operating systems, you will need to register on the site gain access the files.
# rpm -i oracle-instantclient12.1-basic-12.1.0.2.0-1.x86_64.rpm
# rpm -i oracle-instantclient12.1-devel-12.1.0.2.0-1.x86_64.rpm

Now to install the python connector:
# easy_install cx-Oracle
or, the recommended method:
# pip install cx-Oracle
Installation for the version 12 client is much more straight forward than that for version 11.

Testing

A quick test to ensure that the expected versions appear, and that you can connect to the database.
Python 2.6.6 (r266:84292, Nov 21 2013, 10:50:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
>>> cx_Oracle.version
'5.2'
>>> oraConn = "<USERNAME>/<PASSWORD>@<DATABASE HOST>:<DATABASE PORT>/<SERVICE>"
>>> ocDB = cx_Oracle.connect(oraConn)
>>> ocDB.version
'12.1.0.2.0'

Sources

Water Splash Photography

Two drops of water colliding, frozen in time using the power of high speed flash photography produce an infinite variety of shapes. While this can be done with a pipette, a camera, a single flash gun, practice and good hand/eye co-ordination. I have an Arduino Uno and I am going to use it. IMG_8887What is happening in this picture? Two carefully timed water droplets have been released from above and are plummeting towards a bowl of water. The first drop has hit the water and is rebounding, just as the up-spout reaches its zenith, the second drop collides with the top resulting in a mushroom shaped splat, with the event captured in the camera with a frame of 1/10,000th of a second. In this post I’ll be sharing my experiences in creating these water drop images, I’ll be looking at the photography equipment, electronics, and technique.
up meets down

Photography Equipment

Camera: This can be any DSLR or advanced compact, it must have Bulb mode, and be triggerable by an electronic wired connection, some have an IR remote but I found this to be difficult to setup. Set the ISO to be around 200.

Lens: I use a 100mm Macro, with focus set to manual and image stabilisation off. The aperture is set high, at least f22 to give a suitable depth of field and improve image sharpness.

Tripod: A good solid one with easy to adjust ball head.

Flash: I use up to five flash guns for my photos, two for back light, one to give an under-light through the glass bowl, another for front light and finally one handheld. Rechargeable batteries for the flashes are recommended, I use 2400mAh NiMh Duracells.

The flash guns need to be in manual mode at their lowest power setting, this is to give the shortest duration of flash for the sharpest results. As you increase the flashes power the duration of the light emitted gets longer, causing burred images. On my Canon flash I set it to 1/128 second and on the Nissin Di622 set the EV to -1.5.

6N2A5694  6N2A5698

For connecting the Arduino to the flashes I use a 2.4GHz wireless remote trigger, with four receivers and a modified hotshoe mount attached to the transmitter Look for the Yongnuo RF-602 Remote Flash trigger on ebay, (not to be confused with the remote shutter release).  Most modern TTL flash guns appear to be missing the wired remote trigger connection that you can just plug into.

The flashes also have a built in slave trigger, where it sees that one flash has gone off so it set itself off too. On the Canon flashes this appears to only work in ETTL mode and can’t be used for this, but the Nissins work well.

Hardware

The frame is bits of wood held together with glue and stands about 75cm high this is to allow the water to accelerate and produce decent sized splashes. At the base is an extra large seed tray, the type without holes, to contain any spillages. This normally has a glass bowl full of water acting as the drip splash event zone. Halfway up the fame is mounted the laser and detector and at the top a reservoir of water and solenoid valve.

The reservoir is a one litre plastic storage tub from Poundland with a hole drilled in the base and a short length of 8mm PVC tubing hot glued into place. The tubing can be difficult to glue as its rather flexible, pushing down a short section of solid tube made from the outer of a disposable biro fixes that. This pipe is connected to the solenoid, observing the correct direction of flow marked on the valve.

The reservoir has a Mariotte Syphon fitted to the lid, this is to provide a constant and stable water pressure to the valve, the pipe from the lid ends about 2cm short of the reservoir base.
resevoir

Electronics

The Arduino and control electronics are all set to produce this photo taking sequence:Trigger Prototype

  1. press ‘play’ button on remote control
  2. lights out – dark room
  3. open shutter on camera
  4. solenoid releases two drops of water
  5. drips pass through laser detector – timer started
  6. drops arrive and do their thing
  7. flash guns triggered by timer – picture taken
  8. shutter closed on camera
  9. lights on

The electronic circuit can be broken down into these five blocks; lights, laser, IR receiver, solenoid control, flash control, camera control, each diagram shows the label name for the pin used rather than a pin number. The diagrams can be enlarged by clicking on them.

IR Receiver: The IR receiver allows use of an old TV remote control. My original design was to have a rotary dial and a small OLED display, but this simplified everything considerably. If you don’t have a spare remote one can be gotten from Poundland. I have the Arduino send any text output to a laptop on the USB port.

ir_rx

Laser: Warning: keep away from eyes, permanent damage can occur with exposure to any laser. The laser is used with a photo-transistor to detect drips of water as they plummet to their splash event. I used a small 3 Volt 5mW red laser with a built-in lens, I have added a resistor and diode in series to prevent over voltage as they’re a bit delicate. Although a modified laser pointer will do just as well. The TEPT4400 phototransistor is a type rated for visible light and has higher sensitivity to change than a photoresistor.

laser control
Laser Control

Lights: Warning: Mains Electricity Can Kill, this is to be avoided. If you are uncertain about this part then don’t do it. I rapidly found that working in darkness between shots just made life difficult, and finding the light switch became a hassle. To fix that I got a pre-made 5v Relay circuit and wired this up to a table lamp to provide some illumination. Using a standard wall socket and backbox connect live through the normally open side of the relay, and the neutral and earth to the socket.

 6N2A5700 6N2A5705

Remember to keep the electricity away from fingers (and any other body parts) and water.

Solenoid Control: I use a 12v solenoid, (search for “12v solenoid valve water arduino” on ebay, a couple of sellers have suitable models with connectors included). I use a mosfet transistor to switch the power, this has been detailed in one of my previous blog postings.

solenoid_control
Solenoid Control

Flash and Camera Control: The electronics for the camera and flash are closely related. Both use the ILD74 optocoupler to electrically isolate the camera and flash equipment from the Arduino. Although the camera focus connection is not used here I have included it as it may be useful later on.

camera control
camera and flash

The Canon camera has two different types of wired connection on the shutter release depending on the model of camera, a standard three pin 2.5mm jack or a N-3 connector (search for “canon N3 connecting cable” on ebay). A list of connectors for other makes of cameras can be found here.

canon remote
Remote shutter connections for Canon

Sound: Although not used here, this setup works well with a piezo microphone for use with popping water balloons and the like, use a buzzer that is enclosed in a plastic housing with a hole on top and buzzes when DC power is applied. Connect the output to an analogue pin on the Arduino, your software can use a very similar method to that for the laser.

sound
Sound Detect
water balloon pop
Balloon pop with sound detector

Setup and Use

Have plenty of dish cloths or towels to hand, this can get a bit moist. Keep an eye on your camera equipment making sure it doesn’t get wet.

For setting up a shot I use a steel ruler with a magnet stuck to it I set the water dripping to make sure it lands where I want on the magnet, the camera is then focused on the magnet, take away the ruler and you have your properly focused splash event.

IMG_8502
Splash Hat on Magnet

Add colour with food dies, adding these to the reservoir seems to work best and keeping the water in the splashdown area clean. Guar Gum thickens the water and makes larger drops and bigger splashes, you only need to add a small amount, about a teaspoon per litre and you’ll need to sieve out any lumps before use. Fluorescein is quite entertaining when used with a UV lamp, adding a green glow to your splashes. Adding diluted water based paints to the reservoir can add a lot of colour, but has a tendency to block the solenoid.

Sparkly backdrops can be gotten from the craft section in stationers. An A4 sized (21cm x 30cm) sheet is normally enough. Try bouncing the flash off the backdrop.

It is all about experimentation, expect to take lots of photos, many of which will be poor. Make notes of timings when you get good results, when you get a good shot, very small changes in timings can produce fairly dramatic effects.

Software

Here is an Arduino sketch, press Play on the remote to start a two drip sequence. adjust the amount of time in milliseconds between the laser detect and flash – flashWait with Volume +/-: +10,-10, Channel +/-: +5,-5, Fast Fwd/Rev: +2,-2, and the time betweenDrips with 4 (+1) and 7 (-1).

Links and Sources

My water drop photos on flickr:

Two Drops of Water

How-To: Raspberry Pi as a 3G/4G Router

Update: 15 April 2016 – Added information about which IP address to use and assigning static IP addresses for printers and servers

Recently I have needed to find an emergency alternative to my broadband due to the regional wide area network, Digital Region, being shut down, and the ISP Origin making a mess of getting all their cutomers onto ASDL. To get quickly back onto the internet, I have bought an ZTE MF823 4G Mobile Broadband Dongle as supplied by the badly named ‘three’ mobile phone company. As I have my own internal wired network, with multiple computers and ‘things’ there is a need to have something more sophisticated than just plugging the dongle into a single PC.

network diagram

Here is my recipe for setting up a Raspberry Pi as a router with an ZTE MF283 Dongle. In this setup all the computers are on a wired Ethernet connection using a switch for the network. The Pi has Raspbian Debian Wheezy installed (June 2014) with all the latest updates made. For testing, the dongle is plugged into the USB port via a powered hub, and the Pi connected to a switch with another PC running Linux Mint.

Which IP addresses to use?

In this How-To I am using the IP address range 192.168.2.xxx this is to avoid conflict with the cable router which uses the 192.168.1.xxx range (the DHCP server is switched off on the router). IPv4 addresses are split into three different ranges, the 192.168.xxx.xxx range – 192.168.0.0 to 192.168.255.255 gives a possible 65,536 addresses but for your home it is unlikely you’ll have more than 255 network devices, so we can simplify things by limiting the address range used to 192.168.2.xxx and avoid the troublesome world of subnet masking.

192.168.xxx.xxx is used as its been designated for use on private networks by the Internet Assigned Numbers Authority this is a well established convention and is best practice. Two other IPv4 address ranges are available for larger private networks: 172.16.0.0-172.31.255.255 and 10.0.0.0-10.255.255.255 with 1,048,576 and 16,777,216 available addresses respectively, the most suitable network class should be chosen for your network.

Setup the Dongle

This USB dongle has its own built in dialer so you do not need ppp or wvdial installed, it appears as a USB ethernet device on the Raspberry Pi. You will need a powered USB hub as the dongle can draw more power than the Pi can provide, the symptoms of too much of power being drawn will be the Pi behaving erratically or restarting unexpectedly.

With the dongle plugged in, check that it is recognised by the Pi with lsusb, it can be seen here as ‘ZTE WCDMA Technologies MSM’:

The device ID is 19d2. and 1405 is the mode, this should be 1405 – CDC ethernet. If it is not, try removing the micro-SD card and rebooting the Pi, the device modes available are:

  • 1225 – Default mode. USB Mass Storage Device + CD-ROM + card reader.
  • 1403 – Modem mode. RNDIS + Mass Storage Device.
  • 1405 – CDC ethernet
  • 0016 – Download mode

As the dongle also has a Mass Storage Device the Raspberry may not switch to CDC ethernet. If the mode does not change, try the following with usb-modeswitch:
$ sudo apt-get install usb-modeswitch
$ sudo usb_modeswitch -v 0x19d2 -p 0x1405 -d

I did not have to change the mode as it was correct already, and it didn’t change when I tried setting it as a Mass Storage device, I have not explored this any further.

When first plugged in the dongle was recognised as a ethernet device but it did not obtain an IP address:

if this is the case with you, add the following two lines to the end of sudo nano /etc/network/interfaces:
auto usb0
iface usb0 inet dhcp

the dongle provides its own address to the computer. Reboot, and you should see the obtained address:

The address 192.168.0.185 is now the internet address of the computer the dongle always assigns this address, there is also a useful web status page on http://192.168.0.1

Configuring the network

first of all enable ip4 forwarding, edit the file sudo nano /etc/sysctl.conf and uncomment the line:
net.ipv4.ip_forward=1
this will enable forwarding on reboot, you can also enable IP forwarding immediately with:
$ sudo sysctl -w net.ipv4.ip_forward=1

We now need to give the Pi a static IP address on the internal network. Edit sudo nano /etc/network/interfaces so you end up with a file that looks like this:

this gives the Pi a static IP address of 192.168.2.1.

DHCP

The next stage is to give the other computers on your network an IP address, this is done with a dhcp server:
$ sudo apt-get install isc-dhcp-server
you will need to configure dhcp sudo nano /etc/dhcp/dhcpd.conf, here is mine:

This will assign IP addresses in the range 192.168.2.50 to 192.168.2.150 to any computer connected to your network. I have used Open DNS for the Domain name Servers, if you wish to use google’s use:
option domain-name-servers 8.8.8.8, 8.8.4.4;instead.

I have also given my network printer a static IP address, it is still assigned by the DHCP server but never changes, the same would apply to any file servers and the like, I would assign static devices addresses that are outside your dynamically assigned range. Reboot the Pi and then your test computer.

Your test computer should now have an IP address (192.168.2.51), and the gateway point to the Pi (92.168.2.1):

Accessing The Internet

The final part is to have the incoming traffic on the the Ethernet port eth0, go out on the dongle usb0. This is achieved with iptables, a firewall and traffic router. Install with:
$ sudo apt-get install iptables
and you need to setup Network Address Translation, NAT and forwarding. This short bash script clears any old settings before applying the new rules:

Where LAN is your internal network, and WAN is the internet. The final line allows you access to the Dongle’s built in web status page from any browser on your internal network, just use: http://192.168.2.1:2525

Save the file in your home directory as ~/ipt.sh, make it executable and run the script.

$ chmod +x ~/ipt.sh
$ sudo ~/ipt.sh

From your test computer, you will now be able to access the internet.

Finally, you now need to have iptables reload when you start the Pi. Export the iptables settings to a file with:

$ sudo sh -c "iptables-save > /etc/iptables.ipv4.nat"

and create a file sudo nano /etc/network/if-up.d/iptables with the following contents:

and make it executable sudo chmod +x /etc/network/if-up.d/iptables

after a reboot you can see your iptables with sudo iptables -L and sudo iptables -t nat -L and you can see web traffic passing through the router with sudo tcpdump -i any -nn port 80.

Adding a Proxy Server

This is optional, but a transparent proxy server and cache may reduce the amount of traffic on your 3G/4G connection, mileage varies and the amount of data cached was less than I thought it would be, I also found that my Humax Freesat box really didn’t like the proxy server and wouldn’t update its TV schedules while it was on. I have used squid3 for this.
sudo apt-get install squid3
Update the squid3 configuration /etc/squid3/squid.conf so it has the following. The original is rather large, so you may want to make a copy and create a new one:

then restart squid3
sudo /etc/init.d squid3 restart
add the following iptables rule to redirect all traffic on port 80 to squid3:
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 3128
you should now be able to watch the web traffic being processed through squid3 with:
sudo tail /var/log/squid3/access.log -f
finish off by exporting your iptables again, so they are reloaded on reboot:
sudo sh -c "iptables-save > /etc/iptables.ipv4.nat"

Raspberry Pi - 4G Router

References and sources:

The Case of OpenCV and the Missing SURF

I have been wanting to have a play with the OpenCV computer vision framework on Python for a while and finally got some time to some experimenting, I am looking to have the computer recognise LEGO parts, after much research and mucking about it seems I should be using cv2.SURF and/or cv2.SIFT for what I want to do. However on Fedora 19 these are not included in the distribution RPM as they are nonfree in that they are not open source. Attempts to use SIFT or SURF result in the following error:

$ python
Python 2.7.5 (default, Nov 12 2013, 16:18:42)
[GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> print cv2.__version__
2.4.6.1
>>> i = cv2.SURF()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'SURF'
>>>
This is annoying as OpenCV will now need to be re-installed the old fashioned way, but there are instructions on the OpenCV site and I will be using them here with additional information I have discovered while following them.

Change to root and add the rpmfusion.org free and nonfree repositories as ffmpeg and libv41 are unavailble from Fedoras, followed by an update (it seems that Fedora 19 always has something to update):

& sudo su
# yum localinstall --nogpgcheck http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
# yum update

Remove the existing OpenCV packages:

# yum erase opencv*

There are a few packages to install in preparation to compiling the code, you may have some of all of these already but yum will skip those. Install the mandatory packages, these are for compiling the source and using GTK for the GUI:

# yum install cmake python-devel numpy gcc gcc-c++ gtk2-devel libdc1394-devel libv4l-devel ffmpeg-devel gstreamer-plugins-base-devel git wget

Install the optional dependencies, I have gone for the install everything approach, they may be useful later:

# yum install libpng-devel libjpeg-turbo-devel jasper-devel openexr-devel libtiff-devel libwebp-devel tbb-devel eigen3-devel python-sphinx texlive

Now, we are ready to get the latest source, here there are two ways to do this, install the latest development version using GIT, or download the latest stable version. My preference is to use the latest stable version.

With GIT, exit back to yourself from root, change to your home directory and download OpenCV:

# exit
$ cd ~
$ git clone https://github.com/Itseez/opencv.git
Cloning into 'opencv'...
$ cd opencv
$ mkdir build
$ cd build

Or using the latest build, at time of writing this is v2.4.7. Get this via the downloads page and save it to your home directory:

# exit
$ cd ~
$ tar -zxvf opencv-2.4.7.tar.gz
$ cd opencv-2.4.7
$ mkdir build
$ cd build

Now configure:

$ cmake -D CMAKE_BUILD_TYPE=RELEASE -D BUILD_PYTHON_SUPPORT=ON -D WITH_TBB=ON -D BUILD_NEW_PYTHON_SUPPORT=ON -D WITH_XINE=ON -D WITH_V4L=ON D WITH_OPENGL=ON -D WITH_OPENCL=OFF -D CMAKE_INSTALL_PREFIX=/usr/local ..

When complete you should check that your options agree with those displayed. There are many others available. Support for other programming languages may be missed out if they are not already installed. In my case those for Java, if I were to try OpenCV in Java I would need to install the appropriate Java packages using yum, then recompile OpenCV.

Now build and install, this can take a while:

$ make
$ sudo make install

You now need to move the module to anywhere on the Python Path, to find this:

$ python
Python 2.7.5 (default, Nov 12 2013, 16:18:42)
[GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print sys.path
['', '/usr/lib/python2.7/site-packages/PIL-1.1.7-py2.7-linux-x86_64.egg', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/PIL', '/usr/lib64/python2.7/site-packages/gst-0.10', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages', '/usr/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info']

The directory /usr/lib/python2.7/site-packages looks suitable:

$ sudo mv /usr/local/lib/python2.7/site-packages/cv2.so /usr/lib/python2.7/site-packages

And add your new installation to the PYTHONPATH, and add the export to the end of your .bashrc so it survives a reboot:

$ export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python2.7/site-packages
$ echo export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python2.7/site-packages >> ~/.bashrc

Now to test:

$ python
Python 2.7.5 (default, Nov 12 2013, 16:18:42)
[GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> print cv2.__version__
2.4.7
>>> i = cv2.SURF()
>>>

Sorted. Now, perhaps, I can get on with what I actually want to do….

Sources: