Category Archives: Computer

Bluetooth Low Energy (BLE) on the Raspberry Pi

Bluetooth Low Energy – BLE – Bluetooth 4.0 is an industry-standard wireless protocol built for the Internet of Things – IoT, it is designed to provide connectivity for devices operating from low capacity power sources such as coin cell batteries.

Raspberry Pi2 with ASUS USB-BT400 Bluetooth 4.0 Dongle

In this introduction to BLE I’ll be configuring a Raspberry Pi2 computer to talk to a smart watch. We will be installing the latest version of BlueZ from source, enabling BLE support. This is not a tutorial on decoding the data from the watch I am just using it as an example, although I may write about decoding it in a future posting.

I am using a ASUS USB-BT400 Bluetooth 4.0 Dongle on a Raspberry Pi2 but this will work on any computer with a Debian based distribution. Your dongle must be BLE/Bluetooth 4.0 capable otherwise this won’t work. I am using an ID107HR activity tracker with pedometer and heart rate monitor, randomly chosen from the list of cheap ones available on Amazon. While using the Pi to talk to the the watch make sure Bluetooth on the phone is off as it can only connect to one device at a time.

The current distribution of Raspbian – jessie on the Raspberry Pi comes with version 5.23 of the BlueZ Bluetooth stack that’s rather old, dating from September 2014 which lacks many of the features we will be needing. The current version 5.44 of the BlueZ has many changes in the package with many familiar components such as hcitool and gatttool being depreciated, so I will be ignoring those and using the available commands, bluetoothctl, on the terminal.

Installing BlueZ

With Raspbian – jessie installed we will need to update the Pi make sure some packages are installed and then installing the latest version of BlueZ. But first, remove the installed version 5.23 of BlueZ:

Next, perform the traditional housekeeping updates then install the build tools and USB libraries. Those parts that are installed already will be automatically skipped.

Now, download the source code, at time of writing the current stable release is version 5.44, check the BlueZ site for the latest version.

Inside the BlueZ directory, configure, make (this takes a while), and install. The experimental option adds BLE support and enabling the library allows for python use later on:

Configuring and Starting BlueZ

At this stage we will need to check that the installation worked and that we can see your bluetooth dongle. With your bluetooth dongle in a USB port you should see it on your list of USB devices, here you see mine as device ID: 0b05:17cb ASUSTek Computer, Inc.:

You will also need to enable the experimental services, edit the file /lib/systemd/system/bluetooth.service and in the [Service] section change the ExecStart line to end with –experimental:

Start the bluetooth service, and while we are at it enable it to load on boot:

Once started, check the status of the bluetooth daemon with:

Should you need to, the service can be stopped and prevented from loading on boot with:

Finally, you may want to enable auto-power on for the device, to do so create this bluetooth config file:

and add these two lines:

You should restart the Pi at this point and check that the daemon has loaded properly with sudo systemctl status bluetooth

Testing BlueZ

Start the bluetooth controller, you should see your dongles MAC address and alias:

for first time use, try scanning to find your watch, if it doesn’t appear it is out of range, its battery is flat, or your dongle does not support BLE, here you can see it as ID107 HR:

bluetoothctl remembers your devices, so when you next use the program the watch appears on the list at the start. The controller has a number of options, these can be seen with help command. You can use show to view the status of your dongle:

The list of UUID’s show the services supported by the Dongle. Now we can power the dongle on, set the agent – this manages the connection, and then connect to the watch on which the bluetooth symbol will appear. Once connected there will be a pause then you will see a list of attributes supported by the watch, it is advertising the services available:

and now that we have connected we can ask for some info:

These UUID’s are used to describe the sevices available on the device, some are pre-defined and can be found in the a href=”https://www.bluetooth.com/specifications/gatt/characteristics” target=”_blank” rel=”noopener noreferrer”>GATT schema, others are vendor specific and unless they publicly release these, decoding can become rather difficult. There are four types of attribute:

  • Services – collections of characteristics and relationships to other services that encapsulate the behavior of part of a device
  • Characteristics – attribute types that contain a single logical value
  • Descriptors – defined attributes that describe a characteristic value
  • Declarations – defined GATT profile attribute types

Each attribute is identified by a 128 bit ID, for example, one of the characteristics from the list above: 00002902-0000-1000-8000-00805f9b34fb, the first eight bits are used as an unique identifier: 00002902 and are shown as UUID’s: 0x2902. Data is contained in services, each service has a number of characteristics that may contain further descriptions depending on the requirement of the characteristic. You can see how the data is mapped out in this chart:

Service Containers

A spreadsheet with the watch data reformatted and tastefully coloured to illustrates this. Observe the Service URL column, it looks a lot like a directory structure:

Here we see two services /service0008 and /service000c looking further into the second service: /service000c we see that it has four characteristics, and to of those have descriptors. We can interrogate the characteristics and descriptors to glean further information by selecting the attribute and reading, like so:

Which is all very nice, but not particularly helpful as the manufacturer has chosen to use custom, proprietary, UUID’s for the watch. We don’t know the instructions to send to have the watch realease its data.

Those Scripting BlueZ

Inevitably, you’ll be wanting to automate connections. This becomes easy with the automation scripting language expect. Install, then make a script file:

In this example the script forgets the watch, finds the watch, connects to the watch, gets some info and then disconnects:

in the script, send sends a command, don’t forget to add the carriage return – \r and expect is used to wait for a response within the timeout period, here it is set to 10 seconds. expect -re is using regex when looking for a reply, otherwise it uses a literal string. So much more can be done with expect and there are many tutorials, such as this one written by FluidBank.

More Bluetooth Data

For analysing bluetooth data a couple of very useful tools are available, Wireshark and Android data logging. I will go through the installation but not look at the data in any detail, this posting is getting a bit long. This Section is in two parts, installing Wireshark and Android Debug Bridge.

Sniffing with the Shark
Wireshark is a network and bluetooth packet sniffer, it shows you network and bluetooth traffic occurring on your Pi. Here is a quick installation method for a reasonably new version of Wireshark (v2.2.4) from the backports, answer yes to the question “Should non-superusers be able to capture packets?”:

and if you get a message about permissions, reconfigure the package and answer yes:

Start Wireshark and double click your bluetooth device on the list, in my case bluetooth0. There is not much to see as Wireshark will only see traffic between the watch and the Pi:

Wireshark Data Capture

Android Debug Bridge – ADB
For Anroid 4.2.2 and above, activate developer mode on the phone, go to Settings, tap About Phone and at the bottom of the list tap Build Number three times. Back in the main settings page Developer Options has appeared, tap developer and turn USB debugging On. With the phone plugged into a USB port a little Android head should appear in the information bar at the top-left of the screen. To begin we will need to install some udev rules written by Nicolas Bernaerts:

Install the android tools, confirm that you have at least version 1.0.31, and start ADB

At this point on the phone an allow USB debugging dialog will appear, give permission and always trust to authorise it. ADB will now show the device as a device:

If the device list is empty, with everything plugged in good and proper and the phone setup in developer mode, start your diagnosis by checking udev; open another terminal window and view logging with udevadm monitor –environment and reload with sudo udevadm control –reload I’m not entirely sure what I did to get it from ‘not working’ to ‘working’. If all else fails elevate yourself to root.

Data Capture
With ADB now setup we can capture the Bluetooth data being exchanged. With bluetooth off, in the Developer Settings find Enable Bluetooth HCI snoop log and turn it On. In the smartwatch app synchronise with your watch, once complete turn Bluetooth off manually – this is to minimise the amount of captured data. Don’t forget to turn logging off on the phone when done. To find where the log file has been stored and copy the file from the phone to the Pi use:

We can now use Wireshark to read the log file…

Wireshark reading the Android Bluetooth Log

This wasn’t quite the posting I originally had in mind, I wanted to decode the data from the watch for my own use, making something more useful, impressive graphs and charts, than that provided by the Android App VeryFit 2.0 but as the manufacturer has chosen to use proprietary GATT codes it makes the job that much harder. It may be much simpler to just buy an expensive FitBit and download the data from them. But with writing this I now know a few things that were previously unknown, and I hope that this has provided some light to your BlueZ (a pun!, right at the end!).

Links and Sources

Fixing the Arduino incoming network connections error on the mac

On the Apple Mac if you use the Teensy micro-controller with the Arduino IDE you may have come across a persistent firewall error message when starting the IDE, I have seen this error for quite a while over a range of system and software upgrades. I have applied this fix to:

  • OS X 10.10 Yosemite and above / macOS 10.12 Sierra
  • Arduino IDE 1.6.13 – all versions, at least 1.5 and above.
  • Teensydunio 1.33 – and older versions

The Arduino IDE is installed in the default applications folder, as is the Teensyduino. Some knowledge of using the terminal is required.

Symptoms

On your Apple Mac, you installed the Teensyduino software for the Teensy and now when you start the Arduino IDE this error message appears:

Do you want the application “Arduino.app” to accept incoming network connections?
Clicking Deny may limit the application’s behaviour. This setting can be changed in the Firewall pane of Security & Privacy preferences.

Incoming Network Connections Error

Cause

When the Arduino IDE is installed it includes a certificate to assure the system that everything is correct, the Teensyduino installation makes changes to the IDE configuration and this causes a mismatch and the signature in the certificate does not match the installation.

You can verify the failed certificate in the terminal with the spctl command:

Without the Teensyduino software installed, the certificate shows correctly:

Another check is to use codesign

Fix

To fix this, first we need to create a self-signed certificate. In finder Keychain Access can be found in Applications > Utilities > Keychain Access

Keychain Access certificates

From the menu choose: Keychain Access > Certificate Assistant > Create a Certificate… and set the following:

  • Name: anything useful, without spaces. You will be using this name later to apply the certificate – I used ‘arduino’
  • Identity Type: Self Signed Root
  • Certificate Type: Code Signing
  • Check the box “Let me override defaults”, this is important
Certificate Creation

click continue, and continue again past the security warning, then over the next few pages:

  • Serial Number: 1 – The serial and certificate name combination must be unique
  • Validity Period: 3650 – this will give you ten years
  • Email, name, etc: anything you like, or leave blank
  • Key pair info: set to RSA, 2048 bits
  • On the next four screens, from “Key usage extension” to “Subject Alternate Name Extension” accept the defaults
  • Location: login keychain.

Once created and back in the list, choose your certificate and from the menu go to File > Get Info (Cmd-i). In the Trust section at the top change: When using this certificate to Always Trust.

Trusted Certificate

Now that the certificate has been created, you need to apply it to the application, in terminal use the codesign command, this takes a few moments:

You’ll be asked to verify this, click ‘always allow’. To verify that your certificate has worked, check with codesign, using spctl will not work as this is a self-signed certificate.

Now, when you start the IDE on first run it will give you the allow/deny message again, click Allow and on subsequent use it will open as expected.

Links and Sources

Upgrading the Python Oracle Client

This is a follow up to one of my previous postings: Python and the Oracle Client. The main databases here are being upgraded to Oracle 12 and I’ve taken the opportunity to update the client used by my Python scripts, also its good practice to install new clients when old versions go out of support.

Current Setup

The system I am upgrading here has the following configuration, but this should work with any RPM based distribution, such as CentOS and SUSE :

  • Red Hat Enterprise Linux Server release 6.6 (Santiago)
  • Python 2.6.6
  • python connector: cx_Oracle – 5.1.2
  • oracle-instantclient11.2-basic-11.2.0.4.0-1.x86_64
  • oracle-instantclient11.2-devel-11.2.0.4.0-1.x86_64

To find the versions of your currently installed software:
$ python
Python 2.6.6 (r266:84292, Nov 21 2013, 10:50:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
>>> print cx_Oracle.version
5.1.2

and the Oracle client:
$ rpm -qa | grep oracle
oracle-instantclient11.2-basic-11.2.0.4.0-1.x86_64
oracle-instantclient11.2-devel-11.2.0.4.0-1.x86_64

Preparing

If you have the old versions installed you will need to do some tidying up by removing the client and python connector, version 11 of the client despite being RPM packaged had some non-standard elements. Use rpm to delete the old version of instant client, remove devel first:
$ sudo su
# rpm -ev oracle-instantclient11.2-devel-11.2.0.4.0-1.x86_64
# rpm -ev oracle-instantclient11.2-basic-11.2.0.4.0-1.x86_64

you may also need to remove the library reference from a previous installation:
# rm /etc/ld.so.conf.d/oracle.conf
# ldconfig

to remove the Python oracle connector, there are two methods. Manually, by finding the previously installed package deleting the files and editing the package list:
# find / -name cx_Oracle.py -print
/usr/lib/python2.6/site-packages/cx_Oracle-5.1.2-py2.6-linux-x86_64.egg/cx_Oracle.py
# cd /usr/lib/python2.6/site-packages
# rm -rf cx_Oracle-5.1.2-py2.6-linux-x86_64.egg

now edit the easy-install.pth file
# nano /usr/lib/python2.6/site-packages/easy-install.pth
and remove the line:
./cx_Oracle-5.1.2-py2.6-linux-x86_64.egg

Or do it the easy way, if you have pip installed:
# sudo pip uninstall cx_Oracle
easy_install does not have an uninstall option.

Installing

Download and install version 12 of the Instant Client and SDK (devel), these can be gotten from: http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html For Linux choose the correct flavour for your installed operating system: x86 or x86-64 for 64bit operating systems, you will need to register on the site gain access the files.
# rpm -i oracle-instantclient12.1-basic-12.1.0.2.0-1.x86_64.rpm
# rpm -i oracle-instantclient12.1-devel-12.1.0.2.0-1.x86_64.rpm

Now to install the python connector:
# easy_install cx-Oracle
or, the recommended method:
# pip install cx-Oracle
Installation for the version 12 client is much more straight forward than that for version 11.

Testing

A quick test to ensure that the expected versions appear, and that you can connect to the database.
Python 2.6.6 (r266:84292, Nov 21 2013, 10:50:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
>>> cx_Oracle.version
'5.2'
>>> oraConn = "<USERNAME>/<PASSWORD>@<DATABASE HOST>:<DATABASE PORT>/<SERVICE>"
>>> ocDB = cx_Oracle.connect(oraConn)
>>> ocDB.version
'12.1.0.2.0'

Sources

Water Splash Photography

Two drops of water colliding, frozen in time using the power of high speed flash photography produce an infinite variety of shapes. While this can be done with a pipette, a camera, a single flash gun, practice and good hand/eye co-ordination. I have an Arduino Uno and I am going to use it. IMG_8887What is happening in this picture? Two carefully timed water droplets have been released from above and are plummeting towards a bowl of water. The first drop has hit the water and is rebounding, just as the up-spout reaches its zenith, the second drop collides with the top resulting in a mushroom shaped splat, with the event captured in the camera with a frame of 1/10,000th of a second. In this post I’ll be sharing my experiences in creating these water drop images, I’ll be looking at the photography equipment, electronics, and technique.
up meets down

Photography Equipment

Camera: This can be any DSLR or advanced compact, it must have Bulb mode, and be triggerable by an electronic wired connection, some have an IR remote but I found this to be difficult to setup. Set the ISO to be around 200.

Lens: I use a 100mm Macro, with focus set to manual and image stabilisation off. The aperture is set high, at least f22 to give a suitable depth of field and improve image sharpness.

Tripod: A good solid one with easy to adjust ball head.

Flash: I use up to five flash guns for my photos, two for back light, one to give an under-light through the glass bowl, another for front light and finally one handheld. Rechargeable batteries for the flashes are recommended, I use 2400mAh NiMh Duracells.

The flash guns need to be in manual mode at their lowest power setting, this is to give the shortest duration of flash for the sharpest results. As you increase the flashes power the duration of the light emitted gets longer, causing burred images. On my Canon flash I set it to 1/128 second and on the Nissin Di622 set the EV to -1.5.

6N2A5694  6N2A5698

For connecting the Arduino to the flashes I use a 2.4GHz wireless remote trigger, with four receivers and a modified hotshoe mount attached to the transmitter Look for the Yongnuo RF-602 Remote Flash trigger on ebay, (not to be confused with the remote shutter release).  Most modern TTL flash guns appear to be missing the wired remote trigger connection that you can just plug into.

The flashes also have a built in slave trigger, where it sees that one flash has gone off so it set itself off too. On the Canon flashes this appears to only work in ETTL mode and can’t be used for this, but the Nissins work well.

Hardware

The frame is bits of wood held together with glue and stands about 75cm high this is to allow the water to accelerate and produce decent sized splashes. At the base is an extra large seed tray, the type without holes, to contain any spillages. This normally has a glass bowl full of water acting as the drip splash event zone. Halfway up the fame is mounted the laser and detector and at the top a reservoir of water and solenoid valve.

The reservoir is a one litre plastic storage tub from Poundland with a hole drilled in the base and a short length of 8mm PVC tubing hot glued into place. The tubing can be difficult to glue as its rather flexible, pushing down a short section of solid tube made from the outer of a disposable biro fixes that. This pipe is connected to the solenoid, observing the correct direction of flow marked on the valve.

The reservoir has a Mariotte Syphon fitted to the lid, this is to provide a constant and stable water pressure to the valve, the pipe from the lid ends about 2cm short of the reservoir base.
resevoir

Electronics

The Arduino and control electronics are all set to produce this photo taking sequence:Trigger Prototype

  1. press ‘play’ button on remote control
  2. lights out – dark room
  3. open shutter on camera
  4. solenoid releases two drops of water
  5. drips pass through laser detector – timer started
  6. drops arrive and do their thing
  7. flash guns triggered by timer – picture taken
  8. shutter closed on camera
  9. lights on

The electronic circuit can be broken down into these five blocks; lights, laser, IR receiver, solenoid control, flash control, camera control, each diagram shows the label name for the pin used rather than a pin number. The diagrams can be enlarged by clicking on them.

IR Receiver: The IR receiver allows use of an old TV remote control. My original design was to have a rotary dial and a small OLED display, but this simplified everything considerably. If you don’t have a spare remote one can be gotten from Poundland. I have the Arduino send any text output to a laptop on the USB port.

ir_rx

Laser: Warning: keep away from eyes, permanent damage can occur with exposure to any laser. The laser is used with a photo-transistor to detect drips of water as they plummet to their splash event. I used a small 3 Volt 5mW red laser with a built-in lens, I have added a resistor and diode in series to prevent over voltage as they’re a bit delicate. Although a modified laser pointer will do just as well. The TEPT4400 phototransistor is a type rated for visible light and has higher sensitivity to change than a photoresistor.

laser control
Laser Control

Lights: Warning: Mains Electricity Can Kill, this is to be avoided. If you are uncertain about this part then don’t do it. I rapidly found that working in darkness between shots just made life difficult, and finding the light switch became a hassle. To fix that I got a pre-made 5v Relay circuit and wired this up to a table lamp to provide some illumination. Using a standard wall socket and backbox connect live through the normally open side of the relay, and the neutral and earth to the socket.

 6N2A5700 6N2A5705

Remember to keep the electricity away from fingers (and any other body parts) and water.

Solenoid Control: I use a 12v solenoid, (search for “12v solenoid valve water arduino” on ebay, a couple of sellers have suitable models with connectors included). I use a mosfet transistor to switch the power, this has been detailed in one of my previous blog postings.

solenoid_control
Solenoid Control

Flash and Camera Control: The electronics for the camera and flash are closely related. Both use the ILD74 optocoupler to electrically isolate the camera and flash equipment from the Arduino. Although the camera focus connection is not used here I have included it as it may be useful later on.

camera control
camera and flash

The Canon camera has two different types of wired connection on the shutter release depending on the model of camera, a standard three pin 2.5mm jack or a N-3 connector (search for “canon N3 connecting cable” on ebay). A list of connectors for other makes of cameras can be found here.

canon remote
Remote shutter connections for Canon

Sound: Although not used here, this setup works well with a piezo microphone for use with popping water balloons and the like, use a buzzer that is enclosed in a plastic housing with a hole on top and buzzes when DC power is applied. Connect the output to an analogue pin on the Arduino, your software can use a very similar method to that for the laser.

sound
Sound Detect
water balloon pop
Balloon pop with sound detector

Setup and Use

Have plenty of dish cloths or towels to hand, this can get a bit moist. Keep an eye on your camera equipment making sure it doesn’t get wet.

For setting up a shot I use a steel ruler with a magnet stuck to it I set the water dripping to make sure it lands where I want on the magnet, the camera is then focused on the magnet, take away the ruler and you have your properly focused splash event.

IMG_8502
Splash Hat on Magnet

Add colour with food dies, adding these to the reservoir seems to work best and keeping the water in the splashdown area clean. Guar Gum thickens the water and makes larger drops and bigger splashes, you only need to add a small amount, about a teaspoon per litre and you’ll need to sieve out any lumps before use. Fluorescein is quite entertaining when used with a UV lamp, adding a green glow to your splashes. Adding diluted water based paints to the reservoir can add a lot of colour, but has a tendency to block the solenoid.

Sparkly backdrops can be gotten from the craft section in stationers. An A4 sized (21cm x 30cm) sheet is normally enough. Try bouncing the flash off the backdrop.

It is all about experimentation, expect to take lots of photos, many of which will be poor. Make notes of timings when you get good results, when you get a good shot, very small changes in timings can produce fairly dramatic effects.

Software

Here is an Arduino sketch, press Play on the remote to start a two drip sequence. adjust the amount of time in milliseconds between the laser detect and flash – flashWait with Volume +/-: +10,-10, Channel +/-: +5,-5, Fast Fwd/Rev: +2,-2, and the time betweenDrips with 4 (+1) and 7 (-1).

Links and Sources

My water drop photos on flickr:

Two Drops of Water

How-To: Raspberry Pi as a 3G/4G Router

Update: 15 April 2016 – Added information about which IP address to use and assigning static IP addresses for printers and servers

Recently I have needed to find an emergency alternative to my broadband due to the regional wide area network, Digital Region, being shut down, and the ISP Origin making a mess of getting all their cutomers onto ASDL. To get quickly back onto the internet, I have bought an ZTE MF823 4G Mobile Broadband Dongle as supplied by the badly named ‘three’ mobile phone company. As I have my own internal wired network, with multiple computers and ‘things’ there is a need to have something more sophisticated than just plugging the dongle into a single PC.

network diagram

Here is my recipe for setting up a Raspberry Pi as a router with an ZTE MF283 Dongle. In this setup all the computers are on a wired Ethernet connection using a switch for the network. The Pi has Raspbian Debian Wheezy installed (June 2014) with all the latest updates made. For testing, the dongle is plugged into the USB port via a powered hub, and the Pi connected to a switch with another PC running Linux Mint.

Which IP addresses to use?

In this How-To I am using the IP address range 192.168.2.xxx this is to avoid conflict with the cable router which uses the 192.168.1.xxx range (the DHCP server is switched off on the router). IPv4 addresses are split into three different ranges, the 192.168.xxx.xxx range – 192.168.0.0 to 192.168.255.255 gives a possible 65,536 addresses but for your home it is unlikely you’ll have more than 255 network devices, so we can simplify things by limiting the address range used to 192.168.2.xxx and avoid the troublesome world of subnet masking.

192.168.xxx.xxx is used as its been designated for use on private networks by the Internet Assigned Numbers Authority this is a well established convention and is best practice. Two other IPv4 address ranges are available for larger private networks: 172.16.0.0-172.31.255.255 and 10.0.0.0-10.255.255.255 with 1,048,576 and 16,777,216 available addresses respectively, the most suitable network class should be chosen for your network.

Setup the Dongle

This USB dongle has its own built in dialer so you do not need ppp or wvdial installed, it appears as a USB ethernet device on the Raspberry Pi. You will need a powered USB hub as the dongle can draw more power than the Pi can provide, the symptoms of too much of power being drawn will be the Pi behaving erratically or restarting unexpectedly.

With the dongle plugged in, check that it is recognised by the Pi with lsusb, it can be seen here as ‘ZTE WCDMA Technologies MSM’:

The device ID is 19d2. and 1405 is the mode, this should be 1405 – CDC ethernet. If it is not, try removing the micro-SD card and rebooting the Pi, the device modes available are:

  • 1225 – Default mode. USB Mass Storage Device + CD-ROM + card reader.
  • 1403 – Modem mode. RNDIS + Mass Storage Device.
  • 1405 – CDC ethernet
  • 0016 – Download mode

As the dongle also has a Mass Storage Device the Raspberry may not switch to CDC ethernet. If the mode does not change, try the following with usb-modeswitch:
$ sudo apt-get install usb-modeswitch
$ sudo usb_modeswitch -v 0x19d2 -p 0x1405 -d

I did not have to change the mode as it was correct already, and it didn’t change when I tried setting it as a Mass Storage device, I have not explored this any further.

When first plugged in the dongle was recognised as a ethernet device but it did not obtain an IP address:

if this is the case with you, add the following two lines to the end of sudo nano /etc/network/interfaces:
auto usb0
iface usb0 inet dhcp

the dongle provides its own address to the computer. Reboot, and you should see the obtained address:

The address 192.168.0.185 is now the internet address of the computer the dongle always assigns this address, there is also a useful web status page on http://192.168.0.1

Configuring the network

first of all enable ip4 forwarding, edit the file sudo nano /etc/sysctl.conf and uncomment the line:
net.ipv4.ip_forward=1
this will enable forwarding on reboot, you can also enable IP forwarding immediately with:
$ sudo sysctl -w net.ipv4.ip_forward=1

We now need to give the Pi a static IP address on the internal network. Edit sudo nano /etc/network/interfaces so you end up with a file that looks like this:

this gives the Pi a static IP address of 192.168.2.1.

DHCP

The next stage is to give the other computers on your network an IP address, this is done with a dhcp server:
$ sudo apt-get install isc-dhcp-server
you will need to configure dhcp sudo nano /etc/dhcp/dhcpd.conf, here is mine:

This will assign IP addresses in the range 192.168.2.50 to 192.168.2.150 to any computer connected to your network. I have used Open DNS for the Domain name Servers, if you wish to use google’s use:
option domain-name-servers 8.8.8.8, 8.8.4.4;instead.

I have also given my network printer a static IP address, it is still assigned by the DHCP server but never changes, the same would apply to any file servers and the like, I would assign static devices addresses that are outside your dynamically assigned range. Reboot the Pi and then your test computer.

Your test computer should now have an IP address (192.168.2.51), and the gateway point to the Pi (92.168.2.1):

Accessing The Internet

The final part is to have the incoming traffic on the the Ethernet port eth0, go out on the dongle usb0. This is achieved with iptables, a firewall and traffic router. Install with:
$ sudo apt-get install iptables
and you need to setup Network Address Translation, NAT and forwarding. This short bash script clears any old settings before applying the new rules:

Where LAN is your internal network, and WAN is the internet. The final line allows you access to the Dongle’s built in web status page from any browser on your internal network, just use: http://192.168.2.1:2525

Save the file in your home directory as ~/ipt.sh, make it executable and run the script.

$ chmod +x ~/ipt.sh
$ sudo ~/ipt.sh

From your test computer, you will now be able to access the internet.

Finally, you now need to have iptables reload when you start the Pi. Export the iptables settings to a file with:

$ sudo sh -c "iptables-save > /etc/iptables.ipv4.nat"

and create a file sudo nano /etc/network/if-up.d/iptables with the following contents:

and make it executable sudo chmod +x /etc/network/if-up.d/iptables

after a reboot you can see your iptables with sudo iptables -L and sudo iptables -t nat -L and you can see web traffic passing through the router with sudo tcpdump -i any -nn port 80.

Adding a Proxy Server

This is optional, but a transparent proxy server and cache may reduce the amount of traffic on your 3G/4G connection, mileage varies and the amount of data cached was less than I thought it would be, I also found that my Humax Freesat box really didn’t like the proxy server and wouldn’t update its TV schedules while it was on. I have used squid3 for this.
sudo apt-get install squid3
Update the squid3 configuration /etc/squid3/squid.conf so it has the following. The original is rather large, so you may want to make a copy and create a new one:

then restart squid3
sudo /etc/init.d squid3 restart
add the following iptables rule to redirect all traffic on port 80 to squid3:
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 3128
you should now be able to watch the web traffic being processed through squid3 with:
sudo tail /var/log/squid3/access.log -f
finish off by exporting your iptables again, so they are reloaded on reboot:
sudo sh -c "iptables-save > /etc/iptables.ipv4.nat"

Raspberry Pi - 4G Router

References and sources:

The Case of OpenCV and the Missing SURF

I have been wanting to have a play with the OpenCV computer vision framework on Python for a while and finally got some time to some experimenting, I am looking to have the computer recognise LEGO parts, after much research and mucking about it seems I should be using cv2.SURF and/or cv2.SIFT for what I want to do. However on Fedora 19 these are not included in the distribution RPM as they are nonfree in that they are not open source. Attempts to use SIFT or SURF result in the following error:

$ python
Python 2.7.5 (default, Nov 12 2013, 16:18:42)
[GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> print cv2.__version__
2.4.6.1
>>> i = cv2.SURF()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'SURF'
>>>
This is annoying as OpenCV will now need to be re-installed the old fashioned way, but there are instructions on the OpenCV site and I will be using them here with additional information I have discovered while following them.

Change to root and add the rpmfusion.org free and nonfree repositories as ffmpeg and libv41 are unavailble from Fedoras, followed by an update (it seems that Fedora 19 always has something to update):

& sudo su
# yum localinstall --nogpgcheck http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
# yum update

Remove the existing OpenCV packages:

# yum erase opencv*

There are a few packages to install in preparation to compiling the code, you may have some of all of these already but yum will skip those. Install the mandatory packages, these are for compiling the source and using GTK for the GUI:

# yum install cmake python-devel numpy gcc gcc-c++ gtk2-devel libdc1394-devel libv4l-devel ffmpeg-devel gstreamer-plugins-base-devel git wget

Install the optional dependencies, I have gone for the install everything approach, they may be useful later:

# yum install libpng-devel libjpeg-turbo-devel jasper-devel openexr-devel libtiff-devel libwebp-devel tbb-devel eigen3-devel python-sphinx texlive

Now, we are ready to get the latest source, here there are two ways to do this, install the latest development version using GIT, or download the latest stable version. My preference is to use the latest stable version.

With GIT, exit back to yourself from root, change to your home directory and download OpenCV:

# exit
$ cd ~
$ git clone https://github.com/Itseez/opencv.git
Cloning into 'opencv'...
$ cd opencv
$ mkdir build
$ cd build

Or using the latest build, at time of writing this is v2.4.7. Get this via the downloads page and save it to your home directory:

# exit
$ cd ~
$ tar -zxvf opencv-2.4.7.tar.gz
$ cd opencv-2.4.7
$ mkdir build
$ cd build

Now configure:

$ cmake -D CMAKE_BUILD_TYPE=RELEASE -D BUILD_PYTHON_SUPPORT=ON -D WITH_TBB=ON -D BUILD_NEW_PYTHON_SUPPORT=ON -D WITH_XINE=ON -D WITH_V4L=ON D WITH_OPENGL=ON -D WITH_OPENCL=OFF -D CMAKE_INSTALL_PREFIX=/usr/local ..

When complete you should check that your options agree with those displayed. There are many others available. Support for other programming languages may be missed out if they are not already installed. In my case those for Java, if I were to try OpenCV in Java I would need to install the appropriate Java packages using yum, then recompile OpenCV.

Now build and install, this can take a while:

$ make
$ sudo make install

You now need to move the module to anywhere on the Python Path, to find this:

$ python
Python 2.7.5 (default, Nov 12 2013, 16:18:42)
[GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print sys.path
['', '/usr/lib/python2.7/site-packages/PIL-1.1.7-py2.7-linux-x86_64.egg', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/PIL', '/usr/lib64/python2.7/site-packages/gst-0.10', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages', '/usr/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info']

The directory /usr/lib/python2.7/site-packages looks suitable:

$ sudo mv /usr/local/lib/python2.7/site-packages/cv2.so /usr/lib/python2.7/site-packages

And add your new installation to the PYTHONPATH, and add the export to the end of your .bashrc so it survives a reboot:

$ export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python2.7/site-packages
$ echo export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python2.7/site-packages >> ~/.bashrc

Now to test:

$ python
Python 2.7.5 (default, Nov 12 2013, 16:18:42)
[GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> print cv2.__version__
2.4.7
>>> i = cv2.SURF()
>>>

Sorted. Now, perhaps, I can get on with what I actually want to do….

Sources:

Converting the Canon GPS log

Having been out and about with the Canon 6D, for once I had remembered to turn on the GPS logging built into the camera. Upon returning home after the seven mile walk I saved the GPS data in the cameras memory to the SD card. Now this LOG file found in the GPS directory of the SD card can be read directly into Google Earth but not into Memory Map for viewing on the far superior OS 1:25,000 Ordnance Survey map, also Memory Map does not read Google Earth’s KML or KMZ files.

So, the LOG file needs converting, but what format is it in? Canon are, it seems, rather unhelpful in revealing the secret but a little googeling found a reference to them using the NMEA-0183 format.

For conversion I found the free, excellent and very comprehensive gpsbabel to do the job (the PDF manual is over 200 pages), for me I convert the GPS files on a linux computer, but they do a Windows version and I assume the method and outcome will be the same. On Debian, install gpsbabel with:
$ sudo apt-get install gpsbabel

The basic use of gpsbabel for converting is as follows:
gpsbabel -i <input format> -f <input file> -o <output format> -F <output file>

As I know Memory Map reads the Garmin GPX format, I chose that as the output:
$ gpsbabel -i nmea -f 13120100.LOG -o gpx -F 13120100.gpx

And the rest is, as they say, Topographic.

walkabout

Recording Sound on the Raspberry Pi

The Raspberry Pi does not have a microphone socket, which is inconvenient when you wish to record sound. To fix this you will need a USB Sound Card, for which I bought a Creative Sound Blaster Play! for about £20 and a short USB extension lead as the sound card is slightly too large and blocks the other USB port.

With the latest Raspbian “wheezy” installed on a Pi Model B with 512Mb of RAM and the overclocking set to High in raspi-config, here is a recipe for getting your Raspberry Pi to record sound from the command line. For the test setup I connected my iPod to the microphone port of the sound card, plugged everything in and powered up.

Raspberry Pi Records

After logging into the Pi, check that the computer can see the card, use lsusb to find it, here the card is highlighted in blue:
$ lsusb
Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp.
Bus 001 Device 004: ID 046d:c52e Logitech, Inc.
Bus 001 Device 005: ID 041e:30d3 Creative Technology, Ltd Sound Blaster Play!
remember that different makes of card will have different names and ID’s

Your user will need to be in the audio group, check this with groups <username>:

$ groups pi
pi : pi adm dialout cdrom sudo audio video plugdev games users netdev input
if not, then add them with:$ sudo usermod -a -G audio <username>

There is/was an issue with the Pi’s USB port that meant it can/could become overwhelmed1 with data which causes popping and bubbling noises to be included in your recordings, this can be fixed with an update of the Pi’s firmware:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install rpi-update
$ sudo rpi-update

The Raspbian image already has the alsa-utils for sound already installed, the programs I am using for recording and playback are:

  • alsamixer – GUI for setting the recording and playback levels
  • amixer – Command Line for setting the recording and playback levels
  • alsactl – for saving the settings set in alsamixer or amixer to use again after a reboot
  • arecord – For recording the sound
  • aplay – For playing back your recording

All of these programs have a –help option.

There are two methods for setting up the the microphone port on the card, the first is alsamixer:

alsa mixer

$ alsamixerPress F6: Select Sound Card, and choose yours from the list, the bcm2835 ALSA is the on-board sound, for me the one to pick was: USB Device 0x41e:0x30d3 and take a note of the card number, in my case: 1. Now select the Mic and increase the volume to 52, or the first white blob, you’ll need to change it later, but its a good place to start. The Auto Gain Control wants to be off, select the gain control and press M to toggle so it displays [MM] for mute. Press Esc to exit and save the settings with:$ sudo alsactl store 1 where 1 is the card number.

Alternatively, you can use amixer. First find your sound card, in amixer there does not appear to be a method of listing the available cards, but on a Raspberry Pi I guess it will always be card 1, you can list the cards current status with:

$ amixer --card 1 contents
numid=1,iface=MIXER,name='Mic Playback Switch'
; type=BOOLEAN,access=rw------,values=1
: values=off
numid=2,iface=MIXER,name='Mic Playback Volume'
; type=INTEGER,access=rw---R--,values=1,min=0,max=32,step=0
: values=21
| dBminmax-min=0.00dB,max=47.81dB
numid=5,iface=MIXER,name='Mic Capture Switch'
; type=BOOLEAN,access=rw------,values=1
: values=on
numid=6,iface=MIXER,name='Mic Capture Volume'
; type=INTEGER,access=rw---R--,values=1,min=0,max=16,step=0
: values=7
| dBminmax-min=0.00dB,max=23.81dB
numid=7,iface=MIXER,name='Auto Gain Control'
; type=BOOLEAN,access=rw------,values=1
: values=on
numid=3,iface=MIXER,name='Speaker Playback Switch'
; type=BOOLEAN,access=rw------,values=1
: values=on
numid=4,iface=MIXER,name='Speaker Playback Volume'
; type=INTEGER,access=rw---R--,values=2,min=0,max=151,step=0
: values=52,52
| dBminmax-min=-28.37dB,max=0.06dB

So I want to turn the Auto Gain Control off, and the recording volume to 14:
$ amixer -c 1 cset numid=7,iface=MIXER,name='Auto Gain Control' 0
numid=7,iface=MIXER,name='Auto Gain Control'
; type=BOOLEAN,access=rw------,values=1
: values=off
$ amixer -c 1 cset numid=6,iface=MIXER,name='Mic Capture Volume' 14
numid=6,iface=MIXER,name='Mic Capture Volume'
; type=INTEGER,access=rw---R--,values=1,min=0,max=16,step=0
: values=14
| dBminmax-min=0.00dB,max=23.81dB

again, store the settings so that they will be used again on a reboot:$ sudo alsactl store 1

Now we are ready to do a test recording, first check that arecord will see your card:

$ arecord -l
**** List of CAPTURE Hardware Devices ****
card 1: U0x41e0x30d3 [USB Device 0x41e:0x30d3], device 0: USB Audio [USB Audio]
Subdevices: 1/1
Subdevice #0: subdevice #0

and now for a ten second test recording, this will create a file called rectest.vav in your home directory. Remember to set the Device (-D plughw:1) number to the right card (card 1):

arecord -D plughw:1 --duration=10 -f cd -vv ~/rectest.wav the vv option displays extra information on the screen as well as a volume meter, this should be peaking at around 95% on the loudest sounds, if it is at 100% all a lot of the time then you are probably recording distortion. Playback the recording with aplay:

aplay ~/rectest.wav

the default settings will play the wav fie through the TV if it is connected by HDMI, To playback through the USB sound card set the device to the card number, like in arecord:

aplay -D plughw:1 ~/rectest.wav

Congratulations, you now have a fully working Pi Recording Device. Remember to experiment with the volume levels, too high and your recording will sound distorted.

References:

  1.  Raspberry Pi Usb Audio fix (10 May 2013)

Python and the Oracle Client

Update 24 Nov 2015: Also, see my post Upgrading the Python Oracle Client for updating from version 11 to 12.

Installing the python cx_Oracle extension module for connecting to Oracle databases on this Fedora 18 workstation turned out to be a bit of a faff by giving an assortment of unhelpful error messages, if you are having the same pain maybe this will help.

You will need two files from the Oracle Database Instant Client download site, the basic client package and the SDK (devel): http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html For Linux choose the correct flavour for your installed operating system: x86 or x86-64 for 64bit operating systems, you will need to register on the site to access the files. For my purposes I got the 11.2 version for x64 RPM files, install them using:
$ sudo rpm -i oracle-instantclient11.2-basic-11.2.0.3.0-1.x86_64.rpm
$ sudo rpm -i oracle-instantclient11.2-devel-11.2.0.3.0-1.x86_64.rpm

You will now need to tell the system where the libraries are:
$ sudo su
# echo /usr/lib/oracle/11.2/client64/lib/ > /etc/ld.so.conf.d/oracle.conf
# ldconfig
# exit

To install cx_Oracle you will need to set some environment variables otherwise you will get an “error: cannot locate an Oracle software installation” message. The easy_install program is found in python-setuptools I have included it in the recipe as a reminder if you have not installed it already.
$ sudo yum install python-setuptools
$ export ORACLE_HOME=/usr/lib/oracle/11.2/client64
$ export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
$ export PATH=$ORACLE_HOME/bin:$PATH
$ sudo -E easy_install cx_Oracle

or by downloading the version from http://cx-oracle.sourceforge.net/ and installing it manually, you will still need to set the exports, as above, and do the following for installation:
$ sudo -E python setup.py build
$ sudo -E python setup.py install

The -E on the sudo takes your environment variables, including the three you just set, into your sudo session.

Success:
$ python
Python 2.7.3 (default, Aug 9 2012, 17:23:57)
[GCC 4.7.1 20120720 (Red Hat 4.7.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
>>> exit()

When the the library cannot be seen, you get this error:
$ python
Python 2.7.3 (default, Aug 9 2012, 17:23:57)
[GCC 4.7.1 20120720 (Red Hat 4.7.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
Traceback (most recent call last):
File "", line 1, in
ImportError: libclntsh.so.11.1: cannot open shared object file: No such file or directory
>>> exit()

Taking Photos on the Canon EOS with an Arduino

For my Spangaly Stick project, more of which later, I have been wanting to add a remote control function for the camera for a while now and with the extended winter I found I had the time. I am using an Arduino micro-controller fitted with an XBee wireless shield to respond to a keyword sent by another Arduino with a similar setup, or from a computer with an XBee on a USB port.

take a picture - board

This board had been built to work with the Canon EOS DSLR range of cameras, but should work with other makes of camera fitted with an electronic remote socket. For basic use the camera should be set in Aperture Priority (Av) mode with the lens set to manual focus and if your using a tripod switch off any image stabilization.

For this project you will need:

  • A Windows (XP and above) Computer
  • An Arduino Uno R3
  • Two XBee’s
  • An XBee USB adapter
  • An XBee Shield for the Arduino
  • A battery to power the Arduino, I use a 12v 1.3Ah Sealed Lead Acid.
  • A remote lead for your camera
  • Ability to solder, read circuit diagrams, etc..

For this remote to work, you will need to configure two XBees to talk to each other. The easy way to do this is using the X-CTU tool (unfortunately it is Windows only) and a XBee USB Adapter. The XBees come in two main types, the Series One (S1) and Series Two (S2) they will need to be of both the same series to talk to each other, I have used Series One bees here, but not the ones without the sticking out antenna as I suspect they may be a bit delicate.

While the XbeeSheid configuration is documented on the arduino site, I shall summarise here. With the XBee plugged into the USB adapter and the adapter plugged into the computer wait for it to be detected by Windows then you can configure it. In X-CTU, on the PC Settings Tab, select the USB Serial Port. By default the XBee is set to baud: 9600, flow: None, data: 8, parity: None, stop: 1. Click the Test/Query button you should see some basic XBee settings, if it says it cannot communicate, try a different baud rate. Higher baud rates are available, but whats the rush?

Now click the Modem Configuration tab, under Modem Parameter and Firmware, click Read, after a pause at the top of the list The modem indicated will be whatever model you have, in my case XB24, select the function set we will be using: XBEE 802.5.4 RS485 ADAPTER and the firmware version: 13E8. You will need to set the Networking & Security Channel (CH) and the PAN ID (ID) to the same on both, as well as check the Serial Interfacing are set correctly, Interface Data Rate (BD): 3 – 9600, Parity (NB): 0 – NONE. Click Write to save the settings to the XBee.

For testing the XBee’s, I had one plugged into the USB adapter and the other placed into the the XBee Shield, an LED fitted across Digital Output 12 and Ground, and sent the following program to the Arduino, remembering to flick the little switch on the shield to USB for programming, and Micro for XBee emissions.

The circuit I made for taking photos includes the focus, this is optional but I have included it so the camera can be woken up before the picture is taken. An ILD74 opto-isolater is used to electrically separate the camera from the Arduino, it also simplified the circuit not having to use transistors. The LED’s are useful additions but are optional.

The diagram also shows the Canon Remote connections. Most use a 2.5mm stereo jack plug, but the more expensive cameras have a proprietary Canon N3 connector, to get the lead you will need to sacrifice a wired remote, one of these can be gotten of ebay very cheaply, about £3.00. The colours for the wires can be anything, so you will need check with a continuity tester.

The program on the Arduino listens for a keyword, in this case ‘PHOTO’ which then triggers the picture taking sequence first by setting the focus to wake the camera up then taking the picture. The Indicator LED is used to show activity on the serial port.