This is a follow up to one of my previous postings: Python and the Oracle Client. The main databases here are being upgraded to Oracle 12 and I’ve taken the opportunity to update the client used by my Python scripts, also its good practice to install new clients when old versions go out of support.
Current Setup
The system I am upgrading here has the following configuration, but this should work with any RPM based distribution, such as CentOS and SUSE :
Red Hat Enterprise Linux Server release 6.6 (Santiago)
To find the versions of your currently installed software: $ python
Python 2.6.6 (r266:84292, Nov 21 2013, 10:50:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
>>> print cx_Oracle.version
5.1.2
and the Oracle client: $ rpm -qa | grep oracle
oracle-instantclient11.2-basic-11.2.0.4.0-1.x86_64
oracle-instantclient11.2-devel-11.2.0.4.0-1.x86_64
Preparing
If you have the old versions installed you will need to do some tidying up by removing the client and python connector, version 11 of the client despite being RPM packaged had some non-standard elements. Use rpm to delete the old version of instant client, remove devel first: $ sudo su
# rpm -ev oracle-instantclient11.2-devel-11.2.0.4.0-1.x86_64
# rpm -ev oracle-instantclient11.2-basic-11.2.0.4.0-1.x86_64
you may also need to remove the library reference from a previous installation: # rm /etc/ld.so.conf.d/oracle.conf
# ldconfig
to remove the Python oracle connector, there are two methods. Manually, by finding the previously installed package deleting the files and editing the package list: # find / -name cx_Oracle.py -print
/usr/lib/python2.6/site-packages/cx_Oracle-5.1.2-py2.6-linux-x86_64.egg/cx_Oracle.py
# cd /usr/lib/python2.6/site-packages
# rm -rf cx_Oracle-5.1.2-py2.6-linux-x86_64.egg
now edit the easy-install.pth file # nano /usr/lib/python2.6/site-packages/easy-install.pth
and remove the line: ./cx_Oracle-5.1.2-py2.6-linux-x86_64.egg
Or do it the easy way, if you have pip installed: # sudo pip uninstall cx_Oracle
easy_install does not have an uninstall option.
Installing
Download and install version 12 of the Instant Client and SDK (devel), these can be gotten from: http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html For Linux choose the correct flavour for your installed operating system: x86 or x86-64 for 64bit operating systems, you will need to register on the site gain access the files. # rpm -i oracle-instantclient12.1-basic-12.1.0.2.0-1.x86_64.rpm
# rpm -i oracle-instantclient12.1-devel-12.1.0.2.0-1.x86_64.rpm
Now to install the python connector: # easy_install cx-Oracle
or, the recommended method: # pip install cx-Oracle
Installation for the version 12 client is much more straight forward than that for version 11.
Testing
A quick test to ensure that the expected versions appear, and that you can connect to the database. Python 2.6.6 (r266:84292, Nov 21 2013, 10:50:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
>>> cx_Oracle.version
'5.2'
>>> oraConn = "<USERNAME>/<PASSWORD>@<DATABASE HOST>:<DATABASE PORT>/<SERVICE>"
>>> ocDB = cx_Oracle.connect(oraConn)
>>> ocDB.version
'12.1.0.2.0'
Update: 15 April 2016 – Added information about which IP address to use and assigning static IP addresses for printers and servers
Recently I have needed to find an emergency alternative to my broadband due to the regional wide area network, Digital Region, being shut down, and the ISP Origin making a mess of getting all their cutomers onto ASDL. To get quickly back onto the internet, I have bought an ZTE MF823 4G Mobile Broadband Dongle as supplied by the badly named ‘three’ mobile phone company. As I have my own internal wired network, with multiple computers and ‘things’ there is a need to have something more sophisticated than just plugging the dongle into a single PC.
Here is my recipe for setting up a Raspberry Pi as a router with an ZTE MF283 Dongle. In this setup all the computers are on a wired Ethernet connection using a switch for the network. The Pi has Raspbian Debian Wheezy installed (June 2014) with all the latest updates made. For testing, the dongle is plugged into the USB port via a powered hub, and the Pi connected to a switch with another PC running Linux Mint.
Which IP addresses to use?
In this How-To I am using the IP address range 192.168.2.xxx this is to avoid conflict with the cable router which uses the 192.168.1.xxx range (the DHCP server is switched off on the router). IPv4 addresses are split into three different ranges, the 192.168.xxx.xxx range – 192.168.0.0 to 192.168.255.255 gives a possible 65,536 addresses but for your home it is unlikely you’ll have more than 255 network devices, so we can simplify things by limiting the address range used to 192.168.2.xxx and avoid the troublesome world of subnet masking.
192.168.xxx.xxx is used as its been designated for use on private networks by the Internet Assigned Numbers Authority this is a well established convention and is best practice. Two other IPv4 address ranges are available for larger private networks: 172.16.0.0-172.31.255.255 and 10.0.0.0-10.255.255.255 with 1,048,576 and 16,777,216 available addresses respectively, the most suitable network class should be chosen for your network.
Setup the Dongle
This USB dongle has its own built in dialer so you do not need ppp or wvdial installed, it appears as a USB ethernet device on the Raspberry Pi. You will need a powered USB hub as the dongle can draw more power than the Pi can provide, the symptoms of too much of power being drawn will be the Pi behaving erratically or restarting unexpectedly.
With the dongle plugged in, check that it is recognised by the Pi with lsusb, it can be seen here as ‘ZTE WCDMA Technologies MSM’:
The device ID is 19d2. and 1405 is the mode, this should be 1405 – CDC ethernet. If it is not, try removing the micro-SD card and rebooting the Pi, the device modes available are:
1225 – Default mode. USB Mass Storage Device + CD-ROM + card reader.
1403 – Modem mode. RNDIS + Mass Storage Device.
1405 – CDC ethernet
0016 – Download mode
As the dongle also has a Mass Storage Device the Raspberry may not switch to CDC ethernet. If the mode does not change, try the following with usb-modeswitch: $ sudo apt-get install usb-modeswitch
$ sudo usb_modeswitch -v 0x19d2 -p 0x1405 -d
I did not have to change the mode as it was correct already, and it didn’t change when I tried setting it as a Mass Storage device, I have not explored this any further.
When first plugged in the dongle was recognised as a ethernet device but it did not obtain an IP address:
ifconfig
Shell
1
2
3
4
5
6
7
$ifconfig
usb0Linkencap:Ethernet HWaddr36:4b:53:b7:e3:6e
UP BROADCAST RUNNING MULTICAST MTU:1500Metric:1
RX packets:0errors:0dropped:0overruns:0frame:0
TX packets:0errors:0dropped:0overruns:0carrier:0
collisions:0txqueuelen:1000
RX bytes:0(0.0B)TX bytes:0(0.0B)
if this is the case with you, add the following two lines to the end of sudo nano /etc/network/interfaces: auto usb0
iface usb0 inet dhcp
the dongle provides its own address to the computer. Reboot, and you should see the obtained address:
The address 192.168.0.185 is now the internet address of the computer the dongle always assigns this address, there is also a useful web status page on http://192.168.0.1
Configuring the network
first of all enable ip4 forwarding, edit the file sudo nano /etc/sysctl.conf and uncomment the line: net.ipv4.ip_forward=1
this will enable forwarding on reboot, you can also enable IP forwarding immediately with: $ sudo sysctl -w net.ipv4.ip_forward=1
We now need to give the Pi a static IP address on the internal network. Edit sudo nano /etc/network/interfaces so you end up with a file that looks like this:
interfaces
Shell
1
2
3
4
5
6
7
8
9
10
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address192.168.2.1
netmask255.255.255.0
auto usb0
iface usb0 inet dhcp
this gives the Pi a static IP address of 192.168.2.1.
DHCP
The next stage is to give the other computers on your network an IP address, this is done with a dhcp server: $ sudo apt-get install isc-dhcp-server
you will need to configure dhcp sudo nano /etc/dhcp/dhcpd.conf, here is mine:
dhcpd.conf
Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
default-lease-time600;
max-lease-time7200;
ddns-update-style none;
# If this DHCP server is the official DHCP server for the local
# network, the authoritative directive should be uncommented.
# give your network printer and file server a static IP addresses
hostprinter1{
hardware ethernet08:00:2b:4c:59:23;
fixed-address192.168.2.45;
}
hostfileserver{
hardware ethernet12:20:cb:3c:32:4d;
fixed-address192.168.2.40;
}
This will assign IP addresses in the range 192.168.2.50 to 192.168.2.150 to any computer connected to your network. I have used Open DNS for the Domain name Servers, if you wish to use google’s use: option domain-name-servers 8.8.8.8, 8.8.4.4;instead.
I have also given my network printer a static IP address, it is still assigned by the DHCP server but never changes, the same would apply to any file servers and the like, I would assign static devices addresses that are outside your dynamically assigned range. Reboot the Pi and then your test computer.
Your test computer should now have an IP address (192.168.2.51), and the gateway point to the Pi (92.168.2.1):
The final part is to have the incoming traffic on the the Ethernet port eth0, go out on the dongle usb0. This is achieved with iptables, a firewall and traffic router. Install with: $ sudo apt-get install iptables
and you need to setup Network Address Translation, NAT and forwarding. This short bash script clears any old settings before applying the new rules:
Where LAN is your internal network, and WAN is the internet. The final line allows you access to the Dongle’s built in web status page from any browser on your internal network, just use: http://192.168.2.1:2525
Save the file in your home directory as ~/ipt.sh, make it executable and run the script.
$ chmod +x ~/ipt.sh
$ sudo ~/ipt.sh
From your test computer, you will now be able to access the internet.
Finally, you now need to have iptables reload when you start the Pi. Export the iptables settings to a file with:
$ sudo sh -c "iptables-save > /etc/iptables.ipv4.nat"
and create a file sudo nano /etc/network/if-up.d/iptables with the following contents:
iptables
Shell
1
2
#!/bin/bash
/sbin/iptables-restore</etc/iptables.ipv4.nat
and make it executable sudo chmod +x /etc/network/if-up.d/iptables
after a reboot you can see your iptables with sudo iptables -L and sudo iptables -t nat -L and you can see web traffic passing through the router with sudo tcpdump -i any -nn port 80.
Adding a Proxy Server
This is optional, but a transparent proxy server and cache may reduce the amount of traffic on your 3G/4G connection, mileage varies and the amount of data cached was less than I thought it would be, I also found that my Humax Freesat box really didn’t like the proxy server and wouldn’t update its TV schedules while it was on. I have used squid3 for this. sudo apt-get install squid3
Update the squid3 configuration /etc/squid3/squid.conf so it has the following. The original is rather large, so you may want to make a copy and create a new one:
then restart squid3 sudo /etc/init.d squid3 restart
add the following iptables rule to redirect all traffic on port 80 to squid3: iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 3128
you should now be able to watch the web traffic being processed through squid3 with: sudo tail /var/log/squid3/access.log -f
finish off by exporting your iptables again, so they are reloaded on reboot: sudo sh -c "iptables-save > /etc/iptables.ipv4.nat" Raspberry Pi as 4G Router
I have been wanting to have a play with the OpenCV computer vision framework on Python for a while and finally got some time to some experimenting, I am looking to have the computer recognise LEGO parts, after much research and mucking about it seems I should be using cv2.SURF and/or cv2.SIFT for what I want to do. However on Fedora 19 these are not included in the distribution RPM as they are nonfree in that they are not open source. Attempts to use SIFT or SURF result in the following error:
$ python
Python 2.7.5 (default, Nov 12 2013, 16:18:42)
[GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> print cv2.__version__
2.4.6.1
>>> i = cv2.SURF()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'SURF'
>>> This is annoying as OpenCV will now need to be re-installed the old fashioned way, but there are instructions on the OpenCV site and I will be using them here with additional information I have discovered while following them.
Change to root and add the rpmfusion.org free and nonfree repositories as ffmpeg and libv41 are unavailble from Fedoras, followed by an update (it seems that Fedora 19 always has something to update):
& sudo su
# yum localinstall --nogpgcheck http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
# yum update
Remove the existing OpenCV packages:
# yum erase opencv*
There are a few packages to install in preparation to compiling the code, you may have some of all of these already but yum will skip those. Install the mandatory packages, these are for compiling the source and using GTK for the GUI:
# yum install cmake python-devel numpy gcc gcc-c++ gtk2-devel libdc1394-devel libv4l-devel ffmpeg-devel gstreamer-plugins-base-devel git wget
Install the optional dependencies, I have gone for the install everything approach, they may be useful later:
# yum install libpng-devel libjpeg-turbo-devel jasper-devel openexr-devel libtiff-devel libwebp-devel tbb-devel eigen3-devel python-sphinx texlive
Now, we are ready to get the latest source, here there are two ways to do this, install the latest development version using GIT, or download the latest stable version. My preference is to use the latest stable version.
With GIT, exit back to yourself from root, change to your home directory and download OpenCV:
# exit
$ cd ~
$ git clone https://github.com/Itseez/opencv.git
Cloning into 'opencv'...
$ cd opencv
$ mkdir build
$ cd build
Or using the latest build, at time of writing this is v2.4.7. Get this via the downloads page and save it to your home directory:
# exit
$ cd ~
$ tar -zxvf opencv-2.4.7.tar.gz
$ cd opencv-2.4.7
$ mkdir build
$ cd build
When complete you should check that your options agree with those displayed. There are many others available. Support for other programming languages may be missed out if they are not already installed. In my case those for Java, if I were to try OpenCV in Java I would need to install the appropriate Java packages using yum, then recompile OpenCV.
Now build and install, this can take a while:
$ make
$ sudo make install
You now need to move the module to anywhere on the Python Path, to find this:
$ python
Python 2.7.5 (default, Nov 12 2013, 16:18:42)
[GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print sys.path
['', '/usr/lib/python2.7/site-packages/PIL-1.1.7-py2.7-linux-x86_64.egg', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/PIL', '/usr/lib64/python2.7/site-packages/gst-0.10', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages', '/usr/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info']
The directory /usr/lib/python2.7/site-packages looks suitable:
$ sudo mv /usr/local/lib/python2.7/site-packages/cv2.so /usr/lib/python2.7/site-packages
And add your new installation to the PYTHONPATH, and add the export to the end of your .bashrc so it survives a reboot:
$ export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python2.7/site-packages
$ echo export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python2.7/site-packages >> ~/.bashrc
Now to test:
$ python
Python 2.7.5 (default, Nov 12 2013, 16:18:42)
[GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> print cv2.__version__
2.4.7
>>> i = cv2.SURF()
>>>
Sorted. Now, perhaps, I can get on with what I actually want to do….
Having been out and about with the Canon 6D, for once I had remembered to turn on the GPS logging built into the camera. Upon returning home after the seven mile walk I saved the GPS data in the cameras memory to the SD card. Now this LOG file found in the GPS directory of the SD card can be read directly into Google Earth but not into Memory Map for viewing on the far superior OS 1:25,000 Ordnance Survey map, also Memory Map does not read Google Earth’s KML or KMZ files.
So, the LOG file needs converting, but what format is it in? Canon are, it seems, rather unhelpful in revealing the secret but a little googeling found a reference to them using the NMEA-0183 format.
For conversion I found the free, excellent and very comprehensive gpsbabel to do the job (the PDF manual is over 200 pages), for me I convert the GPS files on a linux computer, but they do a Windows version and I assume the method and outcome will be the same. On Debian, install gpsbabel with: $ sudo apt-get install gpsbabel
The basic use of gpsbabel for converting is as follows: gpsbabel -i <input format> -f <input file> -o <output format> -F <output file>
As I know Memory Map reads the Garmin GPX format, I chose that as the output: $ gpsbabel -i nmea -f 13120100.LOG -o gpx -F 13120100.gpx
Installing the python cx_Oracle extension module for connecting to Oracle databases on this Fedora 18 workstation turned out to be a bit of a faff by giving an assortment of unhelpful error messages, if you are having the same pain maybe this will help.
You will need two files from the Oracle Database Instant Client download site, the basic client package and the SDK (devel): http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html For Linux choose the correct flavour for your installed operating system: x86 or x86-64 for 64bit operating systems, you will need to register on the site to access the files. For my purposes I got the 11.2 version for x64 RPM files, install them using: $ sudo rpm -i oracle-instantclient11.2-basic-11.2.0.3.0-1.x86_64.rpm
$ sudo rpm -i oracle-instantclient11.2-devel-11.2.0.3.0-1.x86_64.rpm
You will now need to tell the system where the libraries are: $ sudo su
# echo /usr/lib/oracle/11.2/client64/lib/ > /etc/ld.so.conf.d/oracle.conf
# ldconfig
# exit
To install cx_Oracle you will need to set some environment variables otherwise you will get an “error: cannot locate an Oracle software installation” message. The easy_install program is found in python-setuptools I have included it in the recipe as a reminder if you have not installed it already. $ sudo yum install python-setuptools
$ export ORACLE_HOME=/usr/lib/oracle/11.2/client64
$ export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
$ export PATH=$ORACLE_HOME/bin:$PATH
$ sudo -E easy_install cx_Oracle
or by downloading the version from http://cx-oracle.sourceforge.net/ and installing it manually, you will still need to set the exports, as above, and do the following for installation: $ sudo -E python setup.py build
$ sudo -E python setup.py install
The -E on the sudo takes your environment variables, including the three you just set, into your sudo session.
Success: $ python
Python 2.7.3 (default, Aug 9 2012, 17:23:57)
[GCC 4.7.1 20120720 (Red Hat 4.7.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
>>> exit()
When the the library cannot be seen, you get this error: $ python
Python 2.7.3 (default, Aug 9 2012, 17:23:57)
[GCC 4.7.1 20120720 (Red Hat 4.7.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
Traceback (most recent call last):
File "", line 1, in
ImportError: libclntsh.so.11.1: cannot open shared object file: No such file or directory
>>> exit()