Category Archives: linux

Setting up SSH in VS Code to connect to a Raspberry Pi

26th April 2022 Karl

There have been a lot of posts written about how to connect from Visual Studio Code (VS Code) on a Windows PC to a Raspberry PI over SSH using public/private keys avoiding the need to use a password. This is another one, and I aim to be fairly terse.

The PC is running Windows 11, the Raspberry Pi is running the latest version of Raspberry Pi OS – Debian Bullseye. Note that <username> is a placeholder for your own username, the greater than symbol: > represents a Windows PowerShell prompt and the dollar sign: $ is a Linux Bash prompt.

Setup The PI

SSH needs enabling on the PI and an ~/.ssh directory needs creating in your users home directory.

On the Pi, in the terminal enable SSH with: sudo raspi-config then select 3 Interface Options > I2 SSH and choose Yes – Enable. Now create the directory where your keys will be kept: mkdir ~/.ssh

It is also useful to find the IP address of your PI with: hostname -I Mine is 192.168.2.45 and I shall being using this in the examples.

You should now be able to login to the Pi from the Windows PC, open Windows Terminal or PowerShell and use ssh <username>@192.168.2.45 answer yes to add it to the known hosts file and it will ask for your PI’s password. If your Windows username is the same as that on the PI you can login with ssh 192.168.2.45.

Setup Windows for SSH

This stage installs the SSH client on the PC, creates a keypair and copied the public key to the PI.

The OpenSSH client can be installed using Windows Settings on Windows Server 2019 and Windows 10 and above devices.

Open Settings, select Apps > Apps & Features, then select Optional Features.
Scan the list to see if the OpenSSH is already installed. If not, at the top of the page, select Add a feature, then:
Find OpenSSH Client, then click Install

A full description is given on Microsofts Get started with OpenSSH page.

On the Windows PC, in your PowerShell create the directory for your keys if it doesn’t exist already mkdir ~/.ssh and then change into it cd ~/.ssh generate a keypair and add it to the ssh-agent:

> ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (C:\Users\<username>/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\Users\<username>/.ssh/id_ed25519.
Your public key has been saved in C:\Users\<username>/.ssh/id_ed25519.pub.
The key fingerprint is:
SHA256:YCETnl4GmmYSokjE+vE5UaJ1/nzXB1tboE730sxkujw <username>@<computer>
The key's randomart image is:
+--[ED25519 256]--+
|=o  =..          |
|=o +o=o.      .  |
|= =oo==      . . |
|.+o..+..    o o =|
| . o.o oS  o o %o|
|  . +   o . o =.*|
|     .   . . . + |
|              E  |
|               . |
+----[SHA256]-----+
PS C:\Users\<username>\.ssh> ssh-add ~\.ssh\id_ed25519

> ssh-keygen -t ed25519

Generating public/private ed25519 key pair.

Enter file in which to save the key (C:\Users\<username>/.ssh/id_ed25519):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in C:\Users\<username>/.ssh/id_ed25519.

Your public key has been saved in C:\Users\<username>/.ssh/id_ed25519.pub.

The key fingerprint is:

SHA256:YCETnl4GmmYSokjE+vE5UaJ1/nzXB1tboE730sxkujw <username>@<computer>

The key's randomart image is:

+--[ED25519 256]--+

|=o =.. |

|=o +o=o. . |

|= =oo== . . |

|.+o..+.. o o =|

| . o.o oS o o %o|

| . + o . o =.*|

| . . . . + |

| E |

| . |

+----[SHA256]-----+

PS C:\Users\<username>\.ssh> ssh-add ~\.ssh\id_ed25519

We now need to copy the public key to the PI:

> scp ~\.ssh\id_ed25519.pub <username>@192.168.2.45:~/.ssh/

1	> scp ~\.ssh\id_ed25519.pub <username>@192.168.2.45:~/.ssh/

And on the PI we need to add that public key to the authorized keys, connect into the PI ssh <username>@192.168.2.45 and:

$ cd ~/.ssh
$ touch authorized_keys
$ cat id_ed25519.pub >> authorized_keys
$ chmod 600 authorized_keys
$ rm id_ed25519.pub

$ cd ~/.ssh

$ touch authorized_keys

$ cat id_ed25519.pub >> authorized_keys

$ chmod 600 authorized_keys

$ rm id_ed25519.pub

the commands, touch creates a blank file if it doesn’t exist, cat appends the pub file to authorized_keys, making sure that any existing keys are preserved permissions are set with chmod and finally the pub file is deleted.

Setup VS Code

Load up VS Code and add the Remote – SSH and Remote SSH: Editing Configuration Files extensions, once restarted you will see a >< icon in the bottom left-hand corner with a green background, click that and select Edit SSH Configuration file…, choose the one in your home directory C:\Users\<username>\.ssh\config and add the following:

Host piComputer_192.168.2.45
  HostName 192.168.2.45
  User <username>
  ForwardAgent yes
  IdentityFile C:/Users/<username>/.ssh/id_ed25519

Host piComputer_192.168.2.45

HostName 192.168.2.45

User <username>

ForwardAgent yes

IdentityFile C:/Users/<username>/.ssh/id_ed25519

The Host “piComputer_192.168.2.45” is the friendly name displayed within VS Code for when you are choosing it from lists, the User id your user on the PI. Save the config file when complete.

You should now be able to connect to the PI. Click the >< icon in the bottom left-hand corner, and choose Connect To Host… A popup saying Setting Up… should appear, it may ask if the host is Linux or Windows, choose Linux, and within a few seconds it’ll be connected and ready. From within VS Code you can open a Bash Terminal – Ctrl+Shift+’

If it asks for a password, you may have missed a stage, or copied the private key to the PI instead of the public one.

Links and Sources

Computer, linux, Raspberry Pi

Setup a Network Camera with a Raspberry PI

1st July 2021 Karl 2 Comments

In this post I am setting up a network available RTSP video camera on a Raspberry Pi 4 Model B with the latest Debian Buster installed, you will also want another ‘remote’ computer with VLC installed on the same network. I have used a camera module connected via a ribbon cable to the Pi’s video port. The performance is more like what you would see from a security camera rather than some fancy dedicated device. I will be making the configuration with a bash shell using SSH rather than the GUI.

Installation

If you haven’t already, enable the camera on the Pi with sudo raspi-config and go to Interface Options > Camera. To reduce the load on the computer I also set the Pi to boot to the Console in System Options > Boot / Auto Login and selecting Console, you should also check that at least 256MB of memory is allocated to the GPU in Performance Options > GPU Memory, when complete reboot to set the updated configuration.

Install some prerequisites, you may have some or all of these installed already:

$ sudo apt update
$ sudo apt install curl git build-essential software-properties-common cmake

1 2	$ sudo apt update $ sudo apt install curl git build-essential software-properties-common cmake

Next add the Video For Linux (V4L) Repository, this adds the drivers for the video on the hardware side

$ curl https://www.linux-projects.org/listing/uv4l_repo/lpkey.asc | sudo apt-key add -

1	$ curl https://www.linux-projects.org/listing/uv4l_repo/lpkey.asc \| sudo apt-key add -

and add the following line to your apt sources with $ sudo nano /etc/apt/sources.list

deb https://www.linux-projects.org/listing/uv4l_repo/raspbian/stretch stretch main

1	deb https://www.linux-projects.org/listing/uv4l_repo/raspbian/stretch stretch main

We can now install the V4L packages:

$ sudo apt update
$ sudo apt install uv4l uv4l-raspicam liblog4cpp5-dev libv4l-dev uv4l-server uv4l-raspicam-extras

1 2	$ sudo apt update $ sudo apt install uv4l uv4l-raspicam liblog4cpp5-dev libv4l-dev uv4l-server uv4l-raspicam-extras

We now need to install the RTSP server from Git, this server connects to the V4L drivers above:

$ cd ~
$ git clone https://github.com/mpromonet/v4l2rtspserver.git
$ cd v4l2rtspserver
$ cmake .
$ sudo make install
$ sudo reboot

$ cd ~

$ git clone https://github.com/mpromonet/v4l2rtspserver.git

$ cd v4l2rtspserver

$ cmake .

$ sudo make install

$ sudo reboot

Configuration

To setup V4L, edit the configuration file $ sudo nano /etc/uv4l/uv4l-raspicam.conf and update the driver options and uncomment the h264 options:

##################################
# raspicam driver options
##################################
encoding = h264
#720p
width = 1280
height = 720
framerate = 12

### h264 options:
profile = high
level = 4.2
bitrate = 8000000
intra-refresh-mode = dummy
intra-period = #arg
inline-headers = yes
sps-timing = no
quantisation-parameter #arg

##################################

# raspicam driver options

##################################

encoding = h264

#720p

width = 1280

height = 720

framerate = 12

### h264 options:

profile = high

level = 4.2

bitrate = 8000000

intra-refresh-mode = dummy

intra-period = #arg

inline-headers = yes

sps-timing = no

quantisation-parameter #arg

Restart the service once you have saved the file:

$ sudo service uv4l_raspicam restart

1	$ sudo service uv4l_raspicam restart

Useful screen Resolutions; designation, ratio: width x height:

1080p, 16:9: 1920 x 1080
720p, 16:9: 1280 x 720
720, 4:3: 1296 x 792
Standard Definition, 4:3: 640 x 480

I’ve chosen 720p 16:9 widescreen, this gives a good image quality without stressing the Pi 4B. The framerate is a compromise between quality and usability, its unlikely you would want to go above 25fps.

Connecting

If you don’t know it already, find the IP address of your Pi with $ ip a. Look for the address in eth0 or wlan0 depending on if you are using ethernet or wireless. In this case I will be using 192.168.2.33 in the examples.

Start the RTSP server with:

$ v4l2rtspserver -U user:password -F12 -W 1280 -H 720 -P 8555 /dev/video0

1	$ v4l2rtspserver -U user:password -F12 -W 1280 -H 720 -P 8555 /dev/video0

Replace user:password with a suitable username and password. I have assumed that the camera is on /dev/video0, you can test this with:

$ v4l2-ctl -d /dev/video0 -D

1	$ v4l2-ctl -d /dev/video0 -D

and look for a line such as Driver version: 5.10.17, if this is 0.0.0 then try a different device, /dev/video1, /dev/video2 etc.

On a successful load it will tell you the connection:

2021-07-01 10:07:39,251 [NOTICE] - /home/pi/v4l2rtspserver/inc/V4l2RTSPServer.h:81
        Play this stream using the URL "rtsp://192.168.2.33:8555/unicast"

1 2	2021-07-01 10:07:39,251 [NOTICE] - /home/pi/v4l2rtspserver/inc/V4l2RTSPServer.h:81 Play this stream using the URL "rtsp://192.168.2.33:8555/unicast"

Now on your ‘remote’ computer open VLC and choose Media > Open Network Stream and place the given stream address into the network URL, you will then be asked for the username and password. For me it took less than five seconds for an image to appear, there is a fairly high latency on the image shown, the picture lags by about a second from reality.

Using OpenCV

You can access the stream using Python scripts and OpenCV on your ‘remote’ computer for AI/Machine Learning. For me this is a Debian Linux box with FFmpeg as OpenCV uses this to access the stream.

#!/bin/env python

import os
import cv2

# if the script doesn't work, uncomment the following, OpenCV may be 
# defaulting to TCP when we are wanting to use UDP.
#os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = "rtsp_transport;udp"

# replace user:password with your username and password
rtspUrl = "rtsp://user:password@192.168.2.33:8555/unicast"

cap = cv2.VideoCapture(rtspUrl)  

#Check if camera was opened correctly
if not (cap.isOpened()):
    print("Could not open video device")

#Set the resolution
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1280)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 720)

while(True):
    ret, frame = cap.read()
    cv2.imshow('VIDEO', frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

#!/bin/env python

import os

import cv2

# if the script doesn't work, uncomment the following, OpenCV may be

# defaulting to TCP when we are wanting to use UDP.

#os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = "rtsp_transport;udp"

# replace user:password with your username and password

rtspUrl = "rtsp://user:password@192.168.2.33:8555/unicast"

cap = cv2.VideoCapture(rtspUrl)

#Check if camera was opened correctly

if not (cap.isOpened()):

print("Could not open video device")

#Set the resolution

cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1280)

cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 720)

while(True):

ret, frame = cap.read()

cv2.imshow('VIDEO', frame)

if cv2.waitKey(1) & 0xFF == ord('q'):

break

cap.release()

cv2.destroyAllWindows()

You can check OpenCV is working with FFmpeg with:

$ python -c "import cv2; print(cv2.getBuildInformation())"

1	$ python -c "import cv2; print(cv2.getBuildInformation())"

and looking in the output for FFMPEG saying YES:

 Video I/O:
    DC1394:                      NO
    FFMPEG:                      YES
      avcodec:                   YES (58.109.100)
      avformat:                  YES (58.61.100)
      avutil:                    YES (56.60.100)
      swscale:                   YES (5.8.100)
      avresample:                NO
    GStreamer:                   NO
    v4l/v4l2:                    YES (linux/videodev2.h)

Video I/O:

DC1394: NO

FFMPEG: YES

avcodec: YES (58.109.100)

avformat: YES (58.61.100)

avutil: YES (56.60.100)

swscale: YES (5.8.100)

avresample: NO

GStreamer: NO

v4l/v4l2: YES (linux/videodev2.h)

Links and Sources

V4L Installation: https://www.linux-projects.org/uv4l/installation/
RTSP Server https://github.com/mpromonet/v4l2rtspserver

Computer, Jetson, linux, Programming

Jetson Nano Install Notes

8th October 2020 Karl

The Jetson Nano Developer kit – B01 is a small computer comprising of an NVIDA Maxwell GPU, Quad-Core ARM Cortex-A57 Processor and 4GB of Memory along with four USB 3 ports, Gigabit Ethernet, HDMI and Display Port output, main storage is on a MicroSD card and there is a variety of selection of expansion available via GPIO, I2C and UART. On the software side NVIDIA provide their JetPack SDK – a customised version of Ubuntu. This development kit has been produced to provide an entry point into Machine Learning, for which I will be using Python programming language. I got my board from Pimoroni

GPU	128-core Maxwell
CPU	Quad-core ARM A57 @ 1.43 GHz
Memory	4 GB 64-bit LPDDR4 25.6 GB/s
Storage	microSD (not included)
Video Encode	4K @ 30 \| 4x 1080p @ 30 \| 9x 720p @ 30 (H.264/H.265)
Video Decode	4K @ 60 \| 2x 4K @ 30 \| 8x 1080p @ 30 \| 18x 720p @ 30 (H.264/H.265)
Camera	2x MIPI CSI-2 DPHY lanes
Connectivity	Gigabit Ethernet, M.2 Key E
Display	HDMI and display port
USB	4x USB 3.0, USB 2.0 Micro-B
Others	GPIO, I2C, I2S, SPI, UART
Mechanical	69 mm x 45 mm, 260-pin edge connector

These notes cover my process of setting one up and links to the documentation, it not intended to repeat those install guides but to provide an install sequence and any additional commentary as needed. I’m going to assume you have a little experience of using the terminal and am familiar with using the bash command line – I’ve no idea how this would be done through the GUI.

jetson nano ports — Jetson Nano ports [source: NVIDIA]

Initial Startup

I followed the instructions for downloading and installing JetPack 4.4 on https://nvidia.com/jetsonnano-start I used a 64GB Class 10, UHS-I, U3, V30 SanDisk card. I formatted the card in a camera before using balenaEtcher to write the JetPack SDK image, this creates a partition of about 16GB on the card formatted to ext4, during installation the volume is resized to fill the card.

jetson nano connections — Jetson Nano connections [source: NVIDIA]

Despite using a good quality USB power supply with an output of 3 Amps at 5 Volts into the Micro USB port the computer would only boot long enough for the NVIDIA logo to appear on screen but after a few seconds the green power LED would go out and it would be off, the same happened when I tried a variety of USB power supplies used. I got round the problem buy using a 5 Amp power supply connected to the barrel jack J25 on the left (centre pin positive) and connecting the jumper J48 located just behind this connector.

Customising the Setup

There are a couple of thigs to do, get a network volume mounted and set the default version of python.

For the network share install samba, some network utilities and the nano text editor:

$ sudo apt install samba cifs-utils nano

1	$ sudo apt install samba cifs-utils nano

create a text file: sudo nano /etc/samaba/videoserver with the following:

username=<your network username>
password=<your network password>

1 2	username=<your network username> password=<your network password>

And set the permissions sudo chmod 600 /etc/samba/videoserver. In this example I have a network share on my server; 192.168.1.30, called video. Create a mount point for the share: sudo mkdir /mnt/video now you need to edit fstab, sudo nano /etc/fstab and add your network connection to the end:

//192.168.1.30/video /mnt/video cifs credentials=/etc/samba/videoserver,noauto,x-systemd.automount,_netdev,uid=0,vers=1.0 0 0

1	//192.168.1.30/video /mnt/video cifs credentials=/etc/samba/videoserver,noauto,x-systemd.automount,_netdev,uid=0,vers=1.0 0 0

Reload fstab with sudo mount -a and check for any errors. Because of the way that Jetpack boots it does not appear to wait for the network so the share needs to be set to automount and this causes it to only appear in drive listings when accessed. Further reading can be found in this excellent guide to fstab: https://wiki.archlinux.org/index.php/fstab.

Jetpack 4.4 comes with two versions of Python, 2.7 and 3.6, I want it to default to 3.6 and while this is rather out of date I don’t want to go down the hole of upgrading just yet, you will also need to install pip and set pip3 as the default too.

$ sudo apt install python3-pip
$ sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.6 1
$ sudo update-alternatives --install /usr/bin/pip pip /usr/bin/pip3.6 1
$ sudo update-alternatives --set python /usr/bin/python3.6
$ sudo update-alternatives --set pip /usr/bin/pip3.6

$ sudo apt install python3-pip

$ sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.6 1

$ sudo update-alternatives --install /usr/bin/pip pip /usr/bin/pip3.6 1

$ sudo update-alternatives --set python /usr/bin/python3.6

$ sudo update-alternatives --set pip /usr/bin/pip3.6

and test its worked:

$ python
Python 3.6.9 (default, Jul 17 2020, 12:50:27)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

$ python

Python 3.6.9 (default, Jul 17 2020, 12:50:27)

[GCC 8.4.0] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>>

$ pip --version
pip 20.2.3 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)

1 2	$ pip --version pip 20.2.3 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)

I did get an error later on, a crash was reported on the desktop when an occasional python 2 script ran. I fixed the error in /usr/sbin/l4t_payload_updater_t210 by changing the first line of the file from !#/user/bin/python to !#/user/bin/python2

Post Install Problems

A recent update occured, so I did the usual sudo apt get update && sudo apt get upgrade but one of the files gave a script error, this turned out to be with nvidia-l4t-bootloader, like so:

$ sudo apt install nvidia-l4t-bootloader
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed
  nvidia-l4t-bootloader
0 to upgrade, 1 to newly install, 0 to remove and 0 not to upgrade.
Need to get 1,870 kB of archives.
After this operation, 8,251 kB of additional disk space will be used.
Get:1 https://repo.download.nvidia.com/jetson/t210 r32.4/main arm64 nvidia-l4t-bootloader arm64 32.4.3-20200924161615 [1,870 kB]
Fetched 1,870 kB in 3s (692 kB/s)
Selecting previously unselected package nvidia-l4t-bootloader.
(Reading database ... 165354 files and directories currently installed.)
Preparing to unpack .../nvidia-l4t-bootloader_32.4.3-20200924161615_arm64.deb ...
Unpacking nvidia-l4t-bootloader (32.4.3-20200924161615) ...
Setting up nvidia-l4t-bootloader (32.4.3-20200924161615) ...
3448-300---1--jetson-nano-qspi-sd-mmcblk0p1
Starting bootloader post-install procedure.
ERROR. Procedure for bootloader update FAILED.
Cannot install package. Exiting...
dpkg: error processing package nvidia-l4t-bootloader (--configure):
 installed nvidia-l4t-bootloader package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 nvidia-l4t-bootloader
E: Sub-process /usr/bin/dpkg returned an error code (1)

$ sudo apt install nvidia-l4t-bootloader

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following NEW packages will be installed

nvidia-l4t-bootloader

0 to upgrade, 1 to newly install, 0 to remove and 0 not to upgrade.

Need to get 1,870 kB of archives.

After this operation, 8,251 kB of additional disk space will be used.

Get:1 https://repo.download.nvidia.com/jetson/t210 r32.4/main arm64 nvidia-l4t-bootloader arm64 32.4.3-20200924161615 [1,870 kB]

Fetched 1,870 kB in 3s (692 kB/s)

Selecting previously unselected package nvidia-l4t-bootloader.

(Reading database ... 165354 files and directories currently installed.)

Preparing to unpack .../nvidia-l4t-bootloader_32.4.3-20200924161615_arm64.deb ...

Unpacking nvidia-l4t-bootloader (32.4.3-20200924161615) ...

Setting up nvidia-l4t-bootloader (32.4.3-20200924161615) ...

3448-300---1--jetson-nano-qspi-sd-mmcblk0p1

Starting bootloader post-install procedure.

ERROR. Procedure for bootloader update FAILED.

Cannot install package. Exiting...

dpkg: error processing package nvidia-l4t-bootloader (--configure):

installed nvidia-l4t-bootloader package post-installation script subprocess returned error exit status 1

Errors were encountered while processing:

nvidia-l4t-bootloader

E: Sub-process /usr/bin/dpkg returned an error code (1)

after some head scratching, it turns out this script is python2 only. I reverted back to the python 2.7 as the default, and tried again:

$ sudo update-alternatives --install /usr/bin/python python /usr/bin/python2.7 2
update-alternatives: using /usr/bin/python2.7 to provide /usr/bin/python (python) in auto mode
$ sudo update-alternatives --set python /usr/bin/python2.7
$ sudo apt remove nvidia-l4t-bootloader
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package 'nvidia-l4t-bootloader' is not installed, so not removed
0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.
$ sudo apt install nvidia-l4t-bootloader
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed
  nvidia-l4t-bootloader
0 to upgrade, 1 to newly install, 0 to remove and 0 not to upgrade.
Need to get 0 B/1,870 kB of archives.
After this operation, 8,251 kB of additional disk space will be used.
Selecting previously unselected package nvidia-l4t-bootloader.
(Reading database ... 165354 files and directories currently installed.)
Preparing to unpack .../nvidia-l4t-bootloader_32.4.3-20200924161615_arm64.deb ...
Unpacking nvidia-l4t-bootloader (32.4.3-20200924161615) ...
Setting up nvidia-l4t-bootloader (32.4.3-20200924161615) ...
3448-300---1--jetson-nano-qspi-sd-mmcblk0p1
Starting bootloader post-install procedure.
Update bootloader completed.
Reboot the target system for changes to take effect.
Updating extlinux.conf...
Root device is set in the extlinux.conf

$ sudo update-alternatives --install /usr/bin/python python /usr/bin/python2.7 2

update-alternatives: using /usr/bin/python2.7 to provide /usr/bin/python (python) in auto mode

$ sudo update-alternatives --set python /usr/bin/python2.7

$ sudo apt remove nvidia-l4t-bootloader

Reading package lists... Done

Building dependency tree

Reading state information... Done

Package 'nvidia-l4t-bootloader' is not installed, so not removed

0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.

$ sudo apt install nvidia-l4t-bootloader

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following NEW packages will be installed

nvidia-l4t-bootloader

0 to upgrade, 1 to newly install, 0 to remove and 0 not to upgrade.

Need to get 0 B/1,870 kB of archives.

After this operation, 8,251 kB of additional disk space will be used.

Selecting previously unselected package nvidia-l4t-bootloader.

(Reading database ... 165354 files and directories currently installed.)

Preparing to unpack .../nvidia-l4t-bootloader_32.4.3-20200924161615_arm64.deb ...

Unpacking nvidia-l4t-bootloader (32.4.3-20200924161615) ...

Setting up nvidia-l4t-bootloader (32.4.3-20200924161615) ...

3448-300---1--jetson-nano-qspi-sd-mmcblk0p1

Starting bootloader post-install procedure.

Update bootloader completed.

Reboot the target system for changes to take effect.

Updating extlinux.conf...

Root device is set in the extlinux.conf

Afterwards I set the default back to python 3.6 again.

Setup the Machine Learning

There are three Machine Learning packages, OpenCV for Computer Vison, Tensorflow for machine learning models, and TensorRT – accelerated deep learning networks for image recognition.

OpenCV is already installed in Python:

>>> import cv2
>>> print (cv2.__version__)
4.1.1

>>> import cv2

>>> print (cv2.__version__)

4.1.1

Tensorflow installation can be found here: https://docs.nvidia.com/deeplearning/frameworks/install-tf-jetson-platform/ I found this to be straight forward but it did keep the computer busy for a while.

>>> import tensorflow
2020-10-08 11:34:55.934735: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
>>> print (tensorflow.__version__)
2.3.0

>>> import tensorflow

2020-10-08 11:34:55.934735: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2

>>> print (tensorflow.__version__)

2.3.0

TensorRT can be found at: https://github.com/dusty-nv/jetson-inference/blob/master/docs/building-repo-2.md I used the instructions in the Quick Reference section and installed the default models without any errors, I have not yet had the opportunity to test it.

Copying and Resizing a Raspberry Pi SD Card

21st June 2020 Karl

The Micro SD Card I use in a Raspberry PI ran out of space so here is how I copied the contents of the 15GB drive to a new 64GB card and resized the partition. I used a separate computer running Debian and as the machine does not have a monitor or keyboard attached this will be being completed through the bash command line using SSH.

1. Making a copy of the SD Card.

Insert the old card into your computer, if the computer attempts to mount the drive then unmount it. We need to find which mount point has been used, do this with the lsblk command:

$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0   477G  0 disk
├─sda1   8:1    0   512M  0 part /boot/efi
├─sda2   8:2    0 468.9G  0 part /
└─sda3   8:3    0   7.6G  0 part [SWAP]
sdb      8:16   0   477G  0 disk
└─sdb1   8:17   0   477G  0 part /mnt/external_ssd
sdc      8:32   0   1.8T  0 disk
└─sdc1   8:33   0   1.8T  0 part /data
sdd      8:48   0   3.7T  0 disk
└─sdd1   8:49   0   3.7T  0 part /mnt/external_hdd
sde      8:64   1  14.9G  0 disk
├─sde1   8:65   1  43.8M  0 part
└─sde2   8:66   1  14.8G  0 part

$ lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

sda 8:0 0 477G 0 disk

├─sda1 8:1 0 512M 0 part /boot/efi

├─sda2 8:2 0 468.9G 0 part /

└─sda3 8:3 0 7.6G 0 part [SWAP]

sdb 8:16 0 477G 0 disk

└─sdb1 8:17 0 477G 0 part /mnt/external_ssd

sdc 8:32 0 1.8T 0 disk

└─sdc1 8:33 0 1.8T 0 part /data

sdd 8:48 0 3.7T 0 disk

└─sdd1 8:49 0 3.7T 0 part /mnt/external_hdd

sde 8:64 1 14.9G 0 disk

├─sde1 8:65 1 43.8M 0 part

└─sde2 8:66 1 14.8G 0 part

The device sde matches our SD card. So we will use that. The dd command is used to create the ISO image, I am creating the file in my home directory:

$ cd ~
$ sudo dd status=progress if=/dev/sde of=raspberry_cnc.iso

1 2	$ cd ~ $ sudo dd status=progress if=/dev/sde of=raspberry_cnc.iso

once complete, check that the records match:

31116288+0 records in
31116288+0 records out
15931539456 bytes (16 GB, 15 GiB) copied, 464.124 s, 34.3 MB/s

31116288+0 records in

31116288+0 records out

15931539456 bytes (16 GB, 15 GiB) copied, 464.124 s, 34.3 MB/s

Swap the SD cards and check the mount point again:

$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
[snip]
sde      8:64   1  59.5G  0 disk
└─sde1   8:65   1  59.5G  0 part

$ lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

[snip]

sde 8:64 1 59.5G 0 disk

└─sde1 8:65 1 59.5G 0 part

Now copy the iso image to the new card:

$ sudo dd status=progress if=raspberry_cnc.iso of=/dev/sde

1	$ sudo dd status=progress if=raspberry_cnc.iso of=/dev/sde

this can take a while, the write speed is rather slow:

31116288+0 records in
31116288+0 records out
15931539456 bytes (16 GB, 15 GiB) copied, 3023.22 s, 5.3 MB/s

31116288+0 records in

31116288+0 records out

15931539456 bytes (16 GB, 15 GiB) copied, 3023.22 s, 5.3 MB/s

and lsblk shows us the new partitions:

$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
[snip]
sde      8:64   1  59.5G  0 disk
├─sde1   8:65   1  43.8M  0 part
└─sde2   8:66   1  14.8G  0 part

$ lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

[snip]

sde 8:64 1 59.5G 0 disk

├─sde1 8:65 1 43.8M 0 part

└─sde2 8:66 1 14.8G 0 part

2. Resizing the partition

The copied partition is now the same size as the original. If you have space remaining, the new card can be put back in the Pi and use the raspi-config utility, and using the Expand Filesystem option in the Advanced Settings section. However if the drive is completely full you won’t be able to login as there won’t be enough space available for the temporary files created at login, to get round this you can use parted to resize, start with:

$ sudo parted /dev/sde

1	$ sudo parted /dev/sde

if you get the following message:

Warning: Unable to open /dev/sde read-write (Read-only file system).  /dev/sde has been opened read-only.

1	Warning: Unable to open /dev/sde read-write (Read-only file system). /dev/sde has been opened read-only.

then quit from parted, if you are using a full size SD Card, check the Write Protect tab on the side of the card and try again, otherwise try:

$ sudo hdparm -r0 /dev/sde

/dev/sde:
 setting readonly to 0 (off)
 readonly      =  0 (off)

$ sudo hdparm -r0 /dev/sde

/dev/sde:

setting readonly to 0 (off)

readonly = 0 (off)

If problem persists try formatting the new card in a camera, as these have a simple file system, and write the ISO image again.

We need to resize the larger partition /dev/sde2 with the ext4 file system, the smaller is used to boot the Pi and can be ignored. In parted, list the partitions with the print command:

$ sudo parted /dev/sde
(parted) print
Model: Generic- USB3.0 CRW -SD (scsi)
Disk /dev/sde: 63.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  50.1MB  45.9MB  primary  fat32        lba
 2      50.3MB  15.9GB  15.9GB  primary  ext4

$ sudo parted /dev/sde

(parted) print

Model: Generic- USB3.0 CRW -SD (scsi)

Disk /dev/sde: 63.9GB

Sector size (logical/physical): 512B/512B

Partition Table: msdos

Disk Flags:

Number Start End Size Type File system Flags

1 4194kB 50.1MB 45.9MB primary fat32 lba

2 50.3MB 15.9GB 15.9GB primary ext4

Using resizepart set the new size, I set this to larger than the 15GB but smaller than the unallocated space, this to save me having to accurately work out the remaining space manually:

(parted) resizepart 2
End?  [15.9GB]? 40GB

1 2	(parted) resizepart 2 End? [15.9GB]? 40GB

Now update the boundaries to grow and resize the partition into the freshly allocated space:

$ sudo /sbin/e2fsck -f /dev/sde2
$ sudo /sbin/resize2fs /dev/sde2

1 2	$ sudo /sbin/e2fsck -f /dev/sde2 $ sudo /sbin/resize2fs /dev/sde2

Now boot the Pi with the new card, login and use the raspi-config utility then in Advanced Options choose Expand Filesystem and follow the onscreen instructions. Once rebooted you should now be set to fill up your new card.

Links and Sources

RPi Resize Flash Partitions https://elinux.org/RPi_Resize_Flash_Partitions
Parted Guide https://www.tecmint.com/parted-command-to-create-resize-rescue-linux-disk-partitions/

Computer, linux, Programming

Extracting GPS data from the GoPro 7

28th July 2019 Karl 2 Comments

While extracting the telemetry data from the GoPro is reasonably well documented I have found some gaps for getting the extracting utilities installed and when extracting and combining data from multiple files. These notes are for a Debian/Ubuntu installation in a BASH Shell.

Installing the gopro-utils

As I couldn’t find any straightforward instructions for installation, I’ll be going through everything I needed to do to get it working, you may have some of these packages installed already.

sudo apt update
sudo apt upgrade
sudo apt install ffmpeg golang gpsbabel git

sudo apt update

sudo apt upgrade

sudo apt install ffmpeg golang gpsbabel git

Now to get the gopro-utils and install them, I’m placing the source files into my Downloads directory, as well as the GPS data extractor we’ll be adding the other telemetry tools too, this is all a bit long winded.

cd ~/Downloads
git clone https://github.com/JuanIrache/gopro-utils
cd gopro-utils
go get github.com/JuanIrache/gopro-utils/telemetry
go get github.com/mlouielu/gpxgo/gpx
cd bin/gopro2gpx
go build gopro2gpx.go
sudo cp gopro2gpx /usr/local/bin/
cd ../bin/gopro2geojson/
go build gopro2geojson.go
sudo cp gopro2geojson /usr/local/bin/
cd ../gopro2json/
go build gopro2json.go
sudo cp gopro2json /usr/local/bin/
cd ../gpmd2csv/
go build gpmd2csv.go
sudo cp gpmd2csv /usr/local/bin/
cd ../gpmd2info/
go build gpmd2info.go
sudo cp gpmd2info /usr/local/bin/
cd ../gps2kml/
go build gps2kml.go
sudo cp gps2kml /usr/local/bin/

cd ~/Downloads

git clone https://github.com/JuanIrache/gopro-utils

cd gopro-utils

go get github.com/JuanIrache/gopro-utils/telemetry

go get github.com/mlouielu/gpxgo/gpx

cd bin/gopro2gpx

go build gopro2gpx.go

sudo cp gopro2gpx /usr/local/bin/

cd ../bin/gopro2geojson/

go build gopro2geojson.go

sudo cp gopro2geojson /usr/local/bin/

cd ../gopro2json/

go build gopro2json.go

sudo cp gopro2json /usr/local/bin/

cd ../gpmd2csv/

go build gpmd2csv.go

sudo cp gpmd2csv /usr/local/bin/

cd ../gpmd2info/

go build gpmd2info.go

sudo cp gpmd2info /usr/local/bin/

cd ../gps2kml/

go build gps2kml.go

sudo cp gps2kml /usr/local/bin/

Extracting the Data

You will need to find which stream in the video recording the data has been saved to, to find this use ffprobe to examine the recording and look for the stream that contains GoPro MET, for example:

$ ffprobe GX090155.MP4
[snip]
    Stream #0:3(eng): Data: none (gpmd / 0x646D7067), 40 kb/s (default)
    Metadata:
      creation_time   : 2019-07-28T11:09:25.000000Z
      handler_name    : GoPro MET
[snip]

$ ffprobe GX090155.MP4

[snip]

Stream #0:3(eng): Data: none (gpmd / 0x646D7067), 40 kb/s (default)

Metadata:

creation_time : 2019-07-28T11:09:25.000000Z

handler_name : GoPro MET

[snip]

You can see that what we are wanting is on stream 3, as far as I can tell this stays the same every time, I don’t know if it is different for other GoPro models.

This bash script extracts the GPS data in GPX format from all the GoPro GX recordings in the directory, other options have been commented out, if you are using Garmin VIRB edit there is also an option for use with that. The script creates two files, one that contains the raw data and another with the desired GPS data, the GPS output file has the same name as the recording, but in lowercase with a .gpx extension.

#!/bin/bash
FILES="GX*.MP4"
STREAM="0:3"
for MP4FILE in $FILES; do
   OUTFILE=$(basename "$MP4FILE" | cut -d. -f1)
   
   OUTFILE=$(echo "$OUTFILE" | tr '[:upper:]' '[:lower:]')
   BINFILE="$OUTFILE.bin"
   ffmpeg -y -i "$MP4FILE" -codec copy -map $STREAM -f rawvideo "$BINFILE"

   #gpmd2csv -i "$BINFILE" -o "$OUTFILE.csv"
   gopro2gpx -i "$BINFILE" -a 500 -f 2 -o "$OUTFILE.gpx"
   #gopro2gpx -i "$BINFILE" -a 200 -f 3 -o "$OUTFILE-virb.gpx"
   #gopro2json -i "$BINFILE" -o "$OUTFILE.json"
done

#!/bin/bash

FILES="GX*.MP4"

STREAM="0:3"

for MP4FILE in $FILES; do

OUTFILE=$(basename "$MP4FILE" | cut -d. -f1)

OUTFILE=$(echo "$OUTFILE" | tr '[:upper:]' '[:lower:]')

BINFILE="$OUTFILE.bin"

ffmpeg -y -i "$MP4FILE" -codec copy -map $STREAM -f rawvideo "$BINFILE"

#gpmd2csv -i "$BINFILE" -o "$OUTFILE.csv"

gopro2gpx -i "$BINFILE" -a 500 -f 2 -o "$OUTFILE.gpx"

#gopro2gpx -i "$BINFILE" -a 200 -f 3 -o "$OUTFILE-virb.gpx"

#gopro2json -i "$BINFILE" -o "$OUTFILE.json"

done

Merging GPX files

As the GoPro splits recordings into 4GB blocks, when extracting you will get a single GPX file for each recording. Many pages found by Google say that to create a single track from these all you need to do is append the files into one big file. This is wrong, what you end up with is a single file with many short tracks, when what you are after is one long track covering the entire journey. This bash script uses gpsbabel to create single merged file from the extracted GPX data, it creates a file called “gpsoutput.gpx”.

#!/bin/bash
FILES="gx*.gpx"
ff=""
for f in $FILES; do
    ff="$ff -f $f"
done
gpsbabel -t -i gpx $ff -o gpx -x track,merge,title="GOPRO COMBINED LOG"  -F gpsoutpt.gpx

#!/bin/bash

FILES="gx*.gpx"

ff=""

for f in $FILES; do

ff="$ff -f $f"

done

gpsbabel -t -i gpx $ff -o gpx -x track,merge,title="GOPRO COMBINED LOG" -F gpsoutpt.gpx

The next stage will be to write a script that combines all these and completes the job in one easy process.

Links and Sources

Web Page to extract GoPro data: https://tailorandwayne.com/gopro-telemetry-extractor/ Easy to use for individual files
https://github.com/JuanIrache/gopro-utils
GoPro Forum Extracting the metadata in a useful format
GPSBabel, Free software for GPS data conversion and transfer: https://www.gpsbabel.org/

Computer, linux

FFMPEG for Video Conversion

13th November 2018 Karl

FFmpeg is a command line program to manipulate, convert, record and stream video and audio, it is available for Mac, Linux and Windows. Here is a handy list of commands for reference, these have been tested with version 3.1.12 in a Debian Linux environment. I expect this list to grow over time as needs arise.

• Rescale a 4K video to 1080p


ffmpeg -i inputFile.mp4 -c:v libx264 -crf 20 -preset slow -vf scale=1920:1080 outFile.mp4

ffmpeg -i inputFile.mp4 -c:v libx264 -crf 20 -preset slow -vf scale=1920:1080 outFile.mp4

• Convert to H.264 (AVC) codec for use on uploading to YouTube, Vimeo, etc:


ffmpeg -i inputFile.mov -f mp4 -vcodec libx264 outputFile.mp4

ffmpeg -i inputFile.mov -f mp4 -vcodec libx264 outputFile.mp4

Using this codec reduces the time it takes for the video to be available after upload, however YouTube converts the file again to the VP9 codec and unless you have a popular channel, 100 subscribers or more, then this can take a few days or weeks and in the meantime your video can appear quite poor and blocky even when watching at 1080p, especially when there is a lot of movement like in a car dash-cam video. You can use FFmpeg to encode to VP9 webm format with this bash script:

#!/bin/bash
INFILE="$1"

PASS1="-pass 1 -b:v 1000K -threads 8 -speed 4 -tile-columns 6 -frame-parallel 1"
PASS2="-pass 2 -b:v 1000K -threads 8 -speed 1 -tile-columns 6 -frame-parallel 1 -auto-alt-ref 1"

FILENAME=$(basename "$INFILE" | cut -d. -f1)
if [[ -z "${FILENAME// }" ]]; then
   echo "filename missing"
   exit 1
fi

OUTFILE="$FILENAME".webm

ffmpeg -y -i "$INFILE" -c:v libvpx-vp9 $PASS1 -g 9999 -aq-mode 0 -an -f webm /dev/null
ffmpeg -y -i "$INFILE" -c:v libvpx-vp9 $PASS2 -lag-in-frames 25 -g 9999 -aq-mode 0 -c:a libopus -b:a 64k -f webm "$OUTFILE"

#!/bin/bash

INFILE="$1"

PASS1="-pass 1 -b:v 1000K -threads 8 -speed 4 -tile-columns 6 -frame-parallel 1"

PASS2="-pass 2 -b:v 1000K -threads 8 -speed 1 -tile-columns 6 -frame-parallel 1 -auto-alt-ref 1"

FILENAME=$(basename "$INFILE" | cut -d. -f1)

if [[ -z "${FILENAME// }" ]]; then

echo "filename missing"

exit 1

OUTFILE="$FILENAME".webm

ffmpeg -y -i "$INFILE" -c:v libvpx-vp9 $PASS1 -g 9999 -aq-mode 0 -an -f webm /dev/null

ffmpeg -y -i "$INFILE" -c:v libvpx-vp9 $PASS2 -lag-in-frames 25 -g 9999 -aq-mode 0 -c:a libopus -b:a 64k -f webm "$OUTFILE"

This script is based on the encoding method shown in the WebM Wiki on my computer it is very slow and takes a quite a few hours to encode just nine minutes of video and the eventual results are so poor you’ll be wondering why you bothered.

• Convert to MP4 for use in Vegas Studio:


ffmpeg -i inputFile.mkv -codec copy outputFile.mp4

ffmpeg -i inputFile.mkv -codec copy outputFile.mp4

If you have a particularly old/odd video and get lots of pts has no value errors, then try this:


ffmpeg -fflags +genpts -i inputFile.mpg -codec copy outputFile.mp4

ffmpeg -fflags +genpts -i inputFile.mpg -codec copy outputFile.mp4

The -fflags +genpts option adds a Presentation Timestamp (PTS) to the frames, this must be before the -i as shown to work. Source.

• Set the video playback speed, this method adjusts the Presentation Timestamp (PTS) on each frame which may not work with older software. To slow down video divide the PTS by your required speed, this example slows the action by two times setpts=PTS/2.0. You can also reduce the number of dropped frames by increasing the frame-rate -r 50, in this case I went from 25fps to 50fps, but depending in the chosen speed frames may still be dropped.


ffmpeg -i inputFile.mp4 -r 50 -filter:v "setpts=PTS/2.0" outputfile.mp4

ffmpeg -i inputFile.mp4 -r 50 -filter:v "setpts=PTS/2.0" outputfile.mp4

Speed up your video by multiplying the PTS, in this case two times faster: setpts=PTS*2.0


ffmpeg -i inputFile.mp4 -filter:v "setpts=PTS*2.0" outputfile.mp4

ffmpeg -i inputFile.mp4 -filter:v "setpts=PTS*2.0" outputfile.mp4

• Convert file or extract audio from file into an MP3, the output is set to 128K constant bitrate


ffmpeg -i inputFile.mp4 -vn -acodec libmp3lame -b:a 128k outputFile.mp3

ffmpeg -i inputFile.mp4 -vn -acodec libmp3lame -b:a 128k outputFile.mp3

• Concatenate Video Files
This combines two video files, when using formats such as MP4 or MKV you will need to create intermediate files, otherwise only the first file will be included in the output:


ffmpeg -i file1.mp4 -c copy -bsf:v h264_mp4toannexb -f mpegts file1_inter.ts
ffmpeg -i file2.mp4 -c copy -bsf:v h264_mp4toannexb -f mpegts file2_inter.ts
ffmpeg -i "concat:file1_inter.ts|file2_inter.ts" -f mp4 -vcodec libx265 outputFile.mp4

ffmpeg -i file1.mp4 -c copy -bsf:v h264_mp4toannexb -f mpegts file1_inter.ts

ffmpeg -i file2.mp4 -c copy -bsf:v h264_mp4toannexb -f mpegts file2_inter.ts

ffmpeg -i "concat:file1_inter.ts|file2_inter.ts" -f mp4 -vcodec libx265 outputFile.mp4

• The opus not found error
When converting a file and you see an error like Could not find tag for codec opus in stream #1… you will need to state the output format


ffmpeg -i inputFile.mkv -f mp4 -vcodec libx264 outputFile.mp4

ffmpeg -i inputFile.mkv -f mp4 -vcodec libx264 outputFile.mp4

Links and Sources

https://www.ffmpeg.org/
FFmpeg Wiki: https://trac.ffmpeg.org/wiki

APPLE, Computer, linux, Programming, Raspberry Pi

Extracting MP3 audio from video files

29th July 2018 Karl

Here is a small Bash script that converts any supported ffmpeg video format; such as .MKV, .MP4 or .MOV and extracts the audio to an .MP3 file, It will also split that MP3 file into chunks and put them in a convenient directory. You will need to install ffmpeg and mp3splt for your particular platform.

Example Usage:

./mkv2mp3 "big fat file.mkv"

1	./mkv2mp3 "big fat file.mkv"

This uses ffmpeg to convert “big fat file.mkv” to “big fat file.mp3” and then uses mp3splt to create a directory “big fat file” containing the files 01 – big fat file.mp3, 02 – big fat file.mp3, etc. The MP3 files will be encoded at 128k Constant Bit Rate and each file will be around 50 minutes in length. To install in Debian/Ubuntu use: sudo apt-get install ffmpeg mp3splt

#!/bin/bash
INFILE=$1
BITRATE="128k"
LENGTH="50.0"

FILENAME=$(basename "$INFILE" | cut -d. -f1)
if [[ -z "${FILENAME// }" ]]; then
   echo "filename missing"
   exit 1
fi

echo "$FILENAME"
ffmpeg -i "$INFILE" -vn -acodec libmp3lame -b:a "$BITRATE" "$FILENAME".mp3
mp3splt -t "$LENGTH" -d "$FILENAME" -a -o @n+-+@f "$FILENAME".mp3

#!/bin/bash

INFILE=$1

BITRATE="128k"

LENGTH="50.0"

FILENAME=$(basename "$INFILE" | cut -d. -f1)

if [[ -z "${FILENAME// }" ]]; then

echo "filename missing"

exit 1

echo "$FILENAME"

ffmpeg -i "$INFILE" -vn -acodec libmp3lame -b:a "$BITRATE" "$FILENAME".mp3

mp3splt -t "$LENGTH" -d "$FILENAME" -a -o @n+-+@f "$FILENAME".mp3

mp3splt can find the audio in a quiet region near where the split is desired rather than midway through a word, this should make for much cleaner playback across tracks.

Alternative Method

This script gives the same results but uses ffmpeg to split the large MP3 file and then adds track numbering metadata using id3v2. To install in Debian/Ubuntu use: sudo apt-get install ffmpeg id3v2

#!/bin/bash
INFILE="$1"
BITRATE="128k"
LENGTH="3000"    # 50 minutes in seconds

FILENAME=$(basename "$INFILE" | cut -d. -f1)
if [[ -z "${FILENAME// }" ]]; then
   echo "filename missing"
   exit 1
fi

echo "$FILENAME"
MP3FILE="$FILENAME".mp3
ffmpeg -y -i "$INFILE" -vn -acodec libmp3lame -b:a "$BITRATE" "$MP3FILE"

OUTDIR="$FILENAME"
mkdir "$OUTDIR"

ffmpeg -y -i "$MP3FILE" -vn -f segment -segment_time "$LENGTH" -c copy "$OUTDIR"/%02d-"$MP3FILE"

TRACK=1
IFS=$'\n'
FILES=($(find "$OUTDIR" -name "*.mp3" | sort -n))
for file in "${FILES[@]}"; do
  TN=$(printf %02d $TRACK)
  id3v2 --track "$TN" --TIT2 "$FILENAME" "$file"
  TRACK=$((TRACK+1))
done

#!/bin/bash

INFILE="$1"

BITRATE="128k"

LENGTH="3000" # 50 minutes in seconds

FILENAME=$(basename "$INFILE" | cut -d. -f1)

if [[ -z "${FILENAME// }" ]]; then

echo "filename missing"

exit 1

echo "$FILENAME"

MP3FILE="$FILENAME".mp3

ffmpeg -y -i "$INFILE" -vn -acodec libmp3lame -b:a "$BITRATE" "$MP3FILE"

OUTDIR="$FILENAME"

mkdir "$OUTDIR"

ffmpeg -y -i "$MP3FILE" -vn -f segment -segment_time "$LENGTH" -c copy "$OUTDIR"/%02d-"$MP3FILE"

TRACK=1

IFS=$'\n'

FILES=($(find "$OUTDIR" -name "*.mp3" | sort -n))

for file in "${FILES[@]}"; do

TN=$(printf %02d $TRACK)

id3v2 --track "$TN" --TIT2 "$FILENAME" "$file"

TRACK=$((TRACK+1))

done

Creating an Audiobook

Taking this further, I was thinking that it would be nice to have these converted into the M4B Audiobook format for use on my elderly iPod. The script below assumes that you have processed the files as above and have added metadata tags using a tool like mp3tag (yes I know this is for Windows).

To complete this we need to: Combine the multiple MP3 files into one big file, or read the original big file then convert that to M4B format at 96K bit and add chapter marks every ten minutes. For this I have used ffmpeg v3.2.12 and libmp4v2 (for the mp4chaps utility), to install in Debian/Ubuntu use: sudo apt-get install libmp4v2-dev mp4v2-utils ffmpeg

This script works best from a single MP3 file rather than from those that have been re-combined back into a single file, recombining the files caused ffmpeg to exclaim “invalid packet size” and “invalid data” errors. It is able to tell the difference between a directory and a single MP3 and processes the file accordingly, don’t forget to add metadata tags and cover art before you run the script.

#!/bin/bash
INFILE="$1"

BITRATE="96k"
MP3TITLE=""
MP3ARTIST=""
MP3ALBUM=""
MP3COMMENT=""
MP3YEAR=""
MP3PERFORMER=""

####
####  FUNCTIONS
####

function setMP3variables() {
  TAG="$1"
  VALUE="$2"

  case "$TAG" in
    title)
      MP3TITLE="$VALUE"
      ;;
    artist)
      MP3ARTIST="$VALUE"
      ;;
    comment)
      MP3COMMENT="$VALUE"
      ;;
    album_artist)
      MP3PERFORMER="$VALUE"
      ;;
    album)
      MP3ALBUM="$VALUE"
      ;;
    date)
      MP3YEAR="$VALUE"
      ;;
   esac
}

## get the tag text from the metadata
## if the line contains an equals (=) then split by the first equals.
function getMP3tags() {
  MP3METADATA="$1"
  while read -r line
  do
    case "$line" in
      *=*)
        TAG=${line%%"="*}
        VALUE=${line#*"="}
        setMP3variables "$TAG" "$VALUE"
        ;;
    esac
  done < "$MP3METADATA"

  if [[ ! -z "${MP3TITLE// }" ]]; then
    OUTFILE="$MP3TITLE".mp3
    M4BFILE="$MP3TITLE".m4b
  fi
}

####
####   BEGINS
####

FILENAME=$(basename "$INFILE" | cut -d. -f1)
if [[ -z "${FILENAME// }" ]]; then
   echo "filename missing"
   exit 1
fi
OLDIFS=$IFS
OUTFILE="$FILENAME"_out.mp3
M4BFILE="${OUTFILE:0:-4}".m4b
METADATA="${OUTFILE:0:-4}".metadata
JPGFILE="${OUTFILE:0:-4}".jpg


## check if input is a diretory
if [ -d "$INFILE" ]; then
  # get the name of the first MP3 file in the directory
  FIRSTFILE=$(find "$INFILE" -name "*.mp3" | sort -n | head -1)
  FIRSTFILE=$(basename "$FIRSTFILE")

  if [[ -z "${FIRSTFILE// }" ]]; then
    echo "no mp3 files in directory"
    exit 1
  fi

  ## save the MP3 tag metadata from the first file
  ffmpeg -y -i "$INFILE/$FIRSTFILE" -f ffmetadata "$METADATA"
  ## save out the cver art image
  ffmpeg -y -i "$INFILE/$FIRSTFILE" -an -vcodec copy "$JPGFILE"
  ## process the tag metadata - put it into the $MP3* strings, set $OUTFILE and $LOGFILE from the title
  getMP3tags "$METADATA"

  ## make a bar seperated list of the MP3 files that are to be combined
  MP3FILES=""
  IFS=$'\n'
  FILES=($(find "$INFILE" -name "*.mp3" | sort -n))
  for files in "${FILES[@]}"; do
    MP3FILES+="$files|"
  done
  MP3FILES=${MP3FILES:0:-1}

  ## combine all the MP3 files into one big file
  ## $OUTFILE is set using either the $FILENAME or $MP3TITLE if that has been used
  ffmpeg -y -i "concat:$MP3FILES" -vn -vsync 2 -acodec copy "$OUTFILE"
else
  ## process a single MP3 file
  ## save the MP3 tag metadata from the first file
  ffmpeg -y -i "$INFILE" -f ffmetadata "$METADATA"
  ffmpeg -y -i "$INFILE" -an -vcodec copy "$JPGFILE"
  ## process the tag metadata - put it into the $MP3* strings, set $OUTFILE and $LOGFILE from the title
  getMP3tags "$METADATA"
  #ffmpeg -y -i "$INFILE" -vn -vsync 2 -acodec copy "$OUTFILE"
  OUTFILE="$INFILE"
fi

## if there is a cover image, these are are the options for adding it back in
WITHJPG=()
if [ -e "$JPGFILE" ]; then
   IFS=$OLDIFS
   WITHJPG=(-i "$JPGFILE" -metadata:s:v comment="Cover (front)")
fi

## create the audiobook M4B file
## and write the metadata into the output file
ffmpeg -y -i "$OUTFILE" -i "$METADATA" "${WITHJPG[@]}" -map_metadata 1 -id3v2_version 3 -acodec aac -b:a "$BITRATE" -strict -2 -f mp4 "$M4BFILE"
## add the chapters every 10 minutes
mp4chaps -e 600 "$M4BFILE"

## tidy up
rm "$MP3METADATA"
if [ -e "$JPGFILE" ]; then
   rm "$JPGFILE"
fi
#rm "$OUTFILE"
echo "finished"

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

#!/bin/bash

INFILE="$1"

BITRATE="96k"

MP3TITLE=""

MP3ARTIST=""

MP3ALBUM=""

MP3COMMENT=""

MP3YEAR=""

MP3PERFORMER=""

####

#### FUNCTIONS

####

function setMP3variables() {

TAG="$1"

VALUE="$2"

case "$TAG" in

title)

MP3TITLE="$VALUE"

;;

artist)

MP3ARTIST="$VALUE"

;;

comment)

MP3COMMENT="$VALUE"

;;

album_artist)

MP3PERFORMER="$VALUE"

;;

album)

MP3ALBUM="$VALUE"

;;

date)

MP3YEAR="$VALUE"

;;

esac

}

## get the tag text from the metadata

## if the line contains an equals (=) then split by the first equals.

function getMP3tags() {

MP3METADATA="$1"

while read -r line

case "$line" in

*=*)

TAG=${line%%"="*}

VALUE=${line#*"="}

setMP3variables "$TAG" "$VALUE"

;;

esac

done < "$MP3METADATA"

if [[ ! -z "${MP3TITLE// }" ]]; then

OUTFILE="$MP3TITLE".mp3

M4BFILE="$MP3TITLE".m4b

}

####

#### BEGINS

####

FILENAME=$(basename "$INFILE" | cut -d. -f1)

if [[ -z "${FILENAME// }" ]]; then

echo "filename missing"

exit 1

OLDIFS=$IFS

OUTFILE="$FILENAME"_out.mp3

M4BFILE="${OUTFILE:0:-4}".m4b

METADATA="${OUTFILE:0:-4}".metadata

JPGFILE="${OUTFILE:0:-4}".jpg

## check if input is a diretory

if [ -d "$INFILE" ]; then

# get the name of the first MP3 file in the directory

FIRSTFILE=$(find "$INFILE" -name "*.mp3" | sort -n | head -1)

FIRSTFILE=$(basename "$FIRSTFILE")

if [[ -z "${FIRSTFILE// }" ]]; then

echo "no mp3 files in directory"

exit 1

## save the MP3 tag metadata from the first file

ffmpeg -y -i "$INFILE/$FIRSTFILE" -f ffmetadata "$METADATA"

## save out the cver art image

ffmpeg -y -i "$INFILE/$FIRSTFILE" -an -vcodec copy "$JPGFILE"

## process the tag metadata - put it into the $MP3* strings, set $OUTFILE and $LOGFILE from the title

getMP3tags "$METADATA"

## make a bar seperated list of the MP3 files that are to be combined

MP3FILES=""

IFS=$'\n'

FILES=($(find "$INFILE" -name "*.mp3" | sort -n))

for files in "${FILES[@]}"; do

MP3FILES+="$files|"

done

MP3FILES=${MP3FILES:0:-1}

## combine all the MP3 files into one big file

## $OUTFILE is set using either the $FILENAME or $MP3TITLE if that has been used

ffmpeg -y -i "concat:$MP3FILES" -vn -vsync 2 -acodec copy "$OUTFILE"

else

## process a single MP3 file

## save the MP3 tag metadata from the first file

ffmpeg -y -i "$INFILE" -f ffmetadata "$METADATA"

ffmpeg -y -i "$INFILE" -an -vcodec copy "$JPGFILE"

## process the tag metadata - put it into the $MP3* strings, set $OUTFILE and $LOGFILE from the title

getMP3tags "$METADATA"

#ffmpeg -y -i "$INFILE" -vn -vsync 2 -acodec copy "$OUTFILE"

OUTFILE="$INFILE"

## if there is a cover image, these are are the options for adding it back in

WITHJPG=()

if [ -e "$JPGFILE" ]; then

IFS=$OLDIFS

WITHJPG=(-i "$JPGFILE" -metadata:s:v comment="Cover (front)")

## create the audiobook M4B file

## and write the metadata into the output file

ffmpeg -y -i "$OUTFILE" -i "$METADATA" "${WITHJPG[@]}" -map_metadata 1 -id3v2_version 3 -acodec aac -b:a "$BITRATE" -strict -2 -f mp4 "$M4BFILE"

## add the chapters every 10 minutes

mp4chaps -e 600 "$M4BFILE"

## tidy up

rm "$MP3METADATA"

if [ -e "$JPGFILE" ]; then

rm "$JPGFILE"

#rm "$OUTFILE"

echo "finished"

When encoding to the M4B using a re-combined file I saw a few of these errors from ffmpeg:

[mp3 @ 0x7fffc3bc9ea0] Header missing
Error while decoding stream #0:0: Invalid data found when processing input

1 2	[mp3 @ 0x7fffc3bc9ea0] Header missing Error while decoding stream #0:0: Invalid data found when processing input

These appear to be caused by the mp3splt program from when the original MP3 file was being split into 50 minute chunks, but I can’t hear any effect on the output.

Lots of information about the file can be gotten using mediainfo, to install in Debian/Ubuntu use: sudo apt-get install mediainfo, example use:

mediainfo --fullscan "big fat file.m4b"

1	mediainfo --fullscan "big fat file.m4b"

Links and References

ffmpeg, encode AAC: https://trac.ffmpeg.org/wiki/Encode/AAC
ArchLinux audiobook howto

Computer, Hardware, Interesting, linux, Raspberry Pi

Bluetooth Low Energy (BLE) on the Raspberry Pi

14th May 2017 Karl 4 Comments

Bluetooth Low Energy – BLE – Bluetooth 4.0 is an industry-standard wireless protocol built for the Internet of Things – IoT, it is designed to provide connectivity for devices operating from low capacity power sources such as coin cell batteries.

Raspberry Pi2 with ASUS USB-BT400 Bluetooth 4.0 Dongle

In this introduction to BLE I’ll be configuring a Raspberry Pi2 computer to talk to a smart watch. We will be installing the latest version of BlueZ from source, enabling BLE support. This is not a tutorial on decoding the data from the watch I am just using it as an example, although I may write about decoding it in a future posting.

I am using a ASUS USB-BT400 Bluetooth 4.0 Dongle on a Raspberry Pi2 but this will work on any computer with a Debian based distribution. Your dongle must be BLE/Bluetooth 4.0 capable otherwise this won’t work. I am using an ID107HR activity tracker with pedometer and heart rate monitor, randomly chosen from the list of cheap ones available on Amazon. While using the Pi to talk to the the watch make sure Bluetooth on the phone is off as it can only connect to one device at a time.

The current distribution of Raspbian – jessie on the Raspberry Pi comes with version 5.23 of the BlueZ Bluetooth stack that’s rather old, dating from September 2014 which lacks many of the features we will be needing. The current version 5.44 of the BlueZ has many changes in the package with many familiar components such as hcitool and gatttool being depreciated, so I will be ignoring those and using the available commands, bluetoothctl, on the terminal.

Installing BlueZ

With Raspbian – jessie installed we will need to update the Pi make sure some packages are installed and then installing the latest version of BlueZ. But first, remove the installed version 5.23 of BlueZ:

$ sudo apt-get --purge remove bluez
$ sudo apt-get autoremove

1 2	$ sudo apt-get --purge remove bluez $ sudo apt-get autoremove

Next, perform the traditional housekeeping updates then install the build tools and USB libraries. Those parts that are installed already will be automatically skipped.

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install build-essential
$ sudo apt-get install -y libusb-dev libdbus-1-dev libglib2.0-dev libudev-dev libical-dev libreadline-dev

$ sudo apt-get update

$ sudo apt-get upgrade

$ sudo apt-get install build-essential

$ sudo apt-get install -y libusb-dev libdbus-1-dev libglib2.0-dev libudev-dev libical-dev libreadline-dev

Now, download the source code, at time of writing the current stable release is version 5.44, check the BlueZ site for the latest version.

$ cd ~
$ wget http://www.kernel.org/pub/linux/bluetooth/bluez-5.44.tar.xz
$ tar xf bluez-5.44.tar.xz
$ cd bluez-5.44/

$ cd ~

$ wget http://www.kernel.org/pub/linux/bluetooth/bluez-5.44.tar.xz

$ tar xf bluez-5.44.tar.xz

$ cd bluez-5.44/

Inside the BlueZ directory, configure, make (this takes a while), and install. The experimental option adds BLE support and enabling the library allows for python use later on:

$ ./configure --enable-experimental --enable-library
$ make
$ sudo make install

$ ./configure --enable-experimental --enable-library

$ make

$ sudo make install

Configuring and Starting BlueZ

At this stage we will need to check that the installation worked and that we can see your bluetooth dongle. With your bluetooth dongle in a USB port you should see it on your list of USB devices, here you see mine as device ID: 0b05:17cb ASUSTek Computer, Inc.:

$ lsusb
Bus 001 Device 006: ID 0b05:17cb ASUSTek Computer, Inc. 
Bus 001 Device 004: ID 046d:c52e Logitech, Inc. 
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

$ lsusb

Bus 001 Device 006: ID 0b05:17cb ASUSTek Computer, Inc.

Bus 001 Device 004: ID 046d:c52e Logitech, Inc.

Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter

Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp.

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

You will also need to enable the experimental services, edit the file /lib/systemd/system/bluetooth.service and in the [Service] section change the ExecStart line to end with –experimental:

ExecStart=/usr/local/libexec/bluetooth/bluetoothd --experimental

1	ExecStart=/usr/local/libexec/bluetooth/bluetoothd --experimental

Start the bluetooth service, and while we are at it enable it to load on boot:

$ sudo systemctl start bluetooth
$ sudo systemctl enable bluetooth

1 2	$ sudo systemctl start bluetooth $ sudo systemctl enable bluetooth

Once started, check the status of the bluetooth daemon with:

$ sudo systemctl status bluetooth
● bluetooth.service - Bluetooth service
   Loaded: loaded (/lib/systemd/system/bluetooth.service; enabled)
   Active: active (running) since Sat 2017-05-06 19:54:28 BST; 54s ago
     Docs: man:bluetoothd(8)
 Main PID: 10407 (bluetoothd)
   Status: "Running"
   CGroup: /system.slice/bluetooth.service
           └─10407 /usr/local/libexec/bluetooth/bluetoothd --experimental

May 06 19:54:27 raspberrypi bluetoothd[10407]: Bluetooth daemon 5.44
May 06 19:54:28 raspberrypi systemd[1]: Started Bluetooth service.
May 06 19:54:28 raspberrypi bluetoothd[10407]: Starting SDP server
May 06 19:54:28 raspberrypi bluetoothd[10407]: Bluetooth management interface 1.14 initialized

$ sudo systemctl status bluetooth

● bluetooth.service - Bluetooth service

Loaded: loaded (/lib/systemd/system/bluetooth.service; enabled)

Active: active (running) since Sat 2017-05-06 19:54:28 BST; 54s ago

Docs: man:bluetoothd(8)

Main PID: 10407 (bluetoothd)

Status: "Running"

CGroup: /system.slice/bluetooth.service

└─10407 /usr/local/libexec/bluetooth/bluetoothd --experimental

May 06 19:54:27 raspberrypi bluetoothd[10407]: Bluetooth daemon 5.44

May 06 19:54:28 raspberrypi systemd[1]: Started Bluetooth service.

May 06 19:54:28 raspberrypi bluetoothd[10407]: Starting SDP server

May 06 19:54:28 raspberrypi bluetoothd[10407]: Bluetooth management interface 1.14 initialized

Should you need to, the service can be stopped and prevented from loading on boot with:

$ sudo systemctl stop bluetooth
$ sudo systemctl disable bluetooth

1 2	$ sudo systemctl stop bluetooth $ sudo systemctl disable bluetooth

Finally, you may want to enable auto-power on for the device, to do so create this bluetooth config file:

$ sudo mkdir /etc/bluetooth
$ sudo nano /etc/bluetooth/main.conf

1 2	$ sudo mkdir /etc/bluetooth $ sudo nano /etc/bluetooth/main.conf

and add these two lines:

[Policy]
AutoEnable=true

1 2	[Policy] AutoEnable=true

You should restart the Pi at this point and check that the daemon has loaded properly with sudo systemctl status bluetooth

Testing BlueZ

Start the bluetooth controller, you should see your dongles MAC address and alias:

$ sudo bluetoothctl 
[NEW] Controller 5C:F3:70:80:8A:A6 raspberrypi [default]

1 2	$ sudo bluetoothctl [NEW] Controller 5C:F3:70:80:8A:A6 raspberrypi [default]

for first time use, try scanning to find your watch, if it doesn’t appear it is out of range, its battery is flat, or your dongle does not support BLE, here you can see it as ID107 HR:

[bluetooth]# power on
Changing power on succeeded
[CHG] Controller 5C:F3:70:80:8E:FB Powered: yes
[bluetooth]# scan on
Discovery started
[CHG] Controller 5C:F3:70:80:8E:FB Discovering: yes
[NEW] Device C5:E8:FB:BF:F2:6C ID107 HR
[CHG] Device C5:E8:FB:BF:F2:6C RSSI: -63
[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Key: 0x001c
[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Value: 0x02
[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Value: 0x00
[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Value: 0xc5
[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Value: 0xe8
[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Value: 0xfb
[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Value: 0xbf
[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Value: 0xf2
[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Value: 0x6c
[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Value: 0xe3
[CHG] Device C5:E8:FB:BF:F2:6C AdvertisingFlags: 0x00
...etc...
[bluetooth]# scan off

[bluetooth]# power on

Changing power on succeeded

[CHG] Controller 5C:F3:70:80:8E:FB Powered: yes

[bluetooth]# scan on

Discovery started

[CHG] Controller 5C:F3:70:80:8E:FB Discovering: yes

[NEW] Device C5:E8:FB:BF:F2:6C ID107 HR

[CHG] Device C5:E8:FB:BF:F2:6C RSSI: -63

[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Key: 0x001c

[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Value: 0x02

[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Value: 0x00

[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Value: 0xc5

[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Value: 0xe8

[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Value: 0xfb

[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Value: 0xbf

[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Value: 0xf2

[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Value: 0x6c

[CHG] Device C5:E8:FB:BF:F2:6C ManufacturerData Value: 0xe3

[CHG] Device C5:E8:FB:BF:F2:6C AdvertisingFlags: 0x00

...etc...

[bluetooth]# scan off

bluetoothctl remembers your devices, so when you next use the program the watch appears on the list at the start. The controller has a number of options, these can be seen with help command. You can use show to view the status of your dongle:

[bluetooth]# show
Controller 5C:F3:70:80:8A:A6
	Name: raspberrypi
	Alias: raspberrypi
	Class: 0x000000
	Powered: no
	Discoverable: no
	Pairable: yes
	UUID: Generic Attribute Profile (00001801-0000-1000-8000-00805f9b34fb)
	UUID: A/V Remote Control        (0000110e-0000-1000-8000-00805f9b34fb)
	UUID: PnP Information           (00001200-0000-1000-8000-00805f9b34fb)
	UUID: Generic Access Profile    (00001800-0000-1000-8000-00805f9b34fb)
	UUID: A/V Remote Control Target (0000110c-0000-1000-8000-00805f9b34fb)
	Modalias: usb:v1D6Bp0246d052C
	Discovering: no

[bluetooth]# show

Controller 5C:F3:70:80:8A:A6

Name: raspberrypi

Alias: raspberrypi

Class: 0x000000

Powered: no

Discoverable: no

Pairable: yes

UUID: Generic Attribute Profile (00001801-0000-1000-8000-00805f9b34fb)

UUID: A/V Remote Control (0000110e-0000-1000-8000-00805f9b34fb)

UUID: PnP Information (00001200-0000-1000-8000-00805f9b34fb)

UUID: Generic Access Profile (00001800-0000-1000-8000-00805f9b34fb)

UUID: A/V Remote Control Target (0000110c-0000-1000-8000-00805f9b34fb)

Modalias: usb:v1D6Bp0246d052C

Discovering: no

The list of UUID’s show the services supported by the Dongle. Now we can power the dongle on, set the agent – this manages the connection, and then connect to the watch on which the bluetooth symbol will appear. Once connected there will be a pause then you will see a list of attributes supported by the watch, it is advertising the services available:

[bluetooth]# power on
Changing power on succeeded
[CHG] Controller 5C:F3:70:80:8A:A6 Powered: yes
[bluetooth]# agent on
Agent registered
[bluetooth]# default-agent
Default agent request successful
[bluetooth]# connect C5:E8:FB:BF:F2:6C
Attempting to connect to C5:E8:FB:BF:F2:6C
[CHG] Device C5:E8:FB:BF:F2:6C Connected: yes
Connection successful
[NEW] Primary Service
	/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service0008
	00001801-0000-1000-8000-00805f9b34fb
	Generic Attribute Profile
[NEW] Characteristic
	/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service0008/char0009
	00002a05-0000-1000-8000-00805f9b34fb
	Service Changed
[NEW] Descriptor
	/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service0008/char0009/desc000b
	00002902-0000-1000-8000-00805f9b34fb
	Client Characteristic Configuration
[NEW] Primary Service
	/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c
	00000af0-0000-1000-8000-00805f9b34fb
	Unknown
[NEW] Characteristic
	/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000d
	00000af6-0000-1000-8000-00805f9b34fb
	Unknown
[NEW] Characteristic
	/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f
	00000af7-0000-1000-8000-00805f9b34fb
	Unknown
[NEW] Descriptor
	/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f/desc0011
	00002902-0000-1000-8000-00805f9b34fb
	Client Characteristic Configuration
[NEW] Characteristic
	/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char0012
	00000af2-0000-1000-8000-00805f9b34fb
	Unknown
[NEW] Descriptor
	/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char0012/desc0014
	00002902-0000-1000-8000-00805f9b34fb
	Client Characteristic Configuration
[NEW] Characteristic
	/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char0015
	00000af1-0000-1000-8000-00805f9b34fb
	Unknown
[CHG] Device C5:E8:FB:BF:F2:6C UUIDs: 00000af0-0000-1000-8000-00805f9b34fb
[CHG] Device C5:E8:FB:BF:F2:6C UUIDs: 00001800-0000-1000-8000-00805f9b34fb
[CHG] Device C5:E8:FB:BF:F2:6C UUIDs: 00001801-0000-1000-8000-00805f9b34fb
[CHG] Device C5:E8:FB:BF:F2:6C ServicesResolved: yes
[ID107 HR]#

[bluetooth]# power on

Changing power on succeeded

[CHG] Controller 5C:F3:70:80:8A:A6 Powered: yes

[bluetooth]# agent on

Agent registered

[bluetooth]# default-agent

Default agent request successful

[bluetooth]# connect C5:E8:FB:BF:F2:6C

Attempting to connect to C5:E8:FB:BF:F2:6C

[CHG] Device C5:E8:FB:BF:F2:6C Connected: yes

Connection successful

[NEW] Primary Service

/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service0008

00001801-0000-1000-8000-00805f9b34fb

Generic Attribute Profile

[NEW] Characteristic

/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service0008/char0009

00002a05-0000-1000-8000-00805f9b34fb

Service Changed

[NEW] Descriptor

/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service0008/char0009/desc000b

00002902-0000-1000-8000-00805f9b34fb

Client Characteristic Configuration

[NEW] Primary Service

/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c

00000af0-0000-1000-8000-00805f9b34fb

Unknown

[NEW] Characteristic

/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000d

00000af6-0000-1000-8000-00805f9b34fb

Unknown

[NEW] Characteristic

/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f

00000af7-0000-1000-8000-00805f9b34fb

Unknown

[NEW] Descriptor

/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f/desc0011

00002902-0000-1000-8000-00805f9b34fb

Client Characteristic Configuration

[NEW] Characteristic

/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char0012

00000af2-0000-1000-8000-00805f9b34fb

Unknown

[NEW] Descriptor

/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char0012/desc0014

00002902-0000-1000-8000-00805f9b34fb

Client Characteristic Configuration

[NEW] Characteristic

/org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char0015

00000af1-0000-1000-8000-00805f9b34fb

Unknown

[CHG] Device C5:E8:FB:BF:F2:6C UUIDs: 00000af0-0000-1000-8000-00805f9b34fb

[CHG] Device C5:E8:FB:BF:F2:6C UUIDs: 00001800-0000-1000-8000-00805f9b34fb

[CHG] Device C5:E8:FB:BF:F2:6C UUIDs: 00001801-0000-1000-8000-00805f9b34fb

[CHG] Device C5:E8:FB:BF:F2:6C ServicesResolved: yes

[ID107 HR]#

and now that we have connected we can ask for some info:

[ID107 HR]# info C5:E8:FB:BF:F2:6C
Device C5:E8:FB:BF:F2:6C
	Name: ID107 HR
	Alias: ID107 HR
	Appearance: 0x0341
	Paired: no
	Trusted: no
	Blocked: no
	Connected: yes
	LegacyPairing: no
	UUID: Unknown                   (00000af0-0000-1000-8000-00805f9b34fb)
	UUID: Generic Access Profile    (00001800-0000-1000-8000-00805f9b34fb)
	UUID: Generic Attribute Profile (00001801-0000-1000-8000-00805f9b34fb)
	ManufacturerData Key: 0x001c
	ManufacturerData Value: 0x02
	ManufacturerData Value: 0x00
	ManufacturerData Value: 0xc5
	ManufacturerData Value: 0xe8
	ManufacturerData Value: 0xfb
	ManufacturerData Value: 0xbf
	ManufacturerData Value: 0xf2
	ManufacturerData Value: 0x6c
	ManufacturerData Value: 0xe3
[ID107 HR]#

[ID107 HR]# info C5:E8:FB:BF:F2:6C

Device C5:E8:FB:BF:F2:6C

Name: ID107 HR

Alias: ID107 HR

Appearance: 0x0341

Paired: no

Trusted: no

Blocked: no

Connected: yes

LegacyPairing: no

UUID: Unknown (00000af0-0000-1000-8000-00805f9b34fb)

UUID: Generic Access Profile (00001800-0000-1000-8000-00805f9b34fb)

UUID: Generic Attribute Profile (00001801-0000-1000-8000-00805f9b34fb)

ManufacturerData Key: 0x001c

ManufacturerData Value: 0x02

ManufacturerData Value: 0x00

ManufacturerData Value: 0xc5

ManufacturerData Value: 0xe8

ManufacturerData Value: 0xfb

ManufacturerData Value: 0xbf

ManufacturerData Value: 0xf2

ManufacturerData Value: 0x6c

ManufacturerData Value: 0xe3

[ID107 HR]#

These UUID’s are used to describe the sevices available on the device, some are pre-defined and can be found in the a href=”https://www.bluetooth.com/specifications/gatt/characteristics” target=”_blank” rel=”noopener noreferrer”>GATT schema, others are vendor specific and unless they publicly release these, decoding can become rather difficult. There are four types of attribute:

Services – collections of characteristics and relationships to other services that encapsulate the behavior of part of a device
Characteristics – attribute types that contain a single logical value
Descriptors – defined attributes that describe a characteristic value
Declarations – defined GATT profile attribute types

Each attribute is identified by a 128 bit ID, for example, one of the characteristics from the list above: 00002902-0000-1000-8000-00805f9b34fb, the first eight bits are used as an unique identifier: 00002902 and are shown as UUID’s: 0x2902. Data is contained in services, each service has a number of characteristics that may contain further descriptions depending on the requirement of the characteristic. You can see how the data is mapped out in this chart:

A spreadsheet with the watch data reformatted and tastefully coloured to illustrates this. Observe the Service URL column, it looks a lot like a directory structure:

Here we see two services /service0008 and /service000c looking further into the second service: /service000c we see that it has four characteristics, and to of those have descriptors. We can interrogate the characteristics and descriptors to glean further information by selecting the attribute and reading, like so:

[ID107 HR]# select-attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f
[ID107 HR:/service000c/char000f]# read
Attempting to read /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x02
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0xa0
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x23
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x00
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x00
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x00
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x01
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x00
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x00
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x00
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x19
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x00
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x00
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x00
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x92
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x00
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x00
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x00
[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x00
  02 a0 23 00 00 00 01 00 00 00 19 00 00 00 92 00  ..#.............
  00 00 00                                         ...

[ID107 HR]# select-attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f

[ID107 HR:/service000c/char000f]# read

Attempting to read /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f

[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x02

[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0xa0

[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x23

[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x00

[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x01

[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x00

[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x19

[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x00

[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x92

[CHG] Attribute /org/bluez/hci0/dev_C5_E8_FB_BF_F2_6C/service000c/char000f Value: 0x00

02 a0 23 00 00 00 01 00 00 00 19 00 00 00 92 00 ..#.............

00 00 00 ...

Which is all very nice, but not particularly helpful as the manufacturer has chosen to use custom, proprietary, UUID’s for the watch. We don’t know the instructions to send to have the watch realease its data.

Those Scripting BlueZ

Inevitably, you’ll be wanting to automate connections. This becomes easy with the automation scripting language expect. Install, then make a script file:

$ sudo apt-get install expect
$ cd ~
$ nano bttest
$ chmod +x bttest

$ sudo apt-get install expect

$ cd ~

$ nano bttest

$ chmod +x bttest

In this example the script forgets the watch, finds the watch, connects to the watch, gets some info and then disconnects:

#!/usr/bin/expect -f

set timeout 10
set prompt ".*#"
set usrpasswd ""
set address "C5:E8:FB:BF:F2:6C"

## execute blutetoothctl
spawn sudo bluetoothctl 
expect -re ".*password.*" 
send "$usrpasswd\r"

## forget about the device - if connected previously
expect -re $prompt
send "remove $address\r"
expect -re $prompt
sleep 3

## switch on the dongle
send "power on\r"
expect "Changing power on succeeded"
expect -re $prompt

## scan for devices 
send "scan on\r"
expect -re $prompt
sleep 5
send "scan off\r"
expect -re $prompt

## set the agent
send "agent on\r"
expect "Agent registered"
send "default-agent\r"
expect -re $prompt
sleep 2

## connect to watch
send "connect $address\r"
expect -re $prompt

## get some info
send "info $address\r"
expect -re $prompt

## disconnect
send "disconnect $address\r"
sleep 2
expect -re "\[bluetooth\]#"

## bye
send "exit\r"

#!/usr/bin/expect -f

set timeout 10

set prompt ".*#"

set usrpasswd ""

set address "C5:E8:FB:BF:F2:6C"

## execute blutetoothctl

spawn sudo bluetoothctl

expect -re ".*password.*"

send "$usrpasswd\r"

## forget about the device - if connected previously

expect -re $prompt

send "remove $address\r"

expect -re $prompt

sleep 3

## switch on the dongle

send "power on\r"

expect "Changing power on succeeded"

expect -re $prompt

## scan for devices

send "scan on\r"

expect -re $prompt

sleep 5

send "scan off\r"

expect -re $prompt

## set the agent

send "agent on\r"

expect "Agent registered"

send "default-agent\r"

expect -re $prompt

sleep 2

## connect to watch

send "connect $address\r"

expect -re $prompt

## get some info

send "info $address\r"

expect -re $prompt

## disconnect

send "disconnect $address\r"

sleep 2

expect -re "\[bluetooth\]#"

## bye

send "exit\r"

in the script, send sends a command, don’t forget to add the carriage return – \r and expect is used to wait for a response within the timeout period, here it is set to 10 seconds. expect -re is using regex when looking for a reply, otherwise it uses a literal string. So much more can be done with expect and there are many tutorials, such as this one written by FluidBank.

More Bluetooth Data

For analysing bluetooth data a couple of very useful tools are available, Wireshark and Android data logging. I will go through the installation but not look at the data in any detail, this posting is getting a bit long. This Section is in two parts, installing Wireshark and Android Debug Bridge.

Sniffing with the Shark
Wireshark is a network and bluetooth packet sniffer, it shows you network and bluetooth traffic occurring on your Pi. Here is a quick installation method for a reasonably new version of Wireshark (v2.2.4) from the backports, answer yes to the question “Should non-superusers be able to capture packets?”:

$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 7638D0442B90D010
$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 8B48AD6246925553
$ echo 'deb http://httpredir.debian.org/debian jessie-backports main contrib non-free' | sudo tee -a /etc/apt/sources.list.d/jessie-backports.list
$ sudo apt-get update
$ sudo sudo apt-get -t jessie-backports install wireshark wireshark-dev libwireshark-dev

$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 7638D0442B90D010

$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 8B48AD6246925553

$ echo 'deb http://httpredir.debian.org/debian jessie-backports main contrib non-free' | sudo tee -a /etc/apt/sources.list.d/jessie-backports.list

$ sudo apt-get update

$ sudo sudo apt-get -t jessie-backports install wireshark wireshark-dev libwireshark-dev

and if you get a message about permissions, reconfigure the package and answer yes:

$ sudo dpkg-reconfigure wireshark-common

1	$ sudo dpkg-reconfigure wireshark-common

Start Wireshark and double click your bluetooth device on the list, in my case bluetooth0. There is not much to see as Wireshark will only see traffic between the watch and the Pi:

Android Debug Bridge – ADB
For Anroid 4.2.2 and above, activate developer mode on the phone, go to Settings, tap About Phone and at the bottom of the list tap Build Number three times. Back in the main settings page Developer Options has appeared, tap developer and turn USB debugging On. With the phone plugged into a USB port a little Android head should appear in the information bar at the top-left of the screen. To begin we will need to install some udev rules written by Nicolas Bernaerts:

$ sudo wget -O /etc/udev/rules.d/51-android.rules https://raw.githubusercontent.com/NicolasBernaerts/ubuntu-scripts/master/android/51-android.rules
$ sudo chmod a+r /etc/udev/rules.d/51-android.rules
$ sudo service udev restart

$ sudo wget -O /etc/udev/rules.d/51-android.rules https://raw.githubusercontent.com/NicolasBernaerts/ubuntu-scripts/master/android/51-android.rules

$ sudo chmod a+r /etc/udev/rules.d/51-android.rules

$ sudo service udev restart

Install the android tools, confirm that you have at least version 1.0.31, and start ADB

$ sudo apt-get install android-tools-adb android-tools-fastboot
$ adb version
Android Debug Bridge version 1.0.31
$ adb devices
* daemon not running. starting it now on port 5037 *
* daemon started successfully *
List of devices attached 
064be417008eef9f	offline

$ sudo apt-get install android-tools-adb android-tools-fastboot

$ adb version

Android Debug Bridge version 1.0.31

$ adb devices

* daemon not running. starting it now on port 5037 *

* daemon started successfully *

List of devices attached

064be417008eef9f offline

At this point on the phone an allow USB debugging dialog will appear, give permission and always trust to authorise it. ADB will now show the device as a device:

$ adb devices
List of devices attached 
064be417008eef9f	device

$ adb devices

List of devices attached

064be417008eef9f device

If the device list is empty, with everything plugged in good and proper and the phone setup in developer mode, start your diagnosis by checking udev; open another terminal window and view logging with udevadm monitor –environment and reload with sudo udevadm control –reload I’m not entirely sure what I did to get it from ‘not working’ to ‘working’. If all else fails elevate yourself to root.

Data Capture
With ADB now setup we can capture the Bluetooth data being exchanged. With bluetooth off, in the Developer Settings find Enable Bluetooth HCI snoop log and turn it On. In the smartwatch app synchronise with your watch, once complete turn Bluetooth off manually – this is to minimise the amount of captured data. Don’t forget to turn logging off on the phone when done. To find where the log file has been stored and copy the file from the phone to the Pi use:

$ adb shell "cat /etc/bluetooth/bt_stack.conf | grep FileName"
BtSnoopFileName=/sdcard/btsnoop_hci.log
$ adb pull /sdcard/btsnoop_hci.log ~/

$ adb shell "cat /etc/bluetooth/bt_stack.conf | grep FileName"

BtSnoopFileName=/sdcard/btsnoop_hci.log

$ adb pull /sdcard/btsnoop_hci.log ~/

We can now use Wireshark to read the log file…

Wireshark reading the Android Bluetooth Log

This wasn’t quite the posting I originally had in mind, I wanted to decode the data from the watch for my own use, making something more useful, impressive graphs and charts, than that provided by the Android App VeryFit 2.0 but as the manufacturer has chosen to use proprietary GATT codes it makes the job that much harder. It may be much simpler to just buy an expensive FitBit and download the data from them. But with writing this I now know a few things that were previously unknown, and I hope that this has provided some light to your BlueZ (a pun!, right at the end!).

Links and Sources

Bluetooth Low Energy Description
ID107HR activity tracker review
BlueZ – Official Linux Bluetooth protocol stack
List of GATT Services
Raspbian for the Raspberry Pi
Adafuit: Install bluez on the Raspberry Pi
ArchLinux Wiki: Bluetooth
Bluetooth audio paired but not connect? – Very useful BlueZ guide
Getting Started with Bluetooth Low Energy: Tools and Techniques for Low-Power Networking – ISBN: 1491949511

Computer, Database, linux

Upgrading the Python Oracle Client

23rd November 2015 Karl

This is a follow up to one of my previous postings: Python and the Oracle Client. The main databases here are being upgraded to Oracle 12 and I’ve taken the opportunity to update the client used by my Python scripts, also its good practice to install new clients when old versions go out of support.

Current Setup

The system I am upgrading here has the following configuration, but this should work with any RPM based distribution, such as CentOS and SUSE :

Red Hat Enterprise Linux Server release 6.6 (Santiago)
Python 2.6.6
python connector: cx_Oracle – 5.1.2
oracle-instantclient11.2-basic-11.2.0.4.0-1.x86_64
oracle-instantclient11.2-devel-11.2.0.4.0-1.x86_64

To find the versions of your currently installed software:
$ python Python 2.6.6 (r266:84292, Nov 21 2013, 10:50:32) [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import cx_Oracle >>> print cx_Oracle.version 5.1.2
and the Oracle client:
$ rpm -qa | grep oracle oracle-instantclient11.2-basic-11.2.0.4.0-1.x86_64 oracle-instantclient11.2-devel-11.2.0.4.0-1.x86_64

Preparing

If you have the old versions installed you will need to do some tidying up by removing the client and python connector, version 11 of the client despite being RPM packaged had some non-standard elements. Use rpm to delete the old version of instant client, remove devel first:
$ sudo su # rpm -ev oracle-instantclient11.2-devel-11.2.0.4.0-1.x86_64 # rpm -ev oracle-instantclient11.2-basic-11.2.0.4.0-1.x86_64
you may also need to remove the library reference from a previous installation:
# rm /etc/ld.so.conf.d/oracle.conf # ldconfig
to remove the Python oracle connector, there are two methods. Manually, by finding the previously installed package deleting the files and editing the package list:
# find / -name cx_Oracle.py -print /usr/lib/python2.6/site-packages/cx_Oracle-5.1.2-py2.6-linux-x86_64.egg/cx_Oracle.py # cd /usr/lib/python2.6/site-packages # rm -rf cx_Oracle-5.1.2-py2.6-linux-x86_64.egg
now edit the easy-install.pth file
# nano /usr/lib/python2.6/site-packages/easy-install.pth
and remove the line:
./cx_Oracle-5.1.2-py2.6-linux-x86_64.egg

Or do it the easy way, if you have pip installed:
# sudo pip uninstall cx_Oracle
easy_install does not have an uninstall option.

Installing

Download and install version 12 of the Instant Client and SDK (devel), these can be gotten from: http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html For Linux choose the correct flavour for your installed operating system: x86 or x86-64 for 64bit operating systems, you will need to register on the site gain access the files.
# rpm -i oracle-instantclient12.1-basic-12.1.0.2.0-1.x86_64.rpm # rpm -i oracle-instantclient12.1-devel-12.1.0.2.0-1.x86_64.rpm
Now to install the python connector:
# easy_install cx-Oracle
or, the recommended method:
# pip install cx-Oracle
Installation for the version 12 client is much more straight forward than that for version 11.

Testing

A quick test to ensure that the expected versions appear, and that you can connect to the database.
Python 2.6.6 (r266:84292, Nov 21 2013, 10:50:32) [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import cx_Oracle >>> cx_Oracle.version '5.2' >>> oraConn = "<USERNAME>/<PASSWORD>@<DATABASE HOST>:<DATABASE PORT>/<SERVICE>" >>> ocDB = cx_Oracle.connect(oraConn) >>> ocDB.version '12.1.0.2.0'

Sources

https://bitbucket.org/anthony_tuininga/cx_oracle

Computer, linux, Raspberry Pi

How-To: Raspberry Pi as a 3G/4G Router

16th August 2014 Karl 37 Comments

Update: 15 April 2016 – Added information about which IP address to use and assigning static IP addresses for printers and servers

Recently I have needed to find an emergency alternative to my broadband due to the regional wide area network, Digital Region, being shut down, and the ISP Origin making a mess of getting all their cutomers onto ASDL. To get quickly back onto the internet, I have bought an ZTE MF823 4G Mobile Broadband Dongle as supplied by the badly named ‘three’ mobile phone company. As I have my own internal wired network, with multiple computers and ‘things’ there is a need to have something more sophisticated than just plugging the dongle into a single PC.

Here is my recipe for setting up a Raspberry Pi as a router with an ZTE MF283 Dongle. In this setup all the computers are on a wired Ethernet connection using a switch for the network. The Pi has Raspbian Debian Wheezy installed (June 2014) with all the latest updates made. For testing, the dongle is plugged into the USB port via a powered hub, and the Pi connected to a switch with another PC running Linux Mint.

Which IP addresses to use?

In this How-To I am using the IP address range 192.168.2.xxx this is to avoid conflict with the cable router which uses the 192.168.1.xxx range (the DHCP server is switched off on the router). IPv4 addresses are split into three different ranges, the 192.168.xxx.xxx range – 192.168.0.0 to 192.168.255.255 gives a possible 65,536 addresses but for your home it is unlikely you’ll have more than 255 network devices, so we can simplify things by limiting the address range used to 192.168.2.xxx and avoid the troublesome world of subnet masking.

192.168.xxx.xxx is used as its been designated for use on private networks by the Internet Assigned Numbers Authority this is a well established convention and is best practice. Two other IPv4 address ranges are available for larger private networks: 172.16.0.0-172.31.255.255 and 10.0.0.0-10.255.255.255 with 1,048,576 and 16,777,216 available addresses respectively, the most suitable network class should be chosen for your network.

Setup the Dongle

This USB dongle has its own built in dialer so you do not need ppp or wvdial installed, it appears as a USB ethernet device on the Raspberry Pi. You will need a powered USB hub as the dongle can draw more power than the Pi can provide, the symptoms of too much of power being drawn will be the Pi behaving erratically or restarting unexpectedly.

With the dongle plugged in, check that it is recognised by the Pi with lsusb, it can be seen here as ‘ZTE WCDMA Technologies MSM’:

$ lsusb
Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp.
Bus 001 Device 004: ID 1a40:0201 Terminus Technology Inc. FE 2.1 7-port Hub
Bus 001 Device 005: ID 046d:c52e Logitech, Inc.
Bus 001 Device 006: ID 1a40:0101 Terminus Technology Inc. 4-Port HUB
Bus 001 Device 007: ID 19d2:1405 ZTE WCDMA Technologies MSM

$ lsusb

Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp.

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp.

Bus 001 Device 004: ID 1a40:0201 Terminus Technology Inc. FE 2.1 7-port Hub

Bus 001 Device 005: ID 046d:c52e Logitech, Inc.

Bus 001 Device 006: ID 1a40:0101 Terminus Technology Inc. 4-Port HUB

Bus 001 Device 007: ID 19d2:1405 ZTE WCDMA Technologies MSM

The device ID is 19d2. and 1405 is the mode, this should be 1405 – CDC ethernet. If it is not, try removing the micro-SD card and rebooting the Pi, the device modes available are:

1225 – Default mode. USB Mass Storage Device + CD-ROM + card reader.
1403 – Modem mode. RNDIS + Mass Storage Device.
1405 – CDC ethernet
0016 – Download mode

As the dongle also has a Mass Storage Device the Raspberry may not switch to CDC ethernet. If the mode does not change, try the following with usb-modeswitch:
$ sudo apt-get install usb-modeswitch $ sudo usb_modeswitch -v 0x19d2 -p 0x1405 -d
I did not have to change the mode as it was correct already, and it didn’t change when I tried setting it as a Mass Storage device, I have not explored this any further.

When first plugged in the dongle was recognised as a ethernet device but it did not obtain an IP address:

$ ifconfig
usb0     Link encap:Ethernet HWaddr 36:4b:53:b7:e3:6e
         UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

$ ifconfig

usb0 Link encap:Ethernet HWaddr 36:4b:53:b7:e3:6e

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

if this is the case with you, add the following two lines to the end of sudo nano /etc/network/interfaces:
auto usb0 iface usb0 inet dhcp
the dongle provides its own address to the computer. Reboot, and you should see the obtained address:

usb0    Link encap:Ethernet HWaddr 36:4b:53:b7:e3:6e
        inet addr:192.168.0.185 Bcast:192.168.0.255 Mask:255.255.255.0
        UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
        RX packets:40 errors:0 dropped:4 overruns:0 frame:0
        TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:1000
        RX bytes:4443 (4.3 KiB) TX bytes:1382 (1.3 KiB)

usb0 Link encap:Ethernet HWaddr 36:4b:53:b7:e3:6e

inet addr:192.168.0.185 Bcast:192.168.0.255 Mask:255.255.255.0

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:40 errors:0 dropped:4 overruns:0 frame:0

TX packets:15 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:4443 (4.3 KiB) TX bytes:1382 (1.3 KiB)

The address 192.168.0.185 is now the internet address of the computer the dongle always assigns this address, there is also a useful web status page on http://192.168.0.1

Configuring the network

first of all enable ip4 forwarding, edit the file sudo nano /etc/sysctl.conf and uncomment the line:
net.ipv4.ip_forward=1
this will enable forwarding on reboot, you can also enable IP forwarding immediately with:
$ sudo sysctl -w net.ipv4.ip_forward=1

We now need to give the Pi a static IP address on the internal network. Edit sudo nano /etc/network/interfaces so you end up with a file that looks like this:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address 192.168.2.1
netmask 255.255.255.0

auto usb0
iface usb0 inet dhcp

auto lo

iface lo inet loopback

auto eth0

iface eth0 inet static

address 192.168.2.1

netmask 255.255.255.0

auto usb0

iface usb0 inet dhcp

this gives the Pi a static IP address of 192.168.2.1.

DHCP

The next stage is to give the other computers on your network an IP address, this is done with a dhcp server:
$ sudo apt-get install isc-dhcp-server
you will need to configure dhcp sudo nano /etc/dhcp/dhcpd.conf, here is mine:

default-lease-time 600;
max-lease-time 7200;
ddns-update-style none;

# If this DHCP server is the official DHCP server for the local
# network, the authoritative directive should be uncommented.
authoritative;

log-facility local7;

subnet 192.168.2.0 netmask 255.255.255.0 {
  range 192.168.2.50 192.168.2.150;
  option broadcast-address 192.168.2.255;
  option routers 192.168.2.1;
  option subnet-mask 255.255.255.0;
  default-lease-time 600;
  max-lease-time 7200;
  option domain-name "fibble.local";
  option domain-name-servers 208.67.222.222, 208.67.220.220;
  interface eth0;
}
# give your network printer and file server a static IP addresses
host printer1 {
   hardware ethernet 08:00:2b:4c:59:23;
   fixed-address 192.168.2.45;
}
host fileserver {
   hardware ethernet 12:20:cb:3c:32:4d;
   fixed-address 192.168.2.40;
}

default-lease-time 600;

max-lease-time 7200;

ddns-update-style none;

# If this DHCP server is the official DHCP server for the local

# network, the authoritative directive should be uncommented.

authoritative;

log-facility local7;

subnet 192.168.2.0 netmask 255.255.255.0 {

range 192.168.2.50 192.168.2.150;

option broadcast-address 192.168.2.255;

option routers 192.168.2.1;

option subnet-mask 255.255.255.0;

default-lease-time 600;

max-lease-time 7200;

option domain-name "fibble.local";

option domain-name-servers 208.67.222.222, 208.67.220.220;

interface eth0;

}

# give your network printer and file server a static IP addresses

host printer1 {

hardware ethernet 08:00:2b:4c:59:23;

fixed-address 192.168.2.45;

}

host fileserver {

hardware ethernet 12:20:cb:3c:32:4d;

fixed-address 192.168.2.40;

}

This will assign IP addresses in the range 192.168.2.50 to 192.168.2.150 to any computer connected to your network. I have used Open DNS for the Domain name Servers, if you wish to use google’s use:
option domain-name-servers 8.8.8.8, 8.8.4.4;instead.

I have also given my network printer a static IP address, it is still assigned by the DHCP server but never changes, the same would apply to any file servers and the like, I would assign static devices addresses that are outside your dynamically assigned range. Reboot the Pi and then your test computer.

Your test computer should now have an IP address (192.168.2.51), and the gateway point to the Pi (92.168.2.1):

$ ifconfig
eth0      Link encap:Ethernet  HWaddr 2c:6f:63:8c:21:5f
          inet addr:192.168.2.51  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::1e6f:65ff:fe89:216f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:24232 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15622 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15146663 (15.1 MB)  TX bytes:2571110 (2.5 MB)
$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.2.1     0.0.0.0         UG    0      0        0 eth0
192.168.2.0     *               255.255.255.0   U     1      0        0 eth0

$ ifconfig

eth0 Link encap:Ethernet HWaddr 2c:6f:63:8c:21:5f

inet addr:192.168.2.51 Bcast:192.168.2.255 Mask:255.255.255.0

inet6 addr: fe80::1e6f:65ff:fe89:216f/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:24232 errors:0 dropped:0 overruns:0 frame:0

TX packets:15622 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:15146663 (15.1 MB) TX bytes:2571110 (2.5 MB)

$ route

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

default 192.168.2.1 0.0.0.0 UG 0 0 0 eth0

192.168.2.0 * 255.255.255.0 U 1 0 0 eth0

Accessing The Internet

The final part is to have the incoming traffic on the the Ethernet port eth0, go out on the dongle usb0. This is achieved with iptables, a firewall and traffic router. Install with:
$ sudo apt-get install iptables
and you need to setup Network Address Translation, NAT and forwarding. This short bash script clears any old settings before applying the new rules:

#!/bin/bash
LAN="eth0"
WAN="usb0"

# flush tables
iptables -F
iptables -t nat -F

# apply routing
iptables -t nat -A POSTROUTING -o $WAN -j MASQUERADE
iptables -A FORWARD -i $WAN -o $LAN -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i $LAN -o $WAN -j ACCEPT

# make ZTE web interface available
iptables -t nat -A PREROUTING -p tcp --dport 2525 -j DNAT --to-destination 192.168.0.1:80

#!/bin/bash

LAN="eth0"

WAN="usb0"

# flush tables

iptables -F

iptables -t nat -F

# apply routing

iptables -t nat -A POSTROUTING -o $WAN -j MASQUERADE

iptables -A FORWARD -i $WAN -o $LAN -m state --state RELATED,ESTABLISHED -j ACCEPT

iptables -A FORWARD -i $LAN -o $WAN -j ACCEPT

# make ZTE web interface available

iptables -t nat -A PREROUTING -p tcp --dport 2525 -j DNAT --to-destination 192.168.0.1:80

Where LAN is your internal network, and WAN is the internet. The final line allows you access to the Dongle’s built in web status page from any browser on your internal network, just use: http://192.168.2.1:2525

Save the file in your home directory as ~/ipt.sh, make it executable and run the script.
$ chmod +x ~/ipt.sh $ sudo ~/ipt.sh
From your test computer, you will now be able to access the internet.

Finally, you now need to have iptables reload when you start the Pi. Export the iptables settings to a file with:
$ sudo sh -c "iptables-save > /etc/iptables.ipv4.nat"
and create a file sudo nano /etc/network/if-up.d/iptables with the following contents:

#!/bin/bash
/sbin/iptables-restore < /etc/iptables.ipv4.nat

1 2	#!/bin/bash /sbin/iptables-restore < /etc/iptables.ipv4.nat

and make it executable sudo chmod +x /etc/network/if-up.d/iptables

after a reboot you can see your iptables with sudo iptables -L and sudo iptables -t nat -L and you can see web traffic passing through the router with sudo tcpdump -i any -nn port 80.

Adding a Proxy Server

This is optional, but a transparent proxy server and cache may reduce the amount of traffic on your 3G/4G connection, mileage varies and the amount of data cached was less than I thought it would be, I also found that my Humax Freesat box really didn’t like the proxy server and wouldn’t update its TV schedules while it was on. I have used squid3 for this.
sudo apt-get install squid3
Update the squid3 configuration /etc/squid3/squid.conf so it has the following. The original is rather large, so you may want to make a copy and create a new one:

http_port 3128 transparent

refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320

acl manager url_regex -i ^cache_object:// +i ^https?://[^/]+/squid-internal-mgr/

acl localnet src 192.168.2.0/24
acl localhost src 127.0.0.1/32

acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT

http_access allow manager localhost
http_access allow manager all
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
http_access allow localnet
http_access deny all

cache allow all
cache_mem 128 MB
cache_dir ufs /var/spool/squid3 5000 16 256

http_port 3128 transparent

refresh_pattern ^ftp: 1440 20% 10080

refresh_pattern ^gopher: 1440 0% 1440

refresh_pattern -i (/cgi-bin/|\?) 0 0% 0

refresh_pattern . 0 20% 4320

acl manager url_regex -i ^cache_object:// +i ^https?://[^/]+/squid-internal-mgr/

acl localnet src 192.168.2.0/24

acl localhost src 127.0.0.1/32

acl SSL_ports port 443

acl Safe_ports port 80 # http

acl Safe_ports port 21 # ftp

acl Safe_ports port 443 # https

acl Safe_ports port 70 # gopher

acl Safe_ports port 210 # wais

acl Safe_ports port 1025-65535 # unregistered ports

acl Safe_ports port 280 # http-mgmt

acl Safe_ports port 488 # gss-http

acl Safe_ports port 591 # filemaker

acl Safe_ports port 777 # multiling http

acl CONNECT method CONNECT

http_access allow manager localhost

http_access allow manager all

http_access deny manager

http_access deny !Safe_ports

http_access deny CONNECT !SSL_ports

http_access allow localhost

http_access allow localnet

http_access deny all

cache allow all

cache_mem 128 MB

cache_dir ufs /var/spool/squid3 5000 16 256

then restart squid3
sudo /etc/init.d squid3 restart
add the following iptables rule to redirect all traffic on port 80 to squid3:
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 3128
you should now be able to watch the web traffic being processed through squid3 with:
sudo tail /var/log/squid3/access.log -f
finish off by exporting your iptables again, so they are reloaded on reboot:
sudo sh -c "iptables-save > /etc/iptables.ipv4.nat"

Pi 4G Router — Raspberry Pi as 4G Router

Setup The PI

Setup Windows for SSH

Setup VS Code

Links and Sources

Installation

Configuration

Connecting

Using OpenCV

Links and Sources

Initial Startup

Customising the Setup

Post Install Problems

Setup the Machine Learning

Further Reading

1. Making a copy of the SD Card.

2. Resizing the partition

Links and Sources

Installing the gopro-utils

Extracting the Data

Merging GPX files

Links and Sources

Links and Sources

Alternative Method

Creating an Audiobook

Links and References

Installing BlueZ

Configuring and Starting BlueZ

Testing BlueZ

Those Scripting BlueZ

More Bluetooth Data

Links and Sources

Current Setup

Preparing

Installing

Testing

Sources

Which IP addresses to use?

Setup the Dongle

Configuring the network

DHCP

Accessing The Internet

Adding a Proxy Server

References and sources:

COMPUTERS, ROBOTS AND PHOTOGRAPHY