Fancy Blink for Arduino

9th August 2022 Karl Leave a comment

This little blink program for the Arduino blinks an LED to give two short flashes and then a longer pause. The milliseconds are kept in an array ledDelays. It iterates through the array, setting the class=’bash’>interval according where the delayCount pointer is set. The LED state is toggled when the interval is complete.

The char pout[50] is used as a tidy way of outputting to the serial port as some older Arduino board don’t support Serial.printf.

#include <Arduino.h>

#define MONITOR_LED LED_BUILTIN

const int delayCount = 4;
// led blink times in millis, LED state: off,  on, off, on
const uint32_t ledDelays[delayCount] = { 1500, 100, 90, 200}; 

// add more blinks, like so:
// const int delayCount = 6;
// const uint32_t ledDelays[delayCount] = { 1500, 100, 90, 200, 50, 100}; 

int delayID = 0;
uint32_t interval;
uint32_t millisNow;
uint32_t millisPrevious;
bool ledToggle = false;
char pout[50] = {0};

void setup() {

  Serial.begin(9600);
  delay(6000);
  Serial.println("ready");

  pinMode(MONITOR_LED, OUTPUT);

  interval = ledDelays[0];
}

void loop() {

  millisNow = millis();

  if ((millisNow - millisPrevious) >= interval) {
    digitalWrite(MONITOR_LED, ledToggle);

    sprintf(pout, "%d, %d, %d", delayID, ledToggle, interval);
    Serial.println(pout);
    // Serial.printf("%d, %d, %d\n", delayID, ledToggle, interval);

    ledToggle = !(ledToggle);       // change the state of the LED
    millisPrevious = millisNow;     // blink without delay()
    interval = ledDelays[delayID];  // set the delay time for the next iteration
    delayID ++;
    if (delayID >= delayCount) {  // reset the delayID 
      delayID = 0;
    }
  }
}

#include <Arduino.h>

#define MONITOR_LED LED_BUILTIN

const int delayCount = 4;

// led blink times in millis, LED state: off, on, off, on

const uint32_t ledDelays[delayCount] = { 1500, 100, 90, 200};

// add more blinks, like so:

// const int delayCount = 6;

// const uint32_t ledDelays[delayCount] = { 1500, 100, 90, 200, 50, 100};

int delayID = 0;

uint32_t interval;

uint32_t millisNow;

uint32_t millisPrevious;

bool ledToggle = false;

char pout[50] = {0};

void setup() {

Serial.begin(9600);

delay(6000);

Serial.println("ready");

pinMode(MONITOR_LED, OUTPUT);

interval = ledDelays[0];

}

void loop() {

millisNow = millis();

if ((millisNow - millisPrevious) >= interval) {

digitalWrite(MONITOR_LED, ledToggle);

sprintf(pout, "%d, %d, %d", delayID, ledToggle, interval);

Serial.println(pout);

// Serial.printf("%d, %d, %d\n", delayID, ledToggle, interval);

ledToggle = !(ledToggle); // change the state of the LED

millisPrevious = millisNow; // blink without delay()

interval = ledDelays[delayID]; // set the delay time for the next iteration

delayID ++;

if (delayID >= delayCount) { // reset the delayID

delayID = 0;

}

Computer, linux, Raspberry Pi

Setting up SSH in VS Code to connect to a Raspberry Pi

26th April 2022 Karl Leave a comment

There have been a lot of posts written about how to connect from Visual Studio Code (VS Code) on a Windows PC to a Raspberry PI over SSH using public/private keys avoiding the need to use a password. This is another one, and I aim to be fairly terse.

The PC is running Windows 11, the Raspberry Pi is running the latest version of Raspberry Pi OS – Debian Bullseye. Note that <username> is a placeholder for your own username, the greater than symbol: > represents a Windows PowerShell prompt and the dollar sign: $ is a Linux Bash prompt.

Setup The PI

SSH needs enabling on the PI and an ~/.ssh directory needs creating in your users home directory.

On the Pi, in the terminal enable SSH with: sudo raspi-config then select 3 Interface Options > I2 SSH and choose Yes – Enable. Now create the directory where your keys will be kept: mkdir ~/.ssh

It is also useful to find the IP address of your PI with: hostname -I Mine is 192.168.2.45 and I shall being using this in the examples.

You should now be able to login to the Pi from the Windows PC, open Windows Terminal or PowerShell and use ssh <username>@192.168.2.45 answer yes to add it to the known hosts file and it will ask for your PI’s password. If your Windows username is the same as that on the PI you can login with ssh 192.168.2.45.

Setup Windows for SSH

This stage installs the SSH client on the PC, creates a keypair and copied the public key to the PI.

The OpenSSH client can be installed using Windows Settings on Windows Server 2019 and Windows 10 and above devices.

Open Settings, select Apps > Apps & Features, then select Optional Features.
Scan the list to see if the OpenSSH is already installed. If not, at the top of the page, select Add a feature, then:
Find OpenSSH Client, then click Install

A full description is given on Microsofts Get started with OpenSSH page.

On the Windows PC, in your PowerShell create the directory for your keys if it doesn’t exist already mkdir ~/.ssh and then change into it cd ~/.ssh generate a keypair and add it to the ssh-agent:

> ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (C:\Users\<username>/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\Users\<username>/.ssh/id_ed25519.
Your public key has been saved in C:\Users\<username>/.ssh/id_ed25519.pub.
The key fingerprint is:
SHA256:YCETnl4GmmYSokjE+vE5UaJ1/nzXB1tboE730sxkujw <username>@<computer>
The key's randomart image is:
+--[ED25519 256]--+
|=o  =..          |
|=o +o=o.      .  |
|= =oo==      . . |
|.+o..+..    o o =|
| . o.o oS  o o %o|
|  . +   o . o =.*|
|     .   . . . + |
|              E  |
|               . |
+----[SHA256]-----+
PS C:\Users\<username>\.ssh> ssh-add ~\.ssh\id_ed25519

> ssh-keygen -t ed25519

Generating public/private ed25519 key pair.

Enter file in which to save the key (C:\Users\<username>/.ssh/id_ed25519):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in C:\Users\<username>/.ssh/id_ed25519.

Your public key has been saved in C:\Users\<username>/.ssh/id_ed25519.pub.

The key fingerprint is:

SHA256:YCETnl4GmmYSokjE+vE5UaJ1/nzXB1tboE730sxkujw <username>@<computer>

The key's randomart image is:

+--[ED25519 256]--+

|=o =.. |

|=o +o=o. . |

|= =oo== . . |

|.+o..+.. o o =|

| . o.o oS o o %o|

| . + o . o =.*|

| . . . . + |

| E |

| . |

+----[SHA256]-----+

PS C:\Users\<username>\.ssh> ssh-add ~\.ssh\id_ed25519

We now need to copy the public key to the PI:

> scp ~\.ssh\id_ed25519.pub <username>@192.168.2.45:~/.ssh/

1	> scp ~\.ssh\id_ed25519.pub <username>@192.168.2.45:~/.ssh/

And on the PI we need to add that public key to the authorized keys, connect into the PI ssh <username>@192.168.2.45 and:

$ cd ~/.ssh
$ touch authorized_keys
$ cat id_ed25519.pub >> authorized_keys
$ chmod 600 authorized_keys
$ rm id_ed25519.pub

$ cd ~/.ssh

$ touch authorized_keys

$ cat id_ed25519.pub >> authorized_keys

$ chmod 600 authorized_keys

$ rm id_ed25519.pub

the commands, touch creates a blank file if it doesn’t exist, cat appends the pub file to authorized_keys, making sure that any existing keys are preserved permissions are set with chmod and finally the pub file is deleted.

Setup VS Code

Load up VS Code and add the Remote – SSH and Remote SSH: Editing Configuration Files extensions, once restarted you will see a >< icon in the bottom left-hand corner with a green background, click that and select Edit SSH Configuration file…, choose the one in your home directory C:\Users\<username>\.ssh\config and add the following:

Host piComputer_192.168.2.45
  HostName 192.168.2.45
  User <username>
  ForwardAgent yes
  IdentityFile C:/Users/<username>/.ssh/id_ed25519

Host piComputer_192.168.2.45

HostName 192.168.2.45

User <username>

ForwardAgent yes

IdentityFile C:/Users/<username>/.ssh/id_ed25519

The Host “piComputer_192.168.2.45” is the friendly name displayed within VS Code for when you are choosing it from lists, the User id your user on the PI. Save the config file when complete.

You should now be able to connect to the PI. Click the >< icon in the bottom left-hand corner, and choose Connect To Host… A popup saying Setting Up… should appear, it may ask if the host is Linux or Windows, choose Linux, and within a few seconds it’ll be connected and ready. From within VS Code you can open a Bash Terminal – Ctrl+Shift+’

If it asks for a password, you may have missed a stage, or copied the private key to the PI instead of the public one.

Links and Sources

Computer

Debian Linux Post Install Notes

21st December 2021 Karl Leave a comment

I have just installed a new Debian Linux server, this will be running mainly as a web server without a GUI and console access will be via an SSH login, some knowledge of using Linux is assumed as these notes are intended as a prompt or overview of things to do to as part of a Linux server installation.

I am using Debian version 11.2 – Bullseye, the current version at time of writing, but these notes should work with any distribution that uses the apt package manager like Ubuntu. The ISO image used for installation came from https://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/ and I have used the Xfce install: debian-live-11.2.0-amd64-xfce.iso (2.4G download). A guide for installing the Debian image can be found on this TecAdmin page.

At the Console

Your computer is now inviting you to login, do so with your local user and once the default desktop has been created open a terminal window to complete the following; switch to the root user and add your local user to the sudoers group:

$ su
<root password>
# /sbin/usermod -aG sudo <username>

$ su

# /sbin/usermod -aG sudo <username>

Now logout and back in, this updates your user account settings, and go back to the terminal to update the system, install ssh access and a couple of useful tools, and finally set the computer to boot to console rather than the GUI:

$ sudo apt update
$ sudo apt upgrade
$ sudo apt install openssh-server wget curl
$ sudo systemctl set-default multi-user.target

$ sudo apt update

$ sudo apt upgrade

$ sudo apt install openssh-server wget curl

$ sudo systemctl set-default multi-user.target

If you have installed into a virtual machine now is a good time to install the guest client, links to popular VM’s:

VMware: https://kb.vmware.com/s/article/1018414
Oracle Vbox: https://linoxide.com/how-to-install-virtualbox-gue…
Synology Virtual Machine Manager: https://kb.synology.com/en-uk/DSM/tutorial/How_to_install_Synology…. (at the bottom of the page)

$ sudo apt install qemu-guest-agent

Finding your IP address is useful:

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 02:11:32:23:83:6c brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet 192.168.1.20/24 brd 192.168.1.255 scope global dynamic noprefixroute ens3
       valid_lft 86253sec preferred_lft 86253sec
    inet6 fe80::11:32ff:fe23:836c/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

$ ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

link/ether 02:11:32:23:83:6c brd ff:ff:ff:ff:ff:ff

altname enp0s3

inet 192.168.1.20/24 brd 192.168.1.255 scope global dynamic noprefixroute ens3

valid_lft 86253sec preferred_lft 86253sec

inet6 fe80::11:32ff:fe23:836c/64 scope link noprefixroute

valid_lft forever preferred_lft forever

In this instance it is 192.168.1.20. For a server I would normally fix the IP address in my router as a Reserved Address using the MAC (in this case: 02:11:32:23:83:6c).

Removing libreoffice clears a useful amount of diskspace:

$ sudo apt remove libreoffice*
$ sudo apt autoremove

1 2	$ sudo apt remove libreoffice* $ sudo apt autoremove

Restart with $ sudo reboot You can now login via SSH from your other computer or continue at the console.

Installing the LAMP service – MySql, Apache and PHP

Start with MySql (MariaDB being the open source version), installing, configuring and starting:

$ sudo apt install mariadb-server
$ sudo mysql_secure_installation
$ sudo systemctl enable mariadb.service
$ sudo systemctl start mariadb.service

$ sudo apt install mariadb-server

$ sudo mysql_secure_installation

$ sudo systemctl enable mariadb.service

$ sudo systemctl start mariadb.service

For the secure installation, use the defaults and set the password when prompted. Next is Apache and PHP:

$ sudo apt install apache2 php libapache2-mod-php php-gd php-mysql php-mbstring php-xml
$ sudo systemctl enable apache2.service
$ sudo systemctl start apache2.service

$ sudo apt install apache2 php libapache2-mod-php php-gd php-mysql php-mbstring php-xml

$ sudo systemctl enable apache2.service

$ sudo systemctl start apache2.service

Point your web browser to the server address, in my case: http://192.168.1.20, and you should see the default Apache page. The web pages are in /var/www I normally add a symbolic link and change permissions on the html directory so I can add files easily:

$ cd ~
$ ln -s /var/www www
$ cd www
$ sudo chown -R <username>:www-data html/

$ cd ~

$ ln -s /var/www www

$ cd www

$ sudo chown -R <username>:www-data html/

Accessing a Network Drive

I want to be able to access some network drives from this server, these are setup as Windows (SMB) file shares.

$ sudo apt install cifs-utils
$ sudo mkdir /mnt/nwData

1 2	$ sudo apt install cifs-utils $ sudo mkdir /mnt/nwData

create a file with your file share username and password, this should go into /etc, I normally call the file the name of the server I want to connect to, and set permissions so it is only readable by root:

$ sudo nano /etc/<servername>
USERNAME=<username> 
PASSWORD=<password> 
$ sudo chmod 500 /etc/<servername>

$ sudo nano /etc/<servername>

USERNAME=<username>

PASSWORD=<password>

$ sudo chmod 500 /etc/<servername>

And add the connection to the end of your /etc/fstab file, and mount the drives:

$ sudo nano /etc/fstab
//192.168.1.101/BigFileStore /mnt/nwData cifs credentials=/etc/<servername>,iocharset=utf8,uid=<username> 0 0
$ sudo mount -a

$ sudo nano /etc/fstab

//192.168.1.101/BigFileStore /mnt/nwData cifs credentials=/etc/<servername>,iocharset=utf8,uid=<username> 0 0

$ sudo mount -a

Check that it mounted with $ df -h. The file share will also automatically mount at boot.

Other Useful Things

Installing a downloaded .deb file with dependencies, in this case for the no longer manufactured Logitech Media Server:

$ cd ~/Downloads/
$ wget https://downloads.slimdevices.com/nightly/8.3/lms/3ee0cb0f42370ae36c97654ff151a0bb08f117a9/logitechmediaserver_8.3.0~1640066892_amd64.deb
$ sudo apt install ./logitechmediaserver_8.3.0~1640066892_amd64.deb
$ sudo systemctl enable logitechmediaserver.service
$ sudo systemctl start logitechmediaserver.service

$ cd ~/Downloads/

$ wget https://downloads.slimdevices.com/nightly/8.3/lms/3ee0cb0f42370ae36c97654ff151a0bb08f117a9/logitechmediaserver_8.3.0~1640066892_amd64.deb

$ sudo apt install ./logitechmediaserver_8.3.0~1640066892_amd64.deb

$ sudo systemctl enable logitechmediaserver.service

$ sudo systemctl start logitechmediaserver.service

Links and Sources

More details on Installing LAMP
Setup Samba Server on Debian 10
Visual Studio Code: Remote Development using SSH
Installing Node JS on Debian 11
phpMyAdmin – a web frontend for MySql

Computer, Hardware, Interesting

Setting up a Cisco SPA122 for use with UK Telephones

28th September 2021 Karl Leave a comment

This is more of a nostalgia trip / messing about thing rather than a serious use of Voice over IP (VoIP). The Cisco SPA122 ATA with Router (Analog Telephone Adapter) is a small box that allows old analog phones to be connected to a VoIP network. At the time of writing Cisco no longer manufactures these and they are currently in an end of life plan.

I am setting this up to use two handsets that can be used to call each other through the SPA122. I don’t have a VoIP service here and don’t have need for one. Later, I will be setting up a couple of modems and using them for a dialup connection, in this part I will be going through the setup and configuration of the SPA122.

You will need:

An SPA122 – I got mine from eBay, check that you are purchasing a vanilla model, rather than one that has been configured by a carrier, be wary if it has any other logos printed on it as it may be locked to that carrier.
A 5V power supply, this must be capable of providing at least 2A.
Two telephones – I got these from the Charity Shop, get ones with tone dialing (DTMF), pulse dialing will not work – I’ve also found using better quality phones made in the early 2000’s work better than cheap new ones, I ended up with BT Decor 1100 and BT Decor 1200 phones.
Two adapters to convert the UK phone plug to RJ11 sockets on the SPA122. Search for “RJ11 to BT Socket” on eBay or Amazon, these are cheap.
A wired LAN connection to your network.
A computer, preferably with WiFi as well as a Wired Connection, I used a Raspberry Pi 4.

Default Settings

IP Address: 192.168.15.1 – with a computer plugged into the yellow ethernet port
Username: admin
Password: admin

Resetting the SPA122 and Gaining Access

With a network lead from your computer to the yellow ethernet port on the SPA122 open your web browser and try the address 192.168.1.15. You should now see the login screen, try the login details above.

That probably didn’t work. To reset to factory default settings use a paperclip on the rear panel reset button. With the unit powered on, use the end of a paperclip to press into the red hole marked RESET located on the bottom-left side of the back panel of the SPA122. Press and hold for at least a full 20 seconds and the Power LED will start to blink when the restore process begins, this will take around two minutes to complete. Now try logging in as admin again.

Once logged in, change the admin password to something else, if left at the default it will refuse to work.

If you are really unlucky the reset button may have been disabled in the firmware. At this stage I don’t know what to do if this has happened.

You can also configure the device by pressing the star button on your handset four times (****) but that way madness lies.

Upgrade the Firmware

At time of writing, the firmware version available from Cisco is Release 1.4.1 SR5. The update appears to be fixing a variety of security problems. You can find the current Firmware Version in Status > System Information.

Updating is straight forward. Download the file, and unzip it to extract the Payton_1.4.1_SR5_101419_1250_pfmwr.bin file, then in the SPA122 pages in your web browser go to Administration > Firmware Upgrade and follow the instructions. Everything here has been written for this version of the firmware.

Network Configuration

Even though in this case we won’t be using this as VoIP over the network, it still needs a network connection to operate.

In the Network Setup > Network Service set this to Bridge. This will allow a computer plugged into the yellow ethernet port to access / pass through to the blue Internet port which would be connected to your LAN.

Make a note of the SPA122’s WAN MAC Address, found in Status > System Settings, it may be useful for finding the box on your network when you make it available on your LAN in the next step.

Now to access the SPA122’s web pages from your LAN, go to Administration > Web Access Management and set Admin Access to Enable. Apply the changes and attach a cable from your LAN to the blue Internet port, you should now be able to login from any computer on your network. Your main router will have a list of attached devices, use the MAC address to help quickly find the designated IP address. The computer you were using to configure the ATA may need restarting to pick up an IP address from the LAN.

Although I have not tested this, with a handset attached to one of the phone ports dial * * * * and listen, if you can hear a slightly sinister automated voice saying “System Configuration Menu” then dial 1 1 0 # slowly, the key presses are read back to you, the device will then tell you its IP address 192.168.x.x.

Telephone Configuration

Now we are in the Voice section. At this stage listening on a handset just produces silence, or a quiet buzzing, like from a poor power supply. For setting up the ATA will need to be restarted several times. Go to Quick Setup and set the user and dial plan for each line. The dial plan describes how the call is to be handled, numbers can be blocked, routed to different ones, or in this case be routed to another port.

Line 1:
User ID: 101
Dial Plan:
(*xx|[3469]11|0|00|[2-9]xxxxxx|1xxx[2-9]xxxxxxS0|xxxxxxxxxxxx.|<018118055:101>S0<:@127.0.0.1:5061>|)

Line 2:
User ID: 100
Dial Plan:
(*xx|[3469]11|0|00|[2-9]xxxxxx|1xxx[2-9]xxxxxxS0|xxxxxxxxxxxx.|<018118055:100>S0<:@127.0.0.1:5060>|)

I have used 018118055 as the number, but you can set this to anything you like. Both phones are on the same number, but they behave separately so you can ring one from the other. If the phone has a caller ID display it will show the user ID, 100 or 101.

Click submit to save the changes, once restarted and logged back in, we now need to add a bit more detail, go to Voice > Line 1, and set the following:

Proxy and Registration:
Make Call Without Reg:	yes
Ans Call Without Reg:	yes
Audio Configuration:
Preferred Codec:	G729a
Second Preferred Codec:	G711u
All lines that start with FAX:	set to no or none as appropriate
Modem Line:	yes

The FAX settings are to do with using modems in the future. All other settings can be left at their defaults. Click submit to save the settings, this takes a few seconds, and once you have logged back in go to Voice > Line 2 and set the same settings as those you just set for Line 1.

You should now be able to make a call from one phone to the other, but the ring and dial tones will sound wrong as they are for those used in the USA.

Setting the Tones

Now we need to set the dial tones and ringer for those used in the UK, in Voice > Regional you can see there are quite a few options to be set, but here I will only be setting the ones most commonly used.

Call Progress Tones:
Dial Tone:	350@-19,440@-22;10(*/0/1+2)
Busy Tone:	400@-20;10(.375/.375/1)
Off Hook Warning Tone:	480@-10,620@-16,1400@0,2060@0,2450@0,2600@0;60(.2/0/1,.2/0/2);240(.1/.1/3+4+5+6)
Distinctive Ring/CWT Pattern Names:
Ring1 Cadence:	60(.4/.2,.4/2)
Ring2 Cadence:	60(1/2)
Ring3 Cadence:	60(.25/.25,.25/.25,.25/1.75)
Ring4 Cadence:	60(.4/.8)
Ring5 Cadence:	60(2/4)
Ring and Call Waiting Tone Spec:
Ring Waveform:	Trapezoid
Ring Frequency:	25
Ring Voltage:	60
Frequency:	400@-10
Miscellaneous:
FXS Port Impedance:	370+620\|\|310nF
FXS Port Input Gain:	0
Caller ID Method:	ETSI FSK With PR (UK)

My source for UK Regional Settings is below in the Links and Sources where you can find the complete list.

My Work Here is Done

Congratulations, you can now ring one telephone from another using the old fashioned technique of pushing buttons.

Next in this series will be setting up for modems, and involves using a Raspberry Pi as a router.

Links and Sources

UK Regional Settings: https://basichelp.sipgate.co.uk/hc/en-gb/articles/206289069-UK-Regional-Settings-Cisco-Linksys-Sipura-Adaptors-
Resetting the SPA112: https://www.ukvoipforums.com/viewtopic.php?t=38
Cisco Firmware Update: https://software.cisco.com/download/home/283998793/type/282463187/release/1.4.1%20SR5
Cisco EOL Notice: https://www.cisco.com/c/en/us/support/unified-communications/spa122-ata-router/model.html?dtid=osscdc000283
How [UK] Telephones work (line voltages) https://www.britishtelephones.com/howtele.htm
Cisco SPA100 Series quick start guide, with configuration using a phone https://www.cisco.com/c/dam/en/us/td/docs/voice_ip_comm/csbpvga/spa100-200/quick_start/SPA100_ATA_78-19933.pdf [PDF]
How Dial Plans Work: Chapter: Configuring Dial Plans
VoIP.MS Wiki https://wiki.voip.ms/article/Cisco_SPA112

Computer, linux, Raspberry Pi

Setup a Network Camera with a Raspberry PI

1st July 2021 Karl 2 Comments

In this post I am setting up a network available RTSP video camera on a Raspberry Pi 4 Model B with the latest Debian Buster installed, you will also want another ‘remote’ computer with VLC installed on the same network. I have used a camera module connected via a ribbon cable to the Pi’s video port. The performance is more like what you would see from a security camera rather than some fancy dedicated device. I will be making the configuration with a bash shell using SSH rather than the GUI.

Installation

If you haven’t already, enable the camera on the Pi with sudo raspi-config and go to Interface Options > Camera. To reduce the load on the computer I also set the Pi to boot to the Console in System Options > Boot / Auto Login and selecting Console, you should also check that at least 256MB of memory is allocated to the GPU in Performance Options > GPU Memory, when complete reboot to set the updated configuration.

Install some prerequisites, you may have some or all of these installed already:

$ sudo apt update
$ sudo apt install curl git build-essential software-properties-common cmake

1 2	$ sudo apt update $ sudo apt install curl git build-essential software-properties-common cmake

Next add the Video For Linux (V4L) Repository, this adds the drivers for the video on the hardware side

$ curl https://www.linux-projects.org/listing/uv4l_repo/lpkey.asc | sudo apt-key add -

1	$ curl https://www.linux-projects.org/listing/uv4l_repo/lpkey.asc \| sudo apt-key add -

and add the following line to your apt sources with $ sudo nano /etc/apt/sources.list

deb https://www.linux-projects.org/listing/uv4l_repo/raspbian/stretch stretch main

1	deb https://www.linux-projects.org/listing/uv4l_repo/raspbian/stretch stretch main

We can now install the V4L packages:

$ sudo apt update
$ sudo apt install uv4l uv4l-raspicam liblog4cpp5-dev libv4l-dev uv4l-server uv4l-raspicam-extras

1 2	$ sudo apt update $ sudo apt install uv4l uv4l-raspicam liblog4cpp5-dev libv4l-dev uv4l-server uv4l-raspicam-extras

We now need to install the RTSP server from Git, this server connects to the V4L drivers above:

$ cd ~
$ git clone https://github.com/mpromonet/v4l2rtspserver.git
$ cd v4l2rtspserver
$ cmake .
$ sudo make install
$ sudo reboot

$ cd ~

$ git clone https://github.com/mpromonet/v4l2rtspserver.git

$ cd v4l2rtspserver

$ cmake .

$ sudo make install

$ sudo reboot

Configuration

To setup V4L, edit the configuration file $ sudo nano /etc/uv4l/uv4l-raspicam.conf and update the driver options and uncomment the h264 options:

##################################
# raspicam driver options
##################################
encoding = h264
#720p
width = 1280
height = 720
framerate = 12

### h264 options:
profile = high
level = 4.2
bitrate = 8000000
intra-refresh-mode = dummy
intra-period = #arg
inline-headers = yes
sps-timing = no
quantisation-parameter #arg

##################################

# raspicam driver options

##################################

encoding = h264

#720p

width = 1280

height = 720

framerate = 12

### h264 options:

profile = high

level = 4.2

bitrate = 8000000

intra-refresh-mode = dummy

intra-period = #arg

inline-headers = yes

sps-timing = no

quantisation-parameter #arg

Restart the service once you have saved the file:

$ sudo service uv4l_raspicam restart

1	$ sudo service uv4l_raspicam restart

Useful screen Resolutions; designation, ratio: width x height:

1080p, 16:9: 1920 x 1080
720p, 16:9: 1280 x 720
720, 4:3: 1296 x 792
Standard Definition, 4:3: 640 x 480

I’ve chosen 720p 16:9 widescreen, this gives a good image quality without stressing the Pi 4B. The framerate is a compromise between quality and usability, its unlikely you would want to go above 25fps.

Connecting

If you don’t know it already, find the IP address of your Pi with $ ip a. Look for the address in eth0 or wlan0 depending on if you are using ethernet or wireless. In this case I will be using 192.168.2.33 in the examples.

Start the RTSP server with:

$ v4l2rtspserver -U user:password -F12 -W 1280 -H 720 -P 8555 /dev/video0

1	$ v4l2rtspserver -U user:password -F12 -W 1280 -H 720 -P 8555 /dev/video0

Replace user:password with a suitable username and password. I have assumed that the camera is on /dev/video0, you can test this with:

$ v4l2-ctl -d /dev/video0 -D

1	$ v4l2-ctl -d /dev/video0 -D

and look for a line such as Driver version: 5.10.17, if this is 0.0.0 then try a different device, /dev/video1, /dev/video2 etc.

On a successful load it will tell you the connection:

2021-07-01 10:07:39,251 [NOTICE] - /home/pi/v4l2rtspserver/inc/V4l2RTSPServer.h:81
        Play this stream using the URL "rtsp://192.168.2.33:8555/unicast"

1 2	2021-07-01 10:07:39,251 [NOTICE] - /home/pi/v4l2rtspserver/inc/V4l2RTSPServer.h:81 Play this stream using the URL "rtsp://192.168.2.33:8555/unicast"

Now on your ‘remote’ computer open VLC and choose Media > Open Network Stream and place the given stream address into the network URL, you will then be asked for the username and password. For me it took less than five seconds for an image to appear, there is a fairly high latency on the image shown, the picture lags by about a second from reality.

Using OpenCV

You can access the stream using Python scripts and OpenCV on your ‘remote’ computer for AI/Machine Learning. For me this is a Debian Linux box with FFmpeg as OpenCV uses this to access the stream.

#!/bin/env python

import os
import cv2

# if the script doesn't work, uncomment the following, OpenCV may be 
# defaulting to TCP when we are wanting to use UDP.
#os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = "rtsp_transport;udp"

# replace user:password with your username and password
rtspUrl = "rtsp://user:password@192.168.2.33:8555/unicast"

cap = cv2.VideoCapture(rtspUrl)  

#Check if camera was opened correctly
if not (cap.isOpened()):
    print("Could not open video device")

#Set the resolution
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1280)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 720)

while(True):
    ret, frame = cap.read()
    cv2.imshow('VIDEO', frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

#!/bin/env python

import os

import cv2

# if the script doesn't work, uncomment the following, OpenCV may be

# defaulting to TCP when we are wanting to use UDP.

#os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = "rtsp_transport;udp"

# replace user:password with your username and password

rtspUrl = "rtsp://user:password@192.168.2.33:8555/unicast"

cap = cv2.VideoCapture(rtspUrl)

#Check if camera was opened correctly

if not (cap.isOpened()):

print("Could not open video device")

#Set the resolution

cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1280)

cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 720)

while(True):

ret, frame = cap.read()

cv2.imshow('VIDEO', frame)

if cv2.waitKey(1) & 0xFF == ord('q'):

break

cap.release()

cv2.destroyAllWindows()

You can check OpenCV is working with FFmpeg with:

$ python -c "import cv2; print(cv2.getBuildInformation())"

1	$ python -c "import cv2; print(cv2.getBuildInformation())"

and looking in the output for FFMPEG saying YES:

 Video I/O:
    DC1394:                      NO
    FFMPEG:                      YES
      avcodec:                   YES (58.109.100)
      avformat:                  YES (58.61.100)
      avutil:                    YES (56.60.100)
      swscale:                   YES (5.8.100)
      avresample:                NO
    GStreamer:                   NO
    v4l/v4l2:                    YES (linux/videodev2.h)

Video I/O:

DC1394: NO

FFMPEG: YES

avcodec: YES (58.109.100)

avformat: YES (58.61.100)

avutil: YES (56.60.100)

swscale: YES (5.8.100)

avresample: NO

GStreamer: NO

v4l/v4l2: YES (linux/videodev2.h)

Links and Sources

V4L Installation: https://www.linux-projects.org/uv4l/installation/
RTSP Server https://github.com/mpromonet/v4l2rtspserver

Interesting, Programming

Karl’s Roast Calculator – Updated

18th November 2020 Karl

I’ve updated my Roast Calculator to make it work clearer with mobile devices. Use it to find the cooking time of your favorite meat roast.

Link: https://www.g7smy.co.uk/roast

history, Sheffield

History: The Watermills of Sheffield

9th November 2020 Karl

There are 149 watermills and dams shown on this interactive map, dating from the 1200’s to the 1930’s all using water power to drive tilt hammers, blowers for forgers, grinding wheels for sharpening knifes and scythes as well for the milling of corn and other grains. Each mill has a brief chapter of the places history taken from the book ‘The Water-Mills of Sheffield’ by W.T. Miller, 2nd Issue, published in 1947.

The map has been designed for use with mobile phones, and the GPS can be used to find your location when you are out and about, on a desktop computer the GPS will not work. Click on a dot to see the mill name, and then click on the name for some interesting facts. On your mobile use the menu (top right) to turn GPS on to show your position.

Link: https://www.g7smy.co.uk/history/watermills/

Computer, Jetson, linux, Programming

Jetson Nano Install Notes

8th October 2020 Karl

The Jetson Nano Developer kit – B01 is a small computer comprising of an NVIDA Maxwell GPU, Quad-Core ARM Cortex-A57 Processor and 4GB of Memory along with four USB 3 ports, Gigabit Ethernet, HDMI and Display Port output, main storage is on a MicroSD card and there is a variety of selection of expansion available via GPIO, I2C and UART. On the software side NVIDIA provide their JetPack SDK – a customised version of Ubuntu. This development kit has been produced to provide an entry point into Machine Learning, for which I will be using Python programming language. I got my board from Pimoroni

GPU	128-core Maxwell
CPU	Quad-core ARM A57 @ 1.43 GHz
Memory	4 GB 64-bit LPDDR4 25.6 GB/s
Storage	microSD (not included)
Video Encode	4K @ 30 \| 4x 1080p @ 30 \| 9x 720p @ 30 (H.264/H.265)
Video Decode	4K @ 60 \| 2x 4K @ 30 \| 8x 1080p @ 30 \| 18x 720p @ 30 (H.264/H.265)
Camera	2x MIPI CSI-2 DPHY lanes
Connectivity	Gigabit Ethernet, M.2 Key E
Display	HDMI and display port
USB	4x USB 3.0, USB 2.0 Micro-B
Others	GPIO, I2C, I2S, SPI, UART
Mechanical	69 mm x 45 mm, 260-pin edge connector

These notes cover my process of setting one up and links to the documentation, it not intended to repeat those install guides but to provide an install sequence and any additional commentary as needed. I’m going to assume you have a little experience of using the terminal and am familiar with using the bash command line – I’ve no idea how this would be done through the GUI.

jetson nano ports — Jetson Nano ports [source: NVIDIA]

Initial Startup

I followed the instructions for downloading and installing JetPack 4.4 on https://nvidia.com/jetsonnano-start I used a 64GB Class 10, UHS-I, U3, V30 SanDisk card. I formatted the card in a camera before using balenaEtcher to write the JetPack SDK image, this creates a partition of about 16GB on the card formatted to ext4, during installation the volume is resized to fill the card.

jetson nano connections — Jetson Nano connections [source: NVIDIA]

Despite using a good quality USB power supply with an output of 3 Amps at 5 Volts into the Micro USB port the computer would only boot long enough for the NVIDIA logo to appear on screen but after a few seconds the green power LED would go out and it would be off, the same happened when I tried a variety of USB power supplies used. I got round the problem buy using a 5 Amp power supply connected to the barrel jack J25 on the left (centre pin positive) and connecting the jumper J48 located just behind this connector.

Customising the Setup

There are a couple of thigs to do, get a network volume mounted and set the default version of python.

For the network share install samba, some network utilities and the nano text editor:

$ sudo apt install samba cifs-utils nano

1	$ sudo apt install samba cifs-utils nano

create a text file: sudo nano /etc/samaba/videoserver with the following:

username=<your network username>
password=<your network password>

1 2	username=<your network username> password=<your network password>

And set the permissions sudo chmod 600 /etc/samba/videoserver. In this example I have a network share on my server; 192.168.1.30, called video. Create a mount point for the share: sudo mkdir /mnt/video now you need to edit fstab, sudo nano /etc/fstab and add your network connection to the end:

//192.168.1.30/video /mnt/video cifs credentials=/etc/samba/videoserver,noauto,x-systemd.automount,_netdev,uid=0,vers=1.0 0 0

1	//192.168.1.30/video /mnt/video cifs credentials=/etc/samba/videoserver,noauto,x-systemd.automount,_netdev,uid=0,vers=1.0 0 0

Reload fstab with sudo mount -a and check for any errors. Because of the way that Jetpack boots it does not appear to wait for the network so the share needs to be set to automount and this causes it to only appear in drive listings when accessed. Further reading can be found in this excellent guide to fstab: https://wiki.archlinux.org/index.php/fstab.

Jetpack 4.4 comes with two versions of Python, 2.7 and 3.6, I want it to default to 3.6 and while this is rather out of date I don’t want to go down the hole of upgrading just yet, you will also need to install pip and set pip3 as the default too.

$ sudo apt install python3-pip
$ sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.6 1
$ sudo update-alternatives --install /usr/bin/pip pip /usr/bin/pip3.6 1
$ sudo update-alternatives --set python /usr/bin/python3.6
$ sudo update-alternatives --set pip /usr/bin/pip3.6

$ sudo apt install python3-pip

$ sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.6 1

$ sudo update-alternatives --install /usr/bin/pip pip /usr/bin/pip3.6 1

$ sudo update-alternatives --set python /usr/bin/python3.6

$ sudo update-alternatives --set pip /usr/bin/pip3.6

and test its worked:

$ python
Python 3.6.9 (default, Jul 17 2020, 12:50:27)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

$ python

Python 3.6.9 (default, Jul 17 2020, 12:50:27)

[GCC 8.4.0] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>>

$ pip --version
pip 20.2.3 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)

1 2	$ pip --version pip 20.2.3 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)

I did get an error later on, a crash was reported on the desktop when an occasional python 2 script ran. I fixed the error in /usr/sbin/l4t_payload_updater_t210 by changing the first line of the file from !#/user/bin/python to !#/user/bin/python2

Post Install Problems

A recent update occured, so I did the usual sudo apt get update && sudo apt get upgrade but one of the files gave a script error, this turned out to be with nvidia-l4t-bootloader, like so:

$ sudo apt install nvidia-l4t-bootloader
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed
  nvidia-l4t-bootloader
0 to upgrade, 1 to newly install, 0 to remove and 0 not to upgrade.
Need to get 1,870 kB of archives.
After this operation, 8,251 kB of additional disk space will be used.
Get:1 https://repo.download.nvidia.com/jetson/t210 r32.4/main arm64 nvidia-l4t-bootloader arm64 32.4.3-20200924161615 [1,870 kB]
Fetched 1,870 kB in 3s (692 kB/s)
Selecting previously unselected package nvidia-l4t-bootloader.
(Reading database ... 165354 files and directories currently installed.)
Preparing to unpack .../nvidia-l4t-bootloader_32.4.3-20200924161615_arm64.deb ...
Unpacking nvidia-l4t-bootloader (32.4.3-20200924161615) ...
Setting up nvidia-l4t-bootloader (32.4.3-20200924161615) ...
3448-300---1--jetson-nano-qspi-sd-mmcblk0p1
Starting bootloader post-install procedure.
ERROR. Procedure for bootloader update FAILED.
Cannot install package. Exiting...
dpkg: error processing package nvidia-l4t-bootloader (--configure):
 installed nvidia-l4t-bootloader package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 nvidia-l4t-bootloader
E: Sub-process /usr/bin/dpkg returned an error code (1)

$ sudo apt install nvidia-l4t-bootloader

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following NEW packages will be installed

nvidia-l4t-bootloader

0 to upgrade, 1 to newly install, 0 to remove and 0 not to upgrade.

Need to get 1,870 kB of archives.

After this operation, 8,251 kB of additional disk space will be used.

Get:1 https://repo.download.nvidia.com/jetson/t210 r32.4/main arm64 nvidia-l4t-bootloader arm64 32.4.3-20200924161615 [1,870 kB]

Fetched 1,870 kB in 3s (692 kB/s)

Selecting previously unselected package nvidia-l4t-bootloader.

(Reading database ... 165354 files and directories currently installed.)

Preparing to unpack .../nvidia-l4t-bootloader_32.4.3-20200924161615_arm64.deb ...

Unpacking nvidia-l4t-bootloader (32.4.3-20200924161615) ...

Setting up nvidia-l4t-bootloader (32.4.3-20200924161615) ...

3448-300---1--jetson-nano-qspi-sd-mmcblk0p1

Starting bootloader post-install procedure.

ERROR. Procedure for bootloader update FAILED.

Cannot install package. Exiting...

dpkg: error processing package nvidia-l4t-bootloader (--configure):

installed nvidia-l4t-bootloader package post-installation script subprocess returned error exit status 1

Errors were encountered while processing:

nvidia-l4t-bootloader

E: Sub-process /usr/bin/dpkg returned an error code (1)

after some head scratching, it turns out this script is python2 only. I reverted back to the python 2.7 as the default, and tried again:

$ sudo update-alternatives --install /usr/bin/python python /usr/bin/python2.7 2
update-alternatives: using /usr/bin/python2.7 to provide /usr/bin/python (python) in auto mode
$ sudo update-alternatives --set python /usr/bin/python2.7
$ sudo apt remove nvidia-l4t-bootloader
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package 'nvidia-l4t-bootloader' is not installed, so not removed
0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.
$ sudo apt install nvidia-l4t-bootloader
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed
  nvidia-l4t-bootloader
0 to upgrade, 1 to newly install, 0 to remove and 0 not to upgrade.
Need to get 0 B/1,870 kB of archives.
After this operation, 8,251 kB of additional disk space will be used.
Selecting previously unselected package nvidia-l4t-bootloader.
(Reading database ... 165354 files and directories currently installed.)
Preparing to unpack .../nvidia-l4t-bootloader_32.4.3-20200924161615_arm64.deb ...
Unpacking nvidia-l4t-bootloader (32.4.3-20200924161615) ...
Setting up nvidia-l4t-bootloader (32.4.3-20200924161615) ...
3448-300---1--jetson-nano-qspi-sd-mmcblk0p1
Starting bootloader post-install procedure.
Update bootloader completed.
Reboot the target system for changes to take effect.
Updating extlinux.conf...
Root device is set in the extlinux.conf

$ sudo update-alternatives --install /usr/bin/python python /usr/bin/python2.7 2

update-alternatives: using /usr/bin/python2.7 to provide /usr/bin/python (python) in auto mode

$ sudo update-alternatives --set python /usr/bin/python2.7

$ sudo apt remove nvidia-l4t-bootloader

Reading package lists... Done

Building dependency tree

Reading state information... Done

Package 'nvidia-l4t-bootloader' is not installed, so not removed

0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.

$ sudo apt install nvidia-l4t-bootloader

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following NEW packages will be installed

nvidia-l4t-bootloader

0 to upgrade, 1 to newly install, 0 to remove and 0 not to upgrade.

Need to get 0 B/1,870 kB of archives.

After this operation, 8,251 kB of additional disk space will be used.

Selecting previously unselected package nvidia-l4t-bootloader.

(Reading database ... 165354 files and directories currently installed.)

Preparing to unpack .../nvidia-l4t-bootloader_32.4.3-20200924161615_arm64.deb ...

Unpacking nvidia-l4t-bootloader (32.4.3-20200924161615) ...

Setting up nvidia-l4t-bootloader (32.4.3-20200924161615) ...

3448-300---1--jetson-nano-qspi-sd-mmcblk0p1

Starting bootloader post-install procedure.

Update bootloader completed.

Reboot the target system for changes to take effect.

Updating extlinux.conf...

Root device is set in the extlinux.conf

Afterwards I set the default back to python 3.6 again.

Setup the Machine Learning

There are three Machine Learning packages, OpenCV for Computer Vison, Tensorflow for machine learning models, and TensorRT – accelerated deep learning networks for image recognition.

OpenCV is already installed in Python:

>>> import cv2
>>> print (cv2.__version__)
4.1.1

>>> import cv2

>>> print (cv2.__version__)

4.1.1

Tensorflow installation can be found here: https://docs.nvidia.com/deeplearning/frameworks/install-tf-jetson-platform/ I found this to be straight forward but it did keep the computer busy for a while.

>>> import tensorflow
2020-10-08 11:34:55.934735: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
>>> print (tensorflow.__version__)
2.3.0

>>> import tensorflow

2020-10-08 11:34:55.934735: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2

>>> print (tensorflow.__version__)

2.3.0

TensorRT can be found at: https://github.com/dusty-nv/jetson-inference/blob/master/docs/building-repo-2.md I used the instructions in the Quick Reference section and installed the default models without any errors, I have not yet had the opportunity to test it.

Copying and Resizing a Raspberry Pi SD Card

21st June 2020 Karl

The Micro SD Card I use in a Raspberry PI ran out of space so here is how I copied the contents of the 15GB drive to a new 64GB card and resized the partition. I used a separate computer running Debian and as the machine does not have a monitor or keyboard attached this will be being completed through the bash command line using SSH.

1. Making a copy of the SD Card.

Insert the old card into your computer, if the computer attempts to mount the drive then unmount it. We need to find which mount point has been used, do this with the lsblk command:

$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0   477G  0 disk
├─sda1   8:1    0   512M  0 part /boot/efi
├─sda2   8:2    0 468.9G  0 part /
└─sda3   8:3    0   7.6G  0 part [SWAP]
sdb      8:16   0   477G  0 disk
└─sdb1   8:17   0   477G  0 part /mnt/external_ssd
sdc      8:32   0   1.8T  0 disk
└─sdc1   8:33   0   1.8T  0 part /data
sdd      8:48   0   3.7T  0 disk
└─sdd1   8:49   0   3.7T  0 part /mnt/external_hdd
sde      8:64   1  14.9G  0 disk
├─sde1   8:65   1  43.8M  0 part
└─sde2   8:66   1  14.8G  0 part

$ lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

sda 8:0 0 477G 0 disk

├─sda1 8:1 0 512M 0 part /boot/efi

├─sda2 8:2 0 468.9G 0 part /

└─sda3 8:3 0 7.6G 0 part [SWAP]

sdb 8:16 0 477G 0 disk

└─sdb1 8:17 0 477G 0 part /mnt/external_ssd

sdc 8:32 0 1.8T 0 disk

└─sdc1 8:33 0 1.8T 0 part /data

sdd 8:48 0 3.7T 0 disk

└─sdd1 8:49 0 3.7T 0 part /mnt/external_hdd

sde 8:64 1 14.9G 0 disk

├─sde1 8:65 1 43.8M 0 part

└─sde2 8:66 1 14.8G 0 part

The device sde matches our SD card. So we will use that. The dd command is used to create the ISO image, I am creating the file in my home directory:

$ cd ~
$ sudo dd status=progress if=/dev/sde of=raspberry_cnc.iso

1 2	$ cd ~ $ sudo dd status=progress if=/dev/sde of=raspberry_cnc.iso

once complete, check that the records match:

31116288+0 records in
31116288+0 records out
15931539456 bytes (16 GB, 15 GiB) copied, 464.124 s, 34.3 MB/s

31116288+0 records in

31116288+0 records out

15931539456 bytes (16 GB, 15 GiB) copied, 464.124 s, 34.3 MB/s

Swap the SD cards and check the mount point again:

$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
[snip]
sde      8:64   1  59.5G  0 disk
└─sde1   8:65   1  59.5G  0 part

$ lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

[snip]

sde 8:64 1 59.5G 0 disk

└─sde1 8:65 1 59.5G 0 part

Now copy the iso image to the new card:

$ sudo dd status=progress if=raspberry_cnc.iso of=/dev/sde

1	$ sudo dd status=progress if=raspberry_cnc.iso of=/dev/sde

this can take a while, the write speed is rather slow:

31116288+0 records in
31116288+0 records out
15931539456 bytes (16 GB, 15 GiB) copied, 3023.22 s, 5.3 MB/s

31116288+0 records in

31116288+0 records out

15931539456 bytes (16 GB, 15 GiB) copied, 3023.22 s, 5.3 MB/s

and lsblk shows us the new partitions:

$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
[snip]
sde      8:64   1  59.5G  0 disk
├─sde1   8:65   1  43.8M  0 part
└─sde2   8:66   1  14.8G  0 part

$ lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

[snip]

sde 8:64 1 59.5G 0 disk

├─sde1 8:65 1 43.8M 0 part

└─sde2 8:66 1 14.8G 0 part

2. Resizing the partition

The copied partition is now the same size as the original. If you have space remaining, the new card can be put back in the Pi and use the raspi-config utility, and using the Expand Filesystem option in the Advanced Settings section. However if the drive is completely full you won’t be able to login as there won’t be enough space available for the temporary files created at login, to get round this you can use parted to resize, start with:

$ sudo parted /dev/sde

1	$ sudo parted /dev/sde

if you get the following message:

Warning: Unable to open /dev/sde read-write (Read-only file system).  /dev/sde has been opened read-only.

1	Warning: Unable to open /dev/sde read-write (Read-only file system). /dev/sde has been opened read-only.

then quit from parted, if you are using a full size SD Card, check the Write Protect tab on the side of the card and try again, otherwise try:

$ sudo hdparm -r0 /dev/sde

/dev/sde:
 setting readonly to 0 (off)
 readonly      =  0 (off)

$ sudo hdparm -r0 /dev/sde

/dev/sde:

setting readonly to 0 (off)

readonly = 0 (off)

If problem persists try formatting the new card in a camera, as these have a simple file system, and write the ISO image again.

We need to resize the larger partition /dev/sde2 with the ext4 file system, the smaller is used to boot the Pi and can be ignored. In parted, list the partitions with the print command:

$ sudo parted /dev/sde
(parted) print
Model: Generic- USB3.0 CRW -SD (scsi)
Disk /dev/sde: 63.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  50.1MB  45.9MB  primary  fat32        lba
 2      50.3MB  15.9GB  15.9GB  primary  ext4

$ sudo parted /dev/sde

(parted) print

Model: Generic- USB3.0 CRW -SD (scsi)

Disk /dev/sde: 63.9GB

Sector size (logical/physical): 512B/512B

Partition Table: msdos

Disk Flags:

Number Start End Size Type File system Flags

1 4194kB 50.1MB 45.9MB primary fat32 lba

2 50.3MB 15.9GB 15.9GB primary ext4

Using resizepart set the new size, I set this to larger than the 15GB but smaller than the unallocated space, this to save me having to accurately work out the remaining space manually:

(parted) resizepart 2
End?  [15.9GB]? 40GB

1 2	(parted) resizepart 2 End? [15.9GB]? 40GB

Now update the boundaries to grow and resize the partition into the freshly allocated space:

$ sudo /sbin/e2fsck -f /dev/sde2
$ sudo /sbin/resize2fs /dev/sde2

1 2	$ sudo /sbin/e2fsck -f /dev/sde2 $ sudo /sbin/resize2fs /dev/sde2

Now boot the Pi with the new card, login and use the raspi-config utility then in Advanced Options choose Expand Filesystem and follow the onscreen instructions. Once rebooted you should now be set to fill up your new card.

Links and Sources

RPi Resize Flash Partitions https://elinux.org/RPi_Resize_Flash_Partitions
Parted Guide https://www.tecmint.com/parted-command-to-create-resize-rescue-linux-disk-partitions/

Computer, linux, Programming

Extracting GPS data from the GoPro 7

28th July 2019 Karl 2 Comments

While extracting the telemetry data from the GoPro is reasonably well documented I have found some gaps for getting the extracting utilities installed and when extracting and combining data from multiple files. These notes are for a Debian/Ubuntu installation in a BASH Shell.

Installing the gopro-utils

As I couldn’t find any straightforward instructions for installation, I’ll be going through everything I needed to do to get it working, you may have some of these packages installed already.

sudo apt update
sudo apt upgrade
sudo apt install ffmpeg golang gpsbabel git

sudo apt update

sudo apt upgrade

sudo apt install ffmpeg golang gpsbabel git

Now to get the gopro-utils and install them, I’m placing the source files into my Downloads directory, as well as the GPS data extractor we’ll be adding the other telemetry tools too, this is all a bit long winded.

cd ~/Downloads
git clone https://github.com/JuanIrache/gopro-utils
cd gopro-utils
go get github.com/JuanIrache/gopro-utils/telemetry
go get github.com/mlouielu/gpxgo/gpx
cd bin/gopro2gpx
go build gopro2gpx.go
sudo cp gopro2gpx /usr/local/bin/
cd ../bin/gopro2geojson/
go build gopro2geojson.go
sudo cp gopro2geojson /usr/local/bin/
cd ../gopro2json/
go build gopro2json.go
sudo cp gopro2json /usr/local/bin/
cd ../gpmd2csv/
go build gpmd2csv.go
sudo cp gpmd2csv /usr/local/bin/
cd ../gpmd2info/
go build gpmd2info.go
sudo cp gpmd2info /usr/local/bin/
cd ../gps2kml/
go build gps2kml.go
sudo cp gps2kml /usr/local/bin/

cd ~/Downloads

git clone https://github.com/JuanIrache/gopro-utils

cd gopro-utils

go get github.com/JuanIrache/gopro-utils/telemetry

go get github.com/mlouielu/gpxgo/gpx

cd bin/gopro2gpx

go build gopro2gpx.go

sudo cp gopro2gpx /usr/local/bin/

cd ../bin/gopro2geojson/

go build gopro2geojson.go

sudo cp gopro2geojson /usr/local/bin/

cd ../gopro2json/

go build gopro2json.go

sudo cp gopro2json /usr/local/bin/

cd ../gpmd2csv/

go build gpmd2csv.go

sudo cp gpmd2csv /usr/local/bin/

cd ../gpmd2info/

go build gpmd2info.go

sudo cp gpmd2info /usr/local/bin/

cd ../gps2kml/

go build gps2kml.go

sudo cp gps2kml /usr/local/bin/

Extracting the Data

You will need to find which stream in the video recording the data has been saved to, to find this use ffprobe to examine the recording and look for the stream that contains GoPro MET, for example:

$ ffprobe GX090155.MP4
[snip]
    Stream #0:3(eng): Data: none (gpmd / 0x646D7067), 40 kb/s (default)
    Metadata:
      creation_time   : 2019-07-28T11:09:25.000000Z
      handler_name    : GoPro MET
[snip]

$ ffprobe GX090155.MP4

[snip]

Stream #0:3(eng): Data: none (gpmd / 0x646D7067), 40 kb/s (default)

Metadata:

creation_time : 2019-07-28T11:09:25.000000Z

handler_name : GoPro MET

[snip]

You can see that what we are wanting is on stream 3, as far as I can tell this stays the same every time, I don’t know if it is different for other GoPro models.

This bash script extracts the GPS data in GPX format from all the GoPro GX recordings in the directory, other options have been commented out, if you are using Garmin VIRB edit there is also an option for use with that. The script creates two files, one that contains the raw data and another with the desired GPS data, the GPS output file has the same name as the recording, but in lowercase with a .gpx extension.

#!/bin/bash
FILES="GX*.MP4"
STREAM="0:3"
for MP4FILE in $FILES; do
   OUTFILE=$(basename "$MP4FILE" | cut -d. -f1)
   
   OUTFILE=$(echo "$OUTFILE" | tr '[:upper:]' '[:lower:]')
   BINFILE="$OUTFILE.bin"
   ffmpeg -y -i "$MP4FILE" -codec copy -map $STREAM -f rawvideo "$BINFILE"

   #gpmd2csv -i "$BINFILE" -o "$OUTFILE.csv"
   gopro2gpx -i "$BINFILE" -a 500 -f 2 -o "$OUTFILE.gpx"
   #gopro2gpx -i "$BINFILE" -a 200 -f 3 -o "$OUTFILE-virb.gpx"
   #gopro2json -i "$BINFILE" -o "$OUTFILE.json"
done

#!/bin/bash

FILES="GX*.MP4"

STREAM="0:3"

for MP4FILE in $FILES; do

OUTFILE=$(basename "$MP4FILE" | cut -d. -f1)

OUTFILE=$(echo "$OUTFILE" | tr '[:upper:]' '[:lower:]')

BINFILE="$OUTFILE.bin"

ffmpeg -y -i "$MP4FILE" -codec copy -map $STREAM -f rawvideo "$BINFILE"

#gpmd2csv -i "$BINFILE" -o "$OUTFILE.csv"

gopro2gpx -i "$BINFILE" -a 500 -f 2 -o "$OUTFILE.gpx"

#gopro2gpx -i "$BINFILE" -a 200 -f 3 -o "$OUTFILE-virb.gpx"

#gopro2json -i "$BINFILE" -o "$OUTFILE.json"

done

Merging GPX files

As the GoPro splits recordings into 4GB blocks, when extracting you will get a single GPX file for each recording. Many pages found by Google say that to create a single track from these all you need to do is append the files into one big file. This is wrong, what you end up with is a single file with many short tracks, when what you are after is one long track covering the entire journey. This bash script uses gpsbabel to create single merged file from the extracted GPX data, it creates a file called “gpsoutput.gpx”.

#!/bin/bash
FILES="gx*.gpx"
ff=""
for f in $FILES; do
    ff="$ff -f $f"
done
gpsbabel -t -i gpx $ff -o gpx -x track,merge,title="GOPRO COMBINED LOG"  -F gpsoutpt.gpx

#!/bin/bash

FILES="gx*.gpx"

ff=""

for f in $FILES; do

ff="$ff -f $f"

done

gpsbabel -t -i gpx $ff -o gpx -x track,merge,title="GOPRO COMBINED LOG" -F gpsoutpt.gpx

The next stage will be to write a script that combines all these and completes the job in one easy process.

Links and Sources

Web Page to extract GoPro data: https://tailorandwayne.com/gopro-telemetry-extractor/ Easy to use for individual files
https://github.com/JuanIrache/gopro-utils
GoPro Forum Extracting the metadata in a useful format
GPSBabel, Free software for GPS data conversion and transfer: https://www.gpsbabel.org/

Setup The PI

Setup Windows for SSH

Setup VS Code

Links and Sources

At the Console

Installing the LAMP service – MySql, Apache and PHP

Accessing a Network Drive

Other Useful Things

Links and Sources

Default Settings

Resetting the SPA122 and Gaining Access

Upgrade the Firmware

Network Configuration

Telephone Configuration

Setting the Tones

My Work Here is Done

Links and Sources

Installation

Configuration

Connecting

Using OpenCV

Links and Sources

Initial Startup

Customising the Setup

Post Install Problems

Setup the Machine Learning

Further Reading

1. Making a copy of the SD Card.

2. Resizing the partition

Links and Sources

Installing the gopro-utils

Extracting the Data

Merging GPX files

Links and Sources

COMPUTERS, ROBOTS AND PHOTOGRAPHY