Category Archives: Programming

Converting a UTC date time to local in PHP

2nd April 2023 Karl Leave a comment

Here is a thing that took me longer to work out than it should have. I am storing some dates in a MySql database in the UTC timezone, but on my webpage I want to have them show as the local time. The problem I found is that PHP assumed that the date from the database was already in the local time (BST) and so did not adjust the hour on the datetime object when the timezone was applied.

This is fixed by setting the timezone when creating the datetime object from the string:

<?php
// this goes at the start of your script, in the index.php
date_default_timezone_set('Europe/London');

// create the datetime object, and set the timezone to UTC:
$sqlDate = "2023-04-02 10:40:36";
$datetime = DateTime::createFromFormat('Y-m-d H:i:s', $sqlDate, new DateTimeZone('UTC'));

// displays: 02 Apr 2023 10:40:36 UTC
echo "<p>". $datetime->format('d M Y H:i:s T') ."</p>";

// now set the timezone:
$datetime->setTimezone(new DateTimeZone(date_default_timezone_get()));

// displays: 02 Apr 2023 11:40:36 BST
echo "<p>". $datetime->format('d M Y H:i:s T') ."</p>";

// format for output:
$output = $datetime->format('d F Y H:i:s T');
echo "<p>".$output."</p>"; // 02 April 2023 11:40:36 BST
?>

<?php

// this goes at the start of your script, in the index.php

date_default_timezone_set('Europe/London');

// create the datetime object, and set the timezone to UTC:

$sqlDate = "2023-04-02 10:40:36";

$datetime = DateTime::createFromFormat('Y-m-d H:i:s', $sqlDate, new DateTimeZone('UTC'));

// displays: 02 Apr 2023 10:40:36 UTC

echo "". $datetime->format('d M Y H:i:s T') ."";

// now set the timezone:

$datetime->setTimezone(new DateTimeZone(date_default_timezone_get()));

// displays: 02 Apr 2023 11:40:36 BST

echo "". $datetime->format('d M Y H:i:s T') ."";

// format for output:

$output = $datetime->format('d F Y H:i:s T');

echo "".$output.""; // 02 April 2023 11:40:36 BST

Also here it is again, but with a Unix Timestamp:

<?php
// this goes at the start of your script, in the index.php
date_default_timezone_set('Europe/London');

// create the datetime object, and set the timezone to UTC:
$unixtime = 1680428135;
$datetime = new DateTime("@$unixtime", new DateTimeZone('UTC'));

// displays: 02 Apr 2023 09:35:35 GMT+0000
echo "<p>". $datetime->format('d M Y H:i:s T') ."</p>";

// now set the timezone:
$datetime->setTimezone(new DateTimeZone(date_default_timezone_get()));

// displays: 02 Apr 2023 10:35:35 BST
echo "<p>". $datetime->format('d M Y H:i:s T') ."</p>";

// format for output:
$output = $datetime->format('d F Y H:i:s T');
echo "<p>".$output."</p>"; // 02 April 2023 10:35:35 BST
?>

<?php

// this goes at the start of your script, in the index.php

date_default_timezone_set('Europe/London');

// create the datetime object, and set the timezone to UTC:

$unixtime = 1680428135;

$datetime = new DateTime("@$unixtime", new DateTimeZone('UTC'));

// displays: 02 Apr 2023 09:35:35 GMT+0000

echo "". $datetime->format('d M Y H:i:s T') ."";

// now set the timezone:

$datetime->setTimezone(new DateTimeZone(date_default_timezone_get()));

// displays: 02 Apr 2023 10:35:35 BST

echo "". $datetime->format('d M Y H:i:s T') ."";

// format for output:

$output = $datetime->format('d F Y H:i:s T');

echo "".$output.""; // 02 April 2023 10:35:35 BST

I hope this saves someone half a day. Although it occurs to me that I should really have the PHP output the date as the UTC and then have the javascript deal with it. Then the date would be displayed as how the users timezone is set rather than the servers.

So, in our PHP code, we setup a suitable output:

<?php
// code snippet
$sqlDate = "2023-04-02 11:55:37";

$data = array();
$data["importantDate"] = setISO8601date($sqlDate);

header("HTTP/1.1 200 OK");
header('Content-Type: application/json; charset=utf-8');
echo json_encode($data);
exit();

function setISO8601date($sqlDate) {
   $datetime = DateTime::createFromFormat('Y-m-d H:i:s', $sqlDate, new DateTimeZone('UTC'));
   return $datetime->format("Y-m-d\TH:i:s.Z\Z"); 
}
?>

<?php

// code snippet

$sqlDate = "2023-04-02 11:55:37";

$data = array();

$data["importantDate"] = setISO8601date($sqlDate);

header("HTTP/1.1 200 OK");

header('Content-Type: application/json; charset=utf-8');

echo json_encode($data);

exit();

function setISO8601date($sqlDate) {

$datetime = DateTime::createFromFormat('Y-m-d H:i:s', $sqlDate, new DateTimeZone('UTC'));

return $datetime->format("Y-m-d\TH:i:s.Z\Z");

}

and in your javascript file, after you have picked up the data with your ajax call, or whatever, I have two functions, one to convert the ISO 8601 format (YYYY-MM-DDTHH:mm:ss.sssZ) UTC date time to local and another to make for a nicely formatted output:

// ... javascript code snippet ...
// using jquery to fill the html void

$("#importantdate").html(unixtimestampToLocaldate(data["importantDate"]));

function unixtimestampToLocaldate(utcDateStr) {

  const unixtimestamp = Date.parse(utcDateStr);
  var dd = new Date(unixtimestamp);  // this date is adjusted to local time

  console.log(ut);        // 2023-04-02T00:21:46.000Z
  console.log(unixtime);  // 1680394906000
  console.log(dd);        // Sun Apr 02 2023 01:21:46 GMT+0100 (British Summer Time)
  return datetimeToStr(dd, true); 
}

// for some reason the native javascript date functions have very poor options for formatting 
// without you having to write lots of extra code.
function datetimeToStr(datetime, withtime=false) {

  const days = ["Sun", "Mon", "Tue", "Wed", "Thr", "Fri", "Sat", "Sun"];
  const months = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"];

  var yy = datetime.getFullYear();
  var mm = datetime.getMonth();
  var dt = datetime.getDate();
  var dd = datetime.getDay();
  var hh = datetime.getHours().toString();
  var ii = datetime.getMinutes().toString();
  var ss = datetime.getSeconds().toString();

  var date = days[dd]+" "+dt+" "+ months[mm] +" "+yy;
  if (withtime) {
    return date + " "+ hh.padStart(2, '0') + ":"+ii.padStart(2, '0')+":"+ss.padStart(2, '0');
    // Sun 2 Apr 2023 01:21:46
  }
  return date;  // Sun 2 Apr 2023
}

// ... javascript code snippet ...

// using jquery to fill the html void

$("#importantdate").html(unixtimestampToLocaldate(data["importantDate"]));

function unixtimestampToLocaldate(utcDateStr) {

const unixtimestamp = Date.parse(utcDateStr);

var dd = new Date(unixtimestamp); // this date is adjusted to local time

console.log(ut); // 2023-04-02T00:21:46.000Z

console.log(unixtime); // 1680394906000

console.log(dd); // Sun Apr 02 2023 01:21:46 GMT+0100 (British Summer Time)

return datetimeToStr(dd, true);

}

// for some reason the native javascript date functions have very poor options for formatting

// without you having to write lots of extra code.

function datetimeToStr(datetime, withtime=false) {

const days = ["Sun", "Mon", "Tue", "Wed", "Thr", "Fri", "Sat", "Sun"];

const months = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"];

var yy = datetime.getFullYear();

var mm = datetime.getMonth();

var dt = datetime.getDate();

var dd = datetime.getDay();

var hh = datetime.getHours().toString();

var ii = datetime.getMinutes().toString();

var ss = datetime.getSeconds().toString();

var date = days[dd]+" "+dt+" "+ months[mm] +" "+yy;

if (withtime) {

return date + " "+ hh.padStart(2, '0') + ":"+ii.padStart(2, '0')+":"+ss.padStart(2, '0');

// Sun 2 Apr 2023 01:21:46

}

return date; // Sun 2 Apr 2023

}

I’ve created the separate function for the date formatting as this has more flexibility.

Links and Sources

PHP Datetime Docs: https://www.php.net/manual/en/book.datetime.php
MDN Web Docs: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date

Interesting, Programming

Karl’s Roast Calculator – Updated

18th November 2020 Karl

I’ve updated my Roast Calculator to make it work clearer with mobile devices. Use it to find the cooking time of your favorite meat roast.

Link: https://www.g7smy.co.uk/roast

Computer, Jetson, linux, Programming

Jetson Nano Install Notes

8th October 2020 Karl

The Jetson Nano Developer kit – B01 is a small computer comprising of an NVIDA Maxwell GPU, Quad-Core ARM Cortex-A57 Processor and 4GB of Memory along with four USB 3 ports, Gigabit Ethernet, HDMI and Display Port output, main storage is on a MicroSD card and there is a variety of selection of expansion available via GPIO, I2C and UART. On the software side NVIDIA provide their JetPack SDK – a customised version of Ubuntu. This development kit has been produced to provide an entry point into Machine Learning, for which I will be using Python programming language. I got my board from Pimoroni

GPU	128-core Maxwell
CPU	Quad-core ARM A57 @ 1.43 GHz
Memory	4 GB 64-bit LPDDR4 25.6 GB/s
Storage	microSD (not included)
Video Encode	4K @ 30 \| 4x 1080p @ 30 \| 9x 720p @ 30 (H.264/H.265)
Video Decode	4K @ 60 \| 2x 4K @ 30 \| 8x 1080p @ 30 \| 18x 720p @ 30 (H.264/H.265)
Camera	2x MIPI CSI-2 DPHY lanes
Connectivity	Gigabit Ethernet, M.2 Key E
Display	HDMI and display port
USB	4x USB 3.0, USB 2.0 Micro-B
Others	GPIO, I2C, I2S, SPI, UART
Mechanical	69 mm x 45 mm, 260-pin edge connector

These notes cover my process of setting one up and links to the documentation, it not intended to repeat those install guides but to provide an install sequence and any additional commentary as needed. I’m going to assume you have a little experience of using the terminal and am familiar with using the bash command line – I’ve no idea how this would be done through the GUI.

jetson nano ports — Jetson Nano ports [source: NVIDIA]

Initial Startup

I followed the instructions for downloading and installing JetPack 4.4 on https://nvidia.com/jetsonnano-start I used a 64GB Class 10, UHS-I, U3, V30 SanDisk card. I formatted the card in a camera before using balenaEtcher to write the JetPack SDK image, this creates a partition of about 16GB on the card formatted to ext4, during installation the volume is resized to fill the card.

jetson nano connections — Jetson Nano connections [source: NVIDIA]

Despite using a good quality USB power supply with an output of 3 Amps at 5 Volts into the Micro USB port the computer would only boot long enough for the NVIDIA logo to appear on screen but after a few seconds the green power LED would go out and it would be off, the same happened when I tried a variety of USB power supplies used. I got round the problem buy using a 5 Amp power supply connected to the barrel jack J25 on the left (centre pin positive) and connecting the jumper J48 located just behind this connector.

Customising the Setup

There are a couple of thigs to do, get a network volume mounted and set the default version of python.

For the network share install samba, some network utilities and the nano text editor:

$ sudo apt install samba cifs-utils nano

1	$ sudo apt install samba cifs-utils nano

create a text file: sudo nano /etc/samaba/videoserver with the following:

username=<your network username>
password=<your network password>

1 2	username=<your network username> password=<your network password>

And set the permissions sudo chmod 600 /etc/samba/videoserver. In this example I have a network share on my server; 192.168.1.30, called video. Create a mount point for the share: sudo mkdir /mnt/video now you need to edit fstab, sudo nano /etc/fstab and add your network connection to the end:

//192.168.1.30/video /mnt/video cifs credentials=/etc/samba/videoserver,noauto,x-systemd.automount,_netdev,uid=0,vers=1.0 0 0

1	//192.168.1.30/video /mnt/video cifs credentials=/etc/samba/videoserver,noauto,x-systemd.automount,_netdev,uid=0,vers=1.0 0 0

Reload fstab with sudo mount -a and check for any errors. Because of the way that Jetpack boots it does not appear to wait for the network so the share needs to be set to automount and this causes it to only appear in drive listings when accessed. Further reading can be found in this excellent guide to fstab: https://wiki.archlinux.org/index.php/fstab.

Jetpack 4.4 comes with two versions of Python, 2.7 and 3.6, I want it to default to 3.6 and while this is rather out of date I don’t want to go down the hole of upgrading just yet, you will also need to install pip and set pip3 as the default too.

$ sudo apt install python3-pip
$ sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.6 1
$ sudo update-alternatives --install /usr/bin/pip pip /usr/bin/pip3.6 1
$ sudo update-alternatives --set python /usr/bin/python3.6
$ sudo update-alternatives --set pip /usr/bin/pip3.6

$ sudo apt install python3-pip

$ sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.6 1

$ sudo update-alternatives --install /usr/bin/pip pip /usr/bin/pip3.6 1

$ sudo update-alternatives --set python /usr/bin/python3.6

$ sudo update-alternatives --set pip /usr/bin/pip3.6

and test its worked:

$ python
Python 3.6.9 (default, Jul 17 2020, 12:50:27)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

$ python

Python 3.6.9 (default, Jul 17 2020, 12:50:27)

[GCC 8.4.0] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>>

$ pip --version
pip 20.2.3 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)

1 2	$ pip --version pip 20.2.3 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)

I did get an error later on, a crash was reported on the desktop when an occasional python 2 script ran. I fixed the error in /usr/sbin/l4t_payload_updater_t210 by changing the first line of the file from !#/user/bin/python to !#/user/bin/python2

Post Install Problems

A recent update occured, so I did the usual sudo apt get update && sudo apt get upgrade but one of the files gave a script error, this turned out to be with nvidia-l4t-bootloader, like so:

$ sudo apt install nvidia-l4t-bootloader
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed
  nvidia-l4t-bootloader
0 to upgrade, 1 to newly install, 0 to remove and 0 not to upgrade.
Need to get 1,870 kB of archives.
After this operation, 8,251 kB of additional disk space will be used.
Get:1 https://repo.download.nvidia.com/jetson/t210 r32.4/main arm64 nvidia-l4t-bootloader arm64 32.4.3-20200924161615 [1,870 kB]
Fetched 1,870 kB in 3s (692 kB/s)
Selecting previously unselected package nvidia-l4t-bootloader.
(Reading database ... 165354 files and directories currently installed.)
Preparing to unpack .../nvidia-l4t-bootloader_32.4.3-20200924161615_arm64.deb ...
Unpacking nvidia-l4t-bootloader (32.4.3-20200924161615) ...
Setting up nvidia-l4t-bootloader (32.4.3-20200924161615) ...
3448-300---1--jetson-nano-qspi-sd-mmcblk0p1
Starting bootloader post-install procedure.
ERROR. Procedure for bootloader update FAILED.
Cannot install package. Exiting...
dpkg: error processing package nvidia-l4t-bootloader (--configure):
 installed nvidia-l4t-bootloader package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 nvidia-l4t-bootloader
E: Sub-process /usr/bin/dpkg returned an error code (1)

$ sudo apt install nvidia-l4t-bootloader

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following NEW packages will be installed

nvidia-l4t-bootloader

0 to upgrade, 1 to newly install, 0 to remove and 0 not to upgrade.

Need to get 1,870 kB of archives.

After this operation, 8,251 kB of additional disk space will be used.

Get:1 https://repo.download.nvidia.com/jetson/t210 r32.4/main arm64 nvidia-l4t-bootloader arm64 32.4.3-20200924161615 [1,870 kB]

Fetched 1,870 kB in 3s (692 kB/s)

Selecting previously unselected package nvidia-l4t-bootloader.

(Reading database ... 165354 files and directories currently installed.)

Preparing to unpack .../nvidia-l4t-bootloader_32.4.3-20200924161615_arm64.deb ...

Unpacking nvidia-l4t-bootloader (32.4.3-20200924161615) ...

Setting up nvidia-l4t-bootloader (32.4.3-20200924161615) ...

3448-300---1--jetson-nano-qspi-sd-mmcblk0p1

Starting bootloader post-install procedure.

ERROR. Procedure for bootloader update FAILED.

Cannot install package. Exiting...

dpkg: error processing package nvidia-l4t-bootloader (--configure):

installed nvidia-l4t-bootloader package post-installation script subprocess returned error exit status 1

Errors were encountered while processing:

nvidia-l4t-bootloader

E: Sub-process /usr/bin/dpkg returned an error code (1)

after some head scratching, it turns out this script is python2 only. I reverted back to the python 2.7 as the default, and tried again:

$ sudo update-alternatives --install /usr/bin/python python /usr/bin/python2.7 2
update-alternatives: using /usr/bin/python2.7 to provide /usr/bin/python (python) in auto mode
$ sudo update-alternatives --set python /usr/bin/python2.7
$ sudo apt remove nvidia-l4t-bootloader
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package 'nvidia-l4t-bootloader' is not installed, so not removed
0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.
$ sudo apt install nvidia-l4t-bootloader
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed
  nvidia-l4t-bootloader
0 to upgrade, 1 to newly install, 0 to remove and 0 not to upgrade.
Need to get 0 B/1,870 kB of archives.
After this operation, 8,251 kB of additional disk space will be used.
Selecting previously unselected package nvidia-l4t-bootloader.
(Reading database ... 165354 files and directories currently installed.)
Preparing to unpack .../nvidia-l4t-bootloader_32.4.3-20200924161615_arm64.deb ...
Unpacking nvidia-l4t-bootloader (32.4.3-20200924161615) ...
Setting up nvidia-l4t-bootloader (32.4.3-20200924161615) ...
3448-300---1--jetson-nano-qspi-sd-mmcblk0p1
Starting bootloader post-install procedure.
Update bootloader completed.
Reboot the target system for changes to take effect.
Updating extlinux.conf...
Root device is set in the extlinux.conf

$ sudo update-alternatives --install /usr/bin/python python /usr/bin/python2.7 2

update-alternatives: using /usr/bin/python2.7 to provide /usr/bin/python (python) in auto mode

$ sudo update-alternatives --set python /usr/bin/python2.7

$ sudo apt remove nvidia-l4t-bootloader

Reading package lists... Done

Building dependency tree

Reading state information... Done

Package 'nvidia-l4t-bootloader' is not installed, so not removed

0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.

$ sudo apt install nvidia-l4t-bootloader

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following NEW packages will be installed

nvidia-l4t-bootloader

0 to upgrade, 1 to newly install, 0 to remove and 0 not to upgrade.

Need to get 0 B/1,870 kB of archives.

After this operation, 8,251 kB of additional disk space will be used.

Selecting previously unselected package nvidia-l4t-bootloader.

(Reading database ... 165354 files and directories currently installed.)

Preparing to unpack .../nvidia-l4t-bootloader_32.4.3-20200924161615_arm64.deb ...

Unpacking nvidia-l4t-bootloader (32.4.3-20200924161615) ...

Setting up nvidia-l4t-bootloader (32.4.3-20200924161615) ...

3448-300---1--jetson-nano-qspi-sd-mmcblk0p1

Starting bootloader post-install procedure.

Update bootloader completed.

Reboot the target system for changes to take effect.

Updating extlinux.conf...

Root device is set in the extlinux.conf

Afterwards I set the default back to python 3.6 again.

Setup the Machine Learning

There are three Machine Learning packages, OpenCV for Computer Vison, Tensorflow for machine learning models, and TensorRT – accelerated deep learning networks for image recognition.

OpenCV is already installed in Python:

>>> import cv2
>>> print (cv2.__version__)
4.1.1

>>> import cv2

>>> print (cv2.__version__)

4.1.1

Tensorflow installation can be found here: https://docs.nvidia.com/deeplearning/frameworks/install-tf-jetson-platform/ I found this to be straight forward but it did keep the computer busy for a while.

>>> import tensorflow
2020-10-08 11:34:55.934735: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
>>> print (tensorflow.__version__)
2.3.0

>>> import tensorflow

2020-10-08 11:34:55.934735: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2

>>> print (tensorflow.__version__)

2.3.0

TensorRT can be found at: https://github.com/dusty-nv/jetson-inference/blob/master/docs/building-repo-2.md I used the instructions in the Quick Reference section and installed the default models without any errors, I have not yet had the opportunity to test it.

Extracting GPS data from the GoPro 7

28th July 2019 Karl 2 Comments

While extracting the telemetry data from the GoPro is reasonably well documented I have found some gaps for getting the extracting utilities installed and when extracting and combining data from multiple files. These notes are for a Debian/Ubuntu installation in a BASH Shell.

Installing the gopro-utils

As I couldn’t find any straightforward instructions for installation, I’ll be going through everything I needed to do to get it working, you may have some of these packages installed already.

sudo apt update
sudo apt upgrade
sudo apt install ffmpeg golang gpsbabel git

sudo apt update

sudo apt upgrade

sudo apt install ffmpeg golang gpsbabel git

Now to get the gopro-utils and install them, I’m placing the source files into my Downloads directory, as well as the GPS data extractor we’ll be adding the other telemetry tools too, this is all a bit long winded.

cd ~/Downloads
git clone https://github.com/JuanIrache/gopro-utils
cd gopro-utils
go get github.com/JuanIrache/gopro-utils/telemetry
go get github.com/mlouielu/gpxgo/gpx
cd bin/gopro2gpx
go build gopro2gpx.go
sudo cp gopro2gpx /usr/local/bin/
cd ../bin/gopro2geojson/
go build gopro2geojson.go
sudo cp gopro2geojson /usr/local/bin/
cd ../gopro2json/
go build gopro2json.go
sudo cp gopro2json /usr/local/bin/
cd ../gpmd2csv/
go build gpmd2csv.go
sudo cp gpmd2csv /usr/local/bin/
cd ../gpmd2info/
go build gpmd2info.go
sudo cp gpmd2info /usr/local/bin/
cd ../gps2kml/
go build gps2kml.go
sudo cp gps2kml /usr/local/bin/

cd ~/Downloads

git clone https://github.com/JuanIrache/gopro-utils

cd gopro-utils

go get github.com/JuanIrache/gopro-utils/telemetry

go get github.com/mlouielu/gpxgo/gpx

cd bin/gopro2gpx

go build gopro2gpx.go

sudo cp gopro2gpx /usr/local/bin/

cd ../bin/gopro2geojson/

go build gopro2geojson.go

sudo cp gopro2geojson /usr/local/bin/

cd ../gopro2json/

go build gopro2json.go

sudo cp gopro2json /usr/local/bin/

cd ../gpmd2csv/

go build gpmd2csv.go

sudo cp gpmd2csv /usr/local/bin/

cd ../gpmd2info/

go build gpmd2info.go

sudo cp gpmd2info /usr/local/bin/

cd ../gps2kml/

go build gps2kml.go

sudo cp gps2kml /usr/local/bin/

Extracting the Data

You will need to find which stream in the video recording the data has been saved to, to find this use ffprobe to examine the recording and look for the stream that contains GoPro MET, for example:

$ ffprobe GX090155.MP4
[snip]
    Stream #0:3(eng): Data: none (gpmd / 0x646D7067), 40 kb/s (default)
    Metadata:
      creation_time   : 2019-07-28T11:09:25.000000Z
      handler_name    : GoPro MET
[snip]

$ ffprobe GX090155.MP4

[snip]

Stream #0:3(eng): Data: none (gpmd / 0x646D7067), 40 kb/s (default)

Metadata:

creation_time : 2019-07-28T11:09:25.000000Z

handler_name : GoPro MET

[snip]

You can see that what we are wanting is on stream 3, as far as I can tell this stays the same every time, I don’t know if it is different for other GoPro models.

This bash script extracts the GPS data in GPX format from all the GoPro GX recordings in the directory, other options have been commented out, if you are using Garmin VIRB edit there is also an option for use with that. The script creates two files, one that contains the raw data and another with the desired GPS data, the GPS output file has the same name as the recording, but in lowercase with a .gpx extension.

#!/bin/bash
FILES="GX*.MP4"
STREAM="0:3"
for MP4FILE in $FILES; do
   OUTFILE=$(basename "$MP4FILE" | cut -d. -f1)
   
   OUTFILE=$(echo "$OUTFILE" | tr '[:upper:]' '[:lower:]')
   BINFILE="$OUTFILE.bin"
   ffmpeg -y -i "$MP4FILE" -codec copy -map $STREAM -f rawvideo "$BINFILE"

   #gpmd2csv -i "$BINFILE" -o "$OUTFILE.csv"
   gopro2gpx -i "$BINFILE" -a 500 -f 2 -o "$OUTFILE.gpx"
   #gopro2gpx -i "$BINFILE" -a 200 -f 3 -o "$OUTFILE-virb.gpx"
   #gopro2json -i "$BINFILE" -o "$OUTFILE.json"
done

#!/bin/bash

FILES="GX*.MP4"

STREAM="0:3"

for MP4FILE in $FILES; do

OUTFILE=$(basename "$MP4FILE" | cut -d. -f1)

OUTFILE=$(echo "$OUTFILE" | tr '[:upper:]' '[:lower:]')

BINFILE="$OUTFILE.bin"

ffmpeg -y -i "$MP4FILE" -codec copy -map $STREAM -f rawvideo "$BINFILE"

#gpmd2csv -i "$BINFILE" -o "$OUTFILE.csv"

gopro2gpx -i "$BINFILE" -a 500 -f 2 -o "$OUTFILE.gpx"

#gopro2gpx -i "$BINFILE" -a 200 -f 3 -o "$OUTFILE-virb.gpx"

#gopro2json -i "$BINFILE" -o "$OUTFILE.json"

done

Merging GPX files

As the GoPro splits recordings into 4GB blocks, when extracting you will get a single GPX file for each recording. Many pages found by Google say that to create a single track from these all you need to do is append the files into one big file. This is wrong, what you end up with is a single file with many short tracks, when what you are after is one long track covering the entire journey. This bash script uses gpsbabel to create single merged file from the extracted GPX data, it creates a file called “gpsoutput.gpx”.

#!/bin/bash
FILES="gx*.gpx"
ff=""
for f in $FILES; do
    ff="$ff -f $f"
done
gpsbabel -t -i gpx $ff -o gpx -x track,merge,title="GOPRO COMBINED LOG"  -F gpsoutpt.gpx

#!/bin/bash

FILES="gx*.gpx"

ff=""

for f in $FILES; do

ff="$ff -f $f"

done

gpsbabel -t -i gpx $ff -o gpx -x track,merge,title="GOPRO COMBINED LOG" -F gpsoutpt.gpx

The next stage will be to write a script that combines all these and completes the job in one easy process.

Links and Sources

Web Page to extract GoPro data: https://tailorandwayne.com/gopro-telemetry-extractor/ Easy to use for individual files
https://github.com/JuanIrache/gopro-utils
GoPro Forum Extracting the metadata in a useful format
GPSBabel, Free software for GPS data conversion and transfer: https://www.gpsbabel.org/

Arduino, Electronics, Hardware, Programming

The Fridge Door is Open

29th August 2018 Karl

My fridge door tends to rebound when closed staying open a smidgen and letting all the cold out. Rather than just checking that the door is properly shut, I thought it about time to have a microcontroller make a noise when the door has been left open too long.

This circuit uses an Arduino compatible Teensy LC for all the work, it has a phototransistor to sense the state of the fridge door light, a couple of LED’s one to indicate the power and another that comes on when the door is open. There is also Piezo buzzer to make an annoying noise after forty five seconds of door open time. The unit runs of a 3.7v rechargeable Lithium-ion battery and I have added a recharging circuit that takes power via the Teensy’s 5v USB port.

Circuitry

Note: These diagrams show 3.7v as the supply voltage. The Teensy LC can only tolerate a maximum 3.3V on the data pins, so these circuits are driven from the 3V output on the Teensy. They will all work without modification on the 5v Arduino Uno.

I have used a phototransistor to detect the fridge door light, there are two variants of this circuit light activated or dark activated, the 100k resistor can be replaced with a 100k variable if you need to adjust the sensitivity, the 330k resistor provides a weak pull-down on the output. The phototransistor is being used in switch mode to provide a logical output (rather than active mode which provides an output relative to the amount of light), so the output is connected to a digital input on the Arduino. The BC547 transistor is half of the darlington pair to provide extra gain on the output.

I chose the light activated switch, either will do but will provide different logical outputs to your controller. The circuit is enclosed in a small box inside the fridge and connected by ribbon cable to the controller, the ribbon cable is flat and does not upset the magnetic ‘seal’ on the fridge door.

To make some noise I used a piezo buzzer from an old computer, this is driven through a transistor as the Teensy does not provide enough current to drive it directly.

There is also a push button to provide a reset function if the buzzer is sounding while the door is open.

I have also added two LED’s, one to show power and anther that illuminates when the door is opened.

The final circuit if for recharging the battery, it connects to the 5V connection on the Teensy LC so charges the battery when the USB connection is in use. This has been copied from the MCP3831T datasheet.

Software

This uses an interrupt to listen for the light sensor, when the state changes, the door open pin is read to determine if the door is open or not. If it is then a timer is started, this gives you forty five seconds to complete your task before the alarm sounds. With the door closed the timer is stopped and set back to zero. If the sounder goes off while you are rummaging in the fridge the reset button can be pressed, this restarts the timer from zero again.

#define TEST_LED 13
#define FRIDGE_DOOR 6
#define DOOR_LED 8
#define POWER_LED 7
#define RESET_BTN 9
#define BUZZER 10

volatile boolean doorOpen = true;   // when the light is on
volatile boolean doorTrigger = false;
volatile unsigned long lastFridgeMillis = 0;
volatile unsigned long fridgeTimeoutMillis = 10;

boolean SerialDebug = false;
unsigned long doorOpenDelay = 45000; // 45 second timeout before buzzer sounds
unsigned long doorOpenTime = 0;
unsigned long startTime;
unsigned long previousToneMillis = 0;
unsigned long toneInterval = 20;
boolean soundTheAlarm = false;
boolean outputTone = false;
int toneMelody[5] = { 1600, 1900, 2000, 2100, 1800 };
const int toneMax = 5; 

unsigned long lastDoorOpenTime = 1;
char pout[80]; // for testing

// interrupt function
void fridgeDoor() {
  if ((millis() - lastFridgeMillis) > fridgeTimeoutMillis) {
    doorOpen = digitalRead(FRIDGE_DOOR); // goes high when dark
    lastFridgeMillis = millis();
  }
}

// see if the door is open or closed
void fridgeDoorOpen() {
  if (doorOpen == true && doorTrigger == false) {
    doorTrigger = true;
    startTime = millis();

    if (SerialDebug) {
      sprintf(pout, "door open: %d", doorOpen);
      Serial.println(pout);
    }
    digitalWrite(DOOR_LED, HIGH);
    return;
  }

  if (doorOpen == false && doorTrigger == true) {
    doorTrigger = false;

    if (SerialDebug) {
      sprintf(pout, "door closed: %d", doorOpen);
      Serial.println(pout);
    }
    startTime = 0;
    doorOpenTime = 0;
    soundTheAlarm = false;
    digitalWrite(DOOR_LED, LOW);
    return;
  }

}

// make some noise
void soundAlarm() {

  if (soundTheAlarm == false) {
    return;
  }

  int ii = 0;
  while (ii < 6) {
    if (soundTheAlarm == false) {
      break;
    }
    int i = 0;
    while (i < toneMax) {
      if (soundTheAlarm == false) {
        break;
      }

      unsigned long currentMillis = millis();
      if((currentMillis - previousToneMillis) > 50) {
        previousToneMillis = currentMillis;
        tone(BUZZER,toneMelody[i]-380,60);
        i++;
        }
    }
    ii++;
  }

}

// door open? then increment the timer - sound the alarm 
void fridgeDoorOpenTimer() {

  if (doorOpen == false && doorTrigger == false) {
    return;
  }

  if (digitalRead(RESET_BTN) == HIGH) {
    digitalWrite(POWER_LED, LOW);
    while (digitalRead(RESET_BTN) == HIGH) {
      delay(10);
    }

    if (SerialDebug) {
      Serial.println("reset");
    }
    startTime = millis();
    soundTheAlarm = false;
    digitalWrite(POWER_LED, HIGH);
  }

  // checking once a second
  doorOpenTime = millis() - startTime;
  if (doorOpenTime % 1000 == 0 && doorOpenTime != lastDoorOpenTime) {
    if (SerialDebug) {
      Serial.println(doorOpenTime);
    }

    if (doorOpenTime > doorOpenDelay) {
      if (SerialDebug) {
        Serial.println("Awwooga Awwooga - fridge door open!");
      }
      soundTheAlarm = true;
      soundAlarm();
    }

    lastDoorOpenTime = doorOpenTime;
  }

}


void setup() {

  pinMode(TEST_LED, OUTPUT);
  digitalWrite(TEST_LED, HIGH);

  if (SerialDebug) {
    Serial.begin(9600);
  }

  delay(2000);

  pinMode(DOOR_LED, OUTPUT);
  pinMode(POWER_LED, OUTPUT);
  pinMode(BUZZER, OUTPUT);

  if (SerialDebug) {
    Serial.println("ready");
  }
  digitalWrite(POWER_LED, HIGH);

  pinMode(FRIDGE_DOOR, INPUT);
  attachInterrupt(digitalPinToInterrupt(FRIDGE_DOOR), fridgeDoor, CHANGE);

  pinMode(RESET_BTN, INPUT);
  digitalWrite(TEST_LED, LOW);
}

void loop() {
  fridgeDoorOpen();
  fridgeDoorOpenTimer();
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

#define TEST_LED 13

#define FRIDGE_DOOR 6

#define DOOR_LED 8

#define POWER_LED 7

#define RESET_BTN 9

#define BUZZER 10

volatile boolean doorOpen = true; // when the light is on

volatile boolean doorTrigger = false;

volatile unsigned long lastFridgeMillis = 0;

volatile unsigned long fridgeTimeoutMillis = 10;

boolean SerialDebug = false;

unsigned long doorOpenDelay = 45000; // 45 second timeout before buzzer sounds

unsigned long doorOpenTime = 0;

unsigned long startTime;

unsigned long previousToneMillis = 0;

unsigned long toneInterval = 20;

boolean soundTheAlarm = false;

boolean outputTone = false;

int toneMelody[5] = { 1600, 1900, 2000, 2100, 1800 };

const int toneMax = 5;

unsigned long lastDoorOpenTime = 1;

char pout[80]; // for testing

// interrupt function

void fridgeDoor() {

if ((millis() - lastFridgeMillis) > fridgeTimeoutMillis) {

doorOpen = digitalRead(FRIDGE_DOOR); // goes high when dark

lastFridgeMillis = millis();

}

// see if the door is open or closed

void fridgeDoorOpen() {

if (doorOpen == true && doorTrigger == false) {

doorTrigger = true;

startTime = millis();

if (SerialDebug) {

sprintf(pout, "door open: %d", doorOpen);

Serial.println(pout);

}

digitalWrite(DOOR_LED, HIGH);

return;

}

if (doorOpen == false && doorTrigger == true) {

doorTrigger = false;

if (SerialDebug) {

sprintf(pout, "door closed: %d", doorOpen);

Serial.println(pout);

}

startTime = 0;

doorOpenTime = 0;

soundTheAlarm = false;

digitalWrite(DOOR_LED, LOW);

return;

}

// make some noise

void soundAlarm() {

if (soundTheAlarm == false) {

return;

}

int ii = 0;

while (ii < 6) {

if (soundTheAlarm == false) {

break;

}

int i = 0;

while (i < toneMax) {

if (soundTheAlarm == false) {

break;

}

unsigned long currentMillis = millis();

if((currentMillis - previousToneMillis) > 50) {

previousToneMillis = currentMillis;

tone(BUZZER,toneMelody[i]-380,60);

i++;

}

ii++;

}

// door open? then increment the timer - sound the alarm

void fridgeDoorOpenTimer() {

if (doorOpen == false && doorTrigger == false) {

return;

}

if (digitalRead(RESET_BTN) == HIGH) {

digitalWrite(POWER_LED, LOW);

while (digitalRead(RESET_BTN) == HIGH) {

delay(10);

}

if (SerialDebug) {

Serial.println("reset");

}

startTime = millis();

soundTheAlarm = false;

digitalWrite(POWER_LED, HIGH);

}

// checking once a second

doorOpenTime = millis() - startTime;

if (doorOpenTime % 1000 == 0 && doorOpenTime != lastDoorOpenTime) {

if (SerialDebug) {

Serial.println(doorOpenTime);

}

if (doorOpenTime > doorOpenDelay) {

if (SerialDebug) {

Serial.println("Awwooga Awwooga - fridge door open!");

}

soundTheAlarm = true;

soundAlarm();

}

lastDoorOpenTime = doorOpenTime;

}

void setup() {

pinMode(TEST_LED, OUTPUT);

digitalWrite(TEST_LED, HIGH);

if (SerialDebug) {

Serial.begin(9600);

}

delay(2000);

pinMode(DOOR_LED, OUTPUT);

pinMode(POWER_LED, OUTPUT);

pinMode(BUZZER, OUTPUT);

if (SerialDebug) {

Serial.println("ready");

}

digitalWrite(POWER_LED, HIGH);

pinMode(FRIDGE_DOOR, INPUT);

attachInterrupt(digitalPinToInterrupt(FRIDGE_DOOR), fridgeDoor, CHANGE);

pinMode(RESET_BTN, INPUT);

digitalWrite(TEST_LED, LOW);

}

void loop() {

fridgeDoorOpen();

fridgeDoorOpenTimer();

}

Links and Sources

Practical Electronics for Inventors, Third Edition. Paul Scherz and Simon Monk, pub. McGraw Hill, 2013. ISBN 978-0-07-177133-7
A pre-made USB LiPo Battery Charger – 3.7V Single Cell from Hobbytronics or from Sparkfun
Battery Charger MCP3831T Datasheet
TEPT4400 Phototransistor Datasheet

APPLE, Computer, linux, Programming, Raspberry Pi

Extracting MP3 audio from video files

29th July 2018 Karl

Here is a small Bash script that converts any supported ffmpeg video format; such as .MKV, .MP4 or .MOV and extracts the audio to an .MP3 file, It will also split that MP3 file into chunks and put them in a convenient directory. You will need to install ffmpeg and mp3splt for your particular platform.

Example Usage:

./mkv2mp3 "big fat file.mkv"

1	./mkv2mp3 "big fat file.mkv"

This uses ffmpeg to convert “big fat file.mkv” to “big fat file.mp3” and then uses mp3splt to create a directory “big fat file” containing the files 01 – big fat file.mp3, 02 – big fat file.mp3, etc. The MP3 files will be encoded at 128k Constant Bit Rate and each file will be around 50 minutes in length. To install in Debian/Ubuntu use: sudo apt-get install ffmpeg mp3splt

#!/bin/bash
INFILE=$1
BITRATE="128k"
LENGTH="50.0"

FILENAME=$(basename "$INFILE" | cut -d. -f1)
if [[ -z "${FILENAME// }" ]]; then
   echo "filename missing"
   exit 1
fi

echo "$FILENAME"
ffmpeg -i "$INFILE" -vn -acodec libmp3lame -b:a "$BITRATE" "$FILENAME".mp3
mp3splt -t "$LENGTH" -d "$FILENAME" -a -o @n+-+@f "$FILENAME".mp3

#!/bin/bash

INFILE=$1

BITRATE="128k"

LENGTH="50.0"

FILENAME=$(basename "$INFILE" | cut -d. -f1)

if [[ -z "${FILENAME// }" ]]; then

echo "filename missing"

exit 1

echo "$FILENAME"

ffmpeg -i "$INFILE" -vn -acodec libmp3lame -b:a "$BITRATE" "$FILENAME".mp3

mp3splt -t "$LENGTH" -d "$FILENAME" -a -o @n+-+@f "$FILENAME".mp3

mp3splt can find the audio in a quiet region near where the split is desired rather than midway through a word, this should make for much cleaner playback across tracks.

Alternative Method

This script gives the same results but uses ffmpeg to split the large MP3 file and then adds track numbering metadata using id3v2. To install in Debian/Ubuntu use: sudo apt-get install ffmpeg id3v2

#!/bin/bash
INFILE="$1"
BITRATE="128k"
LENGTH="3000"    # 50 minutes in seconds

FILENAME=$(basename "$INFILE" | cut -d. -f1)
if [[ -z "${FILENAME// }" ]]; then
   echo "filename missing"
   exit 1
fi

echo "$FILENAME"
MP3FILE="$FILENAME".mp3
ffmpeg -y -i "$INFILE" -vn -acodec libmp3lame -b:a "$BITRATE" "$MP3FILE"

OUTDIR="$FILENAME"
mkdir "$OUTDIR"

ffmpeg -y -i "$MP3FILE" -vn -f segment -segment_time "$LENGTH" -c copy "$OUTDIR"/%02d-"$MP3FILE"

TRACK=1
IFS=$'\n'
FILES=($(find "$OUTDIR" -name "*.mp3" | sort -n))
for file in "${FILES[@]}"; do
  TN=$(printf %02d $TRACK)
  id3v2 --track "$TN" --TIT2 "$FILENAME" "$file"
  TRACK=$((TRACK+1))
done

#!/bin/bash

INFILE="$1"

BITRATE="128k"

LENGTH="3000" # 50 minutes in seconds

FILENAME=$(basename "$INFILE" | cut -d. -f1)

if [[ -z "${FILENAME// }" ]]; then

echo "filename missing"

exit 1

echo "$FILENAME"

MP3FILE="$FILENAME".mp3

ffmpeg -y -i "$INFILE" -vn -acodec libmp3lame -b:a "$BITRATE" "$MP3FILE"

OUTDIR="$FILENAME"

mkdir "$OUTDIR"

ffmpeg -y -i "$MP3FILE" -vn -f segment -segment_time "$LENGTH" -c copy "$OUTDIR"/%02d-"$MP3FILE"

TRACK=1

IFS=$'\n'

FILES=($(find "$OUTDIR" -name "*.mp3" | sort -n))

for file in "${FILES[@]}"; do

TN=$(printf %02d $TRACK)

id3v2 --track "$TN" --TIT2 "$FILENAME" "$file"

TRACK=$((TRACK+1))

done

Creating an Audiobook

Taking this further, I was thinking that it would be nice to have these converted into the M4B Audiobook format for use on my elderly iPod. The script below assumes that you have processed the files as above and have added metadata tags using a tool like mp3tag (yes I know this is for Windows).

To complete this we need to: Combine the multiple MP3 files into one big file, or read the original big file then convert that to M4B format at 96K bit and add chapter marks every ten minutes. For this I have used ffmpeg v3.2.12 and libmp4v2 (for the mp4chaps utility), to install in Debian/Ubuntu use: sudo apt-get install libmp4v2-dev mp4v2-utils ffmpeg

This script works best from a single MP3 file rather than from those that have been re-combined back into a single file, recombining the files caused ffmpeg to exclaim “invalid packet size” and “invalid data” errors. It is able to tell the difference between a directory and a single MP3 and processes the file accordingly, don’t forget to add metadata tags and cover art before you run the script.

#!/bin/bash
INFILE="$1"

BITRATE="96k"
MP3TITLE=""
MP3ARTIST=""
MP3ALBUM=""
MP3COMMENT=""
MP3YEAR=""
MP3PERFORMER=""

####
####  FUNCTIONS
####

function setMP3variables() {
  TAG="$1"
  VALUE="$2"

  case "$TAG" in
    title)
      MP3TITLE="$VALUE"
      ;;
    artist)
      MP3ARTIST="$VALUE"
      ;;
    comment)
      MP3COMMENT="$VALUE"
      ;;
    album_artist)
      MP3PERFORMER="$VALUE"
      ;;
    album)
      MP3ALBUM="$VALUE"
      ;;
    date)
      MP3YEAR="$VALUE"
      ;;
   esac
}

## get the tag text from the metadata
## if the line contains an equals (=) then split by the first equals.
function getMP3tags() {
  MP3METADATA="$1"
  while read -r line
  do
    case "$line" in
      *=*)
        TAG=${line%%"="*}
        VALUE=${line#*"="}
        setMP3variables "$TAG" "$VALUE"
        ;;
    esac
  done < "$MP3METADATA"

  if [[ ! -z "${MP3TITLE// }" ]]; then
    OUTFILE="$MP3TITLE".mp3
    M4BFILE="$MP3TITLE".m4b
  fi
}

####
####   BEGINS
####

FILENAME=$(basename "$INFILE" | cut -d. -f1)
if [[ -z "${FILENAME// }" ]]; then
   echo "filename missing"
   exit 1
fi
OLDIFS=$IFS
OUTFILE="$FILENAME"_out.mp3
M4BFILE="${OUTFILE:0:-4}".m4b
METADATA="${OUTFILE:0:-4}".metadata
JPGFILE="${OUTFILE:0:-4}".jpg


## check if input is a diretory
if [ -d "$INFILE" ]; then
  # get the name of the first MP3 file in the directory
  FIRSTFILE=$(find "$INFILE" -name "*.mp3" | sort -n | head -1)
  FIRSTFILE=$(basename "$FIRSTFILE")

  if [[ -z "${FIRSTFILE// }" ]]; then
    echo "no mp3 files in directory"
    exit 1
  fi

  ## save the MP3 tag metadata from the first file
  ffmpeg -y -i "$INFILE/$FIRSTFILE" -f ffmetadata "$METADATA"
  ## save out the cver art image
  ffmpeg -y -i "$INFILE/$FIRSTFILE" -an -vcodec copy "$JPGFILE"
  ## process the tag metadata - put it into the $MP3* strings, set $OUTFILE and $LOGFILE from the title
  getMP3tags "$METADATA"

  ## make a bar seperated list of the MP3 files that are to be combined
  MP3FILES=""
  IFS=$'\n'
  FILES=($(find "$INFILE" -name "*.mp3" | sort -n))
  for files in "${FILES[@]}"; do
    MP3FILES+="$files|"
  done
  MP3FILES=${MP3FILES:0:-1}

  ## combine all the MP3 files into one big file
  ## $OUTFILE is set using either the $FILENAME or $MP3TITLE if that has been used
  ffmpeg -y -i "concat:$MP3FILES" -vn -vsync 2 -acodec copy "$OUTFILE"
else
  ## process a single MP3 file
  ## save the MP3 tag metadata from the first file
  ffmpeg -y -i "$INFILE" -f ffmetadata "$METADATA"
  ffmpeg -y -i "$INFILE" -an -vcodec copy "$JPGFILE"
  ## process the tag metadata - put it into the $MP3* strings, set $OUTFILE and $LOGFILE from the title
  getMP3tags "$METADATA"
  #ffmpeg -y -i "$INFILE" -vn -vsync 2 -acodec copy "$OUTFILE"
  OUTFILE="$INFILE"
fi

## if there is a cover image, these are are the options for adding it back in
WITHJPG=()
if [ -e "$JPGFILE" ]; then
   IFS=$OLDIFS
   WITHJPG=(-i "$JPGFILE" -metadata:s:v comment="Cover (front)")
fi

## create the audiobook M4B file
## and write the metadata into the output file
ffmpeg -y -i "$OUTFILE" -i "$METADATA" "${WITHJPG[@]}" -map_metadata 1 -id3v2_version 3 -acodec aac -b:a "$BITRATE" -strict -2 -f mp4 "$M4BFILE"
## add the chapters every 10 minutes
mp4chaps -e 600 "$M4BFILE"

## tidy up
rm "$MP3METADATA"
if [ -e "$JPGFILE" ]; then
   rm "$JPGFILE"
fi
#rm "$OUTFILE"
echo "finished"

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

#!/bin/bash

INFILE="$1"

BITRATE="96k"

MP3TITLE=""

MP3ARTIST=""

MP3ALBUM=""

MP3COMMENT=""

MP3YEAR=""

MP3PERFORMER=""

####

#### FUNCTIONS

####

function setMP3variables() {

TAG="$1"

VALUE="$2"

case "$TAG" in

title)

MP3TITLE="$VALUE"

;;

artist)

MP3ARTIST="$VALUE"

;;

comment)

MP3COMMENT="$VALUE"

;;

album_artist)

MP3PERFORMER="$VALUE"

;;

album)

MP3ALBUM="$VALUE"

;;

date)

MP3YEAR="$VALUE"

;;

esac

}

## get the tag text from the metadata

## if the line contains an equals (=) then split by the first equals.

function getMP3tags() {

MP3METADATA="$1"

while read -r line

case "$line" in

*=*)

TAG=${line%%"="*}

VALUE=${line#*"="}

setMP3variables "$TAG" "$VALUE"

;;

esac

done < "$MP3METADATA"

if [[ ! -z "${MP3TITLE// }" ]]; then

OUTFILE="$MP3TITLE".mp3

M4BFILE="$MP3TITLE".m4b

}

####

#### BEGINS

####

FILENAME=$(basename "$INFILE" | cut -d. -f1)

if [[ -z "${FILENAME// }" ]]; then

echo "filename missing"

exit 1

OLDIFS=$IFS

OUTFILE="$FILENAME"_out.mp3

M4BFILE="${OUTFILE:0:-4}".m4b

METADATA="${OUTFILE:0:-4}".metadata

JPGFILE="${OUTFILE:0:-4}".jpg

## check if input is a diretory

if [ -d "$INFILE" ]; then

# get the name of the first MP3 file in the directory

FIRSTFILE=$(find "$INFILE" -name "*.mp3" | sort -n | head -1)

FIRSTFILE=$(basename "$FIRSTFILE")

if [[ -z "${FIRSTFILE// }" ]]; then

echo "no mp3 files in directory"

exit 1

## save the MP3 tag metadata from the first file

ffmpeg -y -i "$INFILE/$FIRSTFILE" -f ffmetadata "$METADATA"

## save out the cver art image

ffmpeg -y -i "$INFILE/$FIRSTFILE" -an -vcodec copy "$JPGFILE"

## process the tag metadata - put it into the $MP3* strings, set $OUTFILE and $LOGFILE from the title

getMP3tags "$METADATA"

## make a bar seperated list of the MP3 files that are to be combined

MP3FILES=""

IFS=$'\n'

FILES=($(find "$INFILE" -name "*.mp3" | sort -n))

for files in "${FILES[@]}"; do

MP3FILES+="$files|"

done

MP3FILES=${MP3FILES:0:-1}

## combine all the MP3 files into one big file

## $OUTFILE is set using either the $FILENAME or $MP3TITLE if that has been used

ffmpeg -y -i "concat:$MP3FILES" -vn -vsync 2 -acodec copy "$OUTFILE"

else

## process a single MP3 file

## save the MP3 tag metadata from the first file

ffmpeg -y -i "$INFILE" -f ffmetadata "$METADATA"

ffmpeg -y -i "$INFILE" -an -vcodec copy "$JPGFILE"

## process the tag metadata - put it into the $MP3* strings, set $OUTFILE and $LOGFILE from the title

getMP3tags "$METADATA"

#ffmpeg -y -i "$INFILE" -vn -vsync 2 -acodec copy "$OUTFILE"

OUTFILE="$INFILE"

## if there is a cover image, these are are the options for adding it back in

WITHJPG=()

if [ -e "$JPGFILE" ]; then

IFS=$OLDIFS

WITHJPG=(-i "$JPGFILE" -metadata:s:v comment="Cover (front)")

## create the audiobook M4B file

## and write the metadata into the output file

ffmpeg -y -i "$OUTFILE" -i "$METADATA" "${WITHJPG[@]}" -map_metadata 1 -id3v2_version 3 -acodec aac -b:a "$BITRATE" -strict -2 -f mp4 "$M4BFILE"

## add the chapters every 10 minutes

mp4chaps -e 600 "$M4BFILE"

## tidy up

rm "$MP3METADATA"

if [ -e "$JPGFILE" ]; then

rm "$JPGFILE"

#rm "$OUTFILE"

echo "finished"

When encoding to the M4B using a re-combined file I saw a few of these errors from ffmpeg:

[mp3 @ 0x7fffc3bc9ea0] Header missing
Error while decoding stream #0:0: Invalid data found when processing input

1 2	[mp3 @ 0x7fffc3bc9ea0] Header missing Error while decoding stream #0:0: Invalid data found when processing input

These appear to be caused by the mp3splt program from when the original MP3 file was being split into 50 minute chunks, but I can’t hear any effect on the output.

Lots of information about the file can be gotten using mediainfo, to install in Debian/Ubuntu use: sudo apt-get install mediainfo, example use:

mediainfo --fullscan "big fat file.m4b"

1	mediainfo --fullscan "big fat file.m4b"

Links and References

ffmpeg, encode AAC: https://trac.ffmpeg.org/wiki/Encode/AAC
ArchLinux audiobook howto

g7smy.co.uk

Category Archives: Programming

Converting a UTC date time to local in PHP

Links and Sources

Karl’s Roast Calculator – Updated

Jetson Nano Install Notes

Initial Startup

Customising the Setup

Post Install Problems

Setup the Machine Learning

Further Reading

Extracting GPS data from the GoPro 7

Installing the gopro-utils

Extracting the Data

Merging GPX files

Links and Sources

The Fridge Door is Open

Circuitry

Software

Links and Sources

Extracting MP3 audio from video files

Alternative Method

Creating an Audiobook

Links and References

COMPUTERS, ROBOTS AND PHOTOGRAPHY