Getting Indexed

Although in theory Google can index single page applications it isn't working for me. So I've put this page together: Links to Boats to try to help. 

Our boats are:

1. Robinetta
2. Tiki
3. Sùlaire

Raspberry Pi as an NMEA Multiplexer

So I got an AIS 'transponder'. They call them transponders - it says it on the box. But it isn't actually a transponder. A transponder transmits in response to a probe. AIS 'transponders' are beacons. They transmit even if nothing is probing them.

Anyway, I want the AIS targets displayed on the chart plotter and the chart plotter only has one NMEA input which is already used for depth so I need a multiplexer.

I'm using a Pi3 with USB/Serial adaptors. The plan was to use all of the four USB sockets with one adaptor going to the AIS transponder, one to the chart plotter, one to the DSC VHF radio and one to the depth device.

But one of my USB ports doesn't work. So I'm sharing one device for the depth and the VHF. The input comes from the depth unit and the output goes to the VHF. I don't have anything to get the DSC targets from the radio but that's OK for now.

There are (at least) two choices for software NMEA multiplexors on Linux. The most mature is kplex. The alternative is Signal K. I'm using both but I'm only using kplex in anger.

There are problems.

1. The wifi is configured as a hotspot. It doesn't always start-up properly. I've put a 10s delay in the init.d script for now and that seems to fix it but its a hack.
2. I can't work out how to make the USB/Serial ports always end up on the same device names.

There is lots of advice on the internet about USB naming. The answer is to write some UDEV rules to sort it out. This works OK for the single PL2303 device but I haven't come up with anything for the two indistinguishable Cxxxx devices.

I think I need to make kplex auto-baud and auto detect the type of device from the sentences it outputs.

Raspberry Pi Kodi CEC for Yamaha A/V Receiver

CEC works out of the box on Kodi on the Pi but not if you have an A/V Receiver and a projector rather than a TV.

If can put put right in the settings in Kodi if you have a working remote but a mouse doesn't seem to do it.

Copying the relevant config file from a working system is the easiest way. For me this file gets created as:
 
/home/kodi/.kodi/userdata/peripheral_data/rpi_2708_1001.xml

2708:1001 is the device ID for the pi's CEC adapter.

The contents of my file are:

<settings>
    <setting id="activate_source" value="0" />
    <setting id="button_release_delay_ms" value="0" />
    <setting id="button_repeat_rate_ms" value="0" />
    <setting id="cec_hdmi_port" value="1" />
    <setting id="cec_standby_screensaver" value="0" />
    <setting id="cec_wake_screensaver" value="1" />
    <setting id="connected_device" value="36037" />
    <setting id="device_name" value="Kodi" />
    <setting id="device_type" value="1" />
    <setting id="double_tap_timeout_ms" value="300" />
    <setting id="enabled" value="1" />
    <setting id="pause_playback_on_deactivate" value="1" />
    <setting id="physical_address" value="0" />
    <setting id="port" value="" />
    <setting id="send_inactive_source" value="1" />
    <setting id="standby_devices" value="231" />
    <setting id="standby_devices_advanced" value="" />
    <setting id="standby_pc_on_tv_standby" value="13005" />
    <setting id="standby_tv_on_pc_standby" value="1" />
    <setting id="tv_vendor" value="0" />
    <setting id="use_tv_menu_language" value="1" />
    <setting id="wake_devices" value="231" />
    <setting id="wake_devices_advanced" value="" />
</settings>

MPEG-DASH and HLS with Gstreamer

Gstreamer does a good job with the modern streaming protocols.

The command line client gst123 can be used to listen to BBC AoD and live streams.

BBC AoD streams can be found here: BBC World Service AoD feed (replace worldservice with your channel of choice).

BBC Live streams can be found here: BBC IMDA transports

Here are some examples using gst123:

Play an MPEG-DASH stream listed in imda_transports.xml:

gst123 -q http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/dash/nonuk/dash_low/ak/bbc_radio_three.mpd

Play an HLS stream listed in imda_transports.xml:

 gst123 -q http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/hls/uk/sbr_vlow/llnw/bbc_radio_three.m3u8

Play an on-demand DASH stream from Radio 4 (you will need to look in the xml file for a current programme):

 gst123 -q http://open.live.bbc.co.uk/mediaselector/5/redir/version/2.0/mediaset/audio-syndication-dash/proto/http/vpid/b084d7wf

It is really easy to roll your own Gstreamer based player using python. There are some examples here.

I rolled this one by adding command line processing to basic-tutorial-1:

#!/usr/bin/env python3
# -*- coding:utf-8 -*-
import sys
import gi
gi.require_version('Gst', '1.0')
from gi.repository import Gst
Gst.init(None)

# Build the pipeline
pipeline = Gst.parse_launch("playbin uri=%s" % sys.argv[1
# Start playing
pipeline.set_state(Gst.State.PLAYING)

# Wait until error or EOS
bus = pipeline.get_bus()
msg = bus.timed_pop_filtered(
    Gst.CLOCK_TIME_NONE, Gst.MessageType.ERROR | Gst.MessageType.EOS)
# Free resourcespipeline.set_state(Gst.State.NULL)

Then you can use the same example streams:

./gst.py http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/dash/nonuk/dash_low/ak/bbc_radio_three.mpd

./gst.py http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/hls/uk/sbr_vlow/llnw/bbc_radio_three.m3u8

It only takes a little more effort to make something useful. The following python3 script takes the name of a programme and the url of an AoD feed on the command line and plays the latest episode of that programme:

# Free #!/usr/bin/env python3
# -*- coding:utf-8 -*-
import sys
import gi
gi.require_version('Gst', '1.0')
from gi.repository import Gst
from xml.etree import ElementTree
from datetime import datetime
import arrow
from urllib.request import urlopen

Gst.init(None)

# Get a url
title = sys.argv[1]
feed = sys.argv[2]
now = arrow.utcnow()
wanted = now.replace(hours=-1)
with urlopen(feed) as f:
    tree = ElementTree.parse(f)
for node in tree.iter('entry'):
    for entry in node.iter('parent'):
        if entry.text == title:
            for field in node.iter():
                if field.tag == 'link' and field.attrib.get('transferformat') == 'dash':
                    thisurl = field.text
                if field.tag == 'availability':
                    start = arrow.get(field.attrib.get("start"))
                    end = arrow.get(field.attrib.get("end"))
                    if (start <= now) and (now <= end) and (start > wanted):
                        wanted = start
                        url = thisurl
# Build the pipeline
pipeline = Gst.parse_launch("playbin uri=%s" % url)

# Start playing
pipeline.set_state(Gst.State.PLAYING)

# Wait until error or EOS
bus = pipeline.get_bus()
msg = bus.timed_pop_filtered(
    Gst.CLOCK_TIME_NONE, Gst.MessageType.ERROR | Gst.MessageType.EOS)

# Free resources
pipeline.set_state(Gst.State.NULL)

So you can do this:

./bbcaod.py   "BBC News" http://www.bbc.co.uk/radio/aod/availability/worldservice.xml

Google Maps Location Sillyness

Google Maps has a funny idea of where things are.

If I view the 'my contributions' page it looks something like this

See that marker in the middle of the North Sea? And the photo on the left which says North Sea?

Now look at this image of Google Maps zoomed in to near Harwich

That's the same picture.

When I look at this on Panoramio it says the location is Harwich.

In fact it says "Photo taken in 13 Lower Marine Parade, Harwich CO12 3SS, UK

51° 54' 50.75" N  1° 18' 15.42" E"

Which is clearly not true but better than "North Sea".

I can't find any way of adding names of seas, bays or channels or of changing where this is, or how come Google Maps says its in the north sea but Panoramio says its in Harwich.

OpenCPN is in the Play Store

Which is great news but it runs very poorly on my Galaxy Tab2 7". I think it needs to be a lot more optimised to work properly.

I can use my Antares charts on it and in theory OpenSeaMap but those folks are not good at making their work easily accessible. I'll find out how best to do it and post that here.

OpenCPN has great built-in NMEA multiplexing so that part is not a problem. I can get my dummy AIS target displayed just fine.

Of course OpenCPN has access to proper charts via VisitMyHarbour. On Windows.

Of course VisitMyHarbour provide proper charts for Android. On Marine Navigator.

Marine Navigator doesn't do AIS or any NMEA overlays.

Serial to Wifi

It looks like the ESP8266 may be the way to do this. I'm not sure if it can do a full TCP relay but I think it can. It needs NMEA 0183 level shifting but apart from that it seems like the perfect low cost way to get away from NMEA wiring.

AIS Engine and Android

A friend asked last night how to connect his Nasa AIS Engine to his Google Nexus tablet so he could see AIS plots on his charts.

He uses Marine Navigator and has the tablet plugged in to the boat power all the time to keep it charged.

I did a quick search and as I expected it is complicated...

Marine Navigator doesn't support AIS - at least it doesn't mention it  on the Google Play page.

MemoryMap does. He would have to buy his charts all over again though. When are we going to get chart portability between apps?

MemoryMap does - NMEA input via wifi or Bluetooth including DSC and AIS.

So the obvious thing to do would be to connect an NMEA to USB converter like the Actisense one to the tablet.

Not so simple. Nexus tablets are rumoured not to support OTG devices when charging.

Not so simple. Android doesn't support serial devices.

I did find one piece of good news. The NMEA output on the Nasa AIS engine is RS232 compatible. So we don't need an expensive NMEA to USB converter. A cheap RS232 to USB converter will do.

Other good news is that libusb works on Android and there is open source software to connect to a libusb connected serial port adapter if it has either an FTDI chip (most do) or a pl2303 chip (mine does).

So there is a way to do it if the charging problem isn't real:

1) Connect the AIS Engine to an RS232 to USB adapter.
2) Connect the RS232 to USB adapter to a Nexus compatible USB OTG adapter.
3) write a little app that uses the open source libusb and serial port drivers and acts as a tcp/ip server for NMEA messages
4) install Memory-Map

If the charging problem is real, or we want to be able to walk around the boat with the tablet, then we can put the app on a Raspberry Pi instead and MemoryMap can connect over wireless.

So this is my bill of materials for the Pi solution.

Hardware:
1 x raspberry Pi 1 Model B (for example)
1 x SD Card
1 x 5V 3A radio control model BEC (DC-DC converter)
1 x RS232/USB Adapter
1 x USB Wifi Module
1 x Mini USB cable
1 x box to put it in

Software
raspian
kplex

I'll get this working and feed it some canned AIS messages and see if I can overlay them on memory map.

It turns out the AIS features in MemoryMap are "pro" features and cost over $100. That's not what we want.

The demo I tried didn't let me use the internal GPS and an external (tcp) AIS receiver. I don't know if this is fundamental but it breaks the model for me.

I wrote an add-on for gpsfeed+ which makes it generate an AIS target near the track in simulation mode. This is just the kind of test software I need for making everything BUT the RF ais decoder work in my NMEA based projects. That is really useful.

Not just code - part 1

I'm way behind the curve here but fancied a go at ultra-low cost boat instruments. Robinetta has a 1970s depth sounder and a small chart plotter and that's about it. Alison would like wind speed and direction and it would be nice to know boat speed. Remoting the depth information to the cockpit would be nice.

There are freeware and open source chart plotter applications now. It should be possible to do most things at very low cost.

I'm going to start with a magnetic compass because that seems easiest. We don't need it but it will talk to the chart plotter and/or the tiller pilot using serial NMEA.

BOM

Arduino Nano v3                                £8.56
HMC5883L triple axis magnetometer £1.88
SN75176AD EIA422 transceiver       £0.26
Prototyping Box                                  £1.77

I've made a gimbal mount out of scrap plywood and I'll mount the magnetometer on that and the arduino and bus transceiver in the prototyping box. They will be wired together with flexible wire left over from a Round-the-pole model aircraft toy I made.

I'll try both USB power and raw boat 12V.

I haven't worked out if the UART is available when the USB is not plugged in but I should be able to use s/w serial port on digital I/O pins.

Porting a Qt Desktop application to Android

I am getting Dream to work on Android. There a many things to do and most of them are hard. I'll post a few things about the experience here. Things to do are:

• port the audio capture and playout
• Make dream work in a single window rather than many overlapping windows
• get menus working
• other things I haven't thought of yet

The first thing I tried was to write an openSL ES sound driver. I couldn't get it to work. Qt is supposed to support capture and playback using Qt Multimedia on Android from Qt 5.2. So I've hacked Dream to use Qt Multimedia. If it works well on Andriod we can make it the default for a GUI build and just keep pulseaudio, portaudio and Windows MM for console builds. I tried making a main window display in QML. It works OK but it will be much quicker to stay mostly with QWidgets. I found the Qt 5.2beta1 needs the QML libraries anyway so I gave it a go. It seems you can embed QML in QWidgets but not the other way round. See QML in a QWidget.

Trying out Wordpress

I promised Julie I would try Wordpress out again. So I have here. It seems just as easy to use as blogger. I'll also try it from my phone and tablet.

The default look is nice. The only things I found a little annoying were the continual prompting to go premium and the mix of secure and non-secure content on the page which causes annoying popups in IE.

Mapping the Old Gaffers

It is the Old Gaffers Association's 50th year and a summer long party is in progress around the British Isles. One of the main items is a circumnavigation and we wanted to track the boats on-line.

Trackaphone kindly offered to sponsor the event and provide phones and apps for the boats to report their positions and a dedicated map. But gaffers are an independent lot and only about a third of the boats signed up. Some are blogging their positions, others have class B AIS transceivers on-board, a few have mAIS/iAIS apps on their phones and others are technology-free.

I thought it would be cool to create a combined map showing all the boats we can. This is how it works.

Feeds

We have three basic sources of information, Trackaphone, Marine Traffic and human beings. Trackaphone collect the data from their apps and present it in a simple xml feed at a specific URL. Trackaphone take care of assigning the inputs from specific phones to our fleet.

Marine Traffic offer a 'my fleet' service. Anyone can register on their website and create a list of boats they are interested in. There are AIS receivers all over the world which receive AIS transmissions and report positions to the site. Additionally, there are many ways ships can send their data to the site without needing a real AIS transceiver. The primary key for ships is their MMSI. Boats (like Robinetta) without a DSC radio can register on the site and Marine Traffic will allocate them a dummy MMSI for use just on the site.

Marine Traffic offer fleet xml feeds as a commercial service. Demitris at Marine Traffic kindly set up a feed for my fleet until the end of August free of charge as a birthday present to the OGA. Thanks Demitris! There are two feeds, a short feed which can be queried every 5 minutes and a long feed which can be queried every hour. Only ships which have reported recently are included in the feed, so I need some way to persist the data.

Finally, for boats with no automatic reporting, I get emails and read blog posts to get updates which can be manually applied to the database.

Hosting

Heroku is a 'Cloud Application Platform' which offers commercial application hosting with a free development tier. The traffic and uptime requirements for my map fit comfortably in the free tier. Heroku apps can be written using Ruby, Node.js, Clojure, Java, Python, and Scala. Of these, I'm most proficient in Java but Heroku support Spring MVC and I only know enough Spring to use Apache Camel and a full MVC is way over the top for building a single map. I know enough Python to get by and Heroku encourage Django which looked about right.

Heroku provide a postgres database for every application and postgres is my favourite!

There are hundreds of add-ons. The only one I needed was the scheduler which provides cron-like facilities.

Server-side Code

This was my first time using a framework like Django in earnest. Like Rails, Django provides a strong set of policies for separating the layers of a web application. It supports Views, Models and Templates and, I'm sure loads more that I haven't needed.

Models are the way in Django that you define your persistant objects. You define a Python class, using some conventions and Django goes away and creates a database and tables for the objects using its' Object-Relational Mapping system. One weakness is that if you add a field to the model there is no way Django will update the database schema. But it's easy enough to do by hand in SQL.

The coolest thing about models is that they come with an automatically generated admin interface. This provides a facility similar to MysqlAdmin but just for the Django model objects. It is possible to enhance and skin this but it works out of the box. This is how I do the manual updates for non-tracked boats.

Views render pages using templates.

For my map, I put all the html and Javascript in a single template (OK, I should separate out the scripts, but this is a quick and dirty project). The template has a template language which can loop over parameters passed in from the view, and this is what it does with each object.

The basic model for the application is as follows:

• Every 10 minutes, Heroku Scheduler calls my refresh script (a normal Django/Python script) which polls the MarineTraffic and Trackaphone xml feeds and updates the database.
• When I get new information about a non-tracked boat I use the Django admin page to update the database.
• When a user views the map page, a Django View reads the database, extracts the boats and passes them to the template processor.
• The map is rendered in the browser.

Map

The map is rendered by the Leaflet Javascript mapping library using the MapQuest Open tileset based on openstreetmap. Each boat or Relay Challenge Token is represented by a marker with a popup.

The markers are custom versions of free markers from Nicolas Mollet's Maps Icons Collection. I hand modified the sailing icon to look more gaffer-ish using Adobe Photoshop Elements and created shadow images for them using Erwin Bolwidt's Shadowmaker.

Where possible, boats have a popup which provides an image of the boat, and a link to the boat's blog. These links are all stored in the database by hand.

Blogging on the iPad

I don't have an iPad but I have an iPhone and my friend Julie wants some advice on this topic.

Before I first started blogging I tried all the free services I could find and settled on blogger.

But the official iPhone blogger app isn't great.

Looking around the web it seems some consolidation has happened. People only talk about blogger and Wordpress. That makes things easier.

Wordpress is a clear choice if you think you might want to host your blog yourself.

You can start on Wordpress.com and the software is free and you can migrate your content.

Blogger is very simple to get started with. I'm still happy with it for robinetta-log.

I found two good posts on this topic. In my next post on the subject I'll report on using Wordpress again.

Unfortunately blogsy which is recommended only works on the iPad.

http://www.macstories.net/reviews/blogsy-a-better-blogging-app-for-ipad/

http://theedublogger.com/2011/06/15/blogging-with-an-ipad-writing-posts-using-apps-and-via-the-dashboard-in-a-web-browser/

List of times in Oracle

Oracle Row Generator Techniques is pretty definitive. Here is a nice example to provide 15 minute times in a day:

select to_char(trunc(sysdate)+(level-1)*15/1440, 'HH24:MI') from dual connect by level <= 240;

GETNEXT and SNMP Tables

I recently needed to understand actually how the ordering for the oids in an SNMP table need to be arranged for commands like snmpwalk and snmptable to correctly traverse a table.

So here is a MIB fragment for a table:

jmxTable OBJECT-TYPE
    SYNTAX      SEQUENCE OF jmxEntry
    MAX-ACCESS  not-accessible
    STATUS      current
    DESCRIPTION
        ""
    ::= { monMIB 2 }
jmxEntry OBJECT-TYPE
    SYNTAX      JmxEntry
    MAX-ACCESS  not-accessible
    STATUS      current
    DESCRIPTION
        ""
    INDEX       { routeIndex }
    ::= { jmxTable 1 }
JmxEntry ::= SEQUENCE {
    routeIndex    INTEGER,
    routeName     DisplayString,
    contextName     DisplayString,
    exchangesFailed INTEGER
}
routeIndex OBJECT-TYPE
    SYNTAX      INTEGER
    MAX-ACCESS  read-only
    STATUS      current
    DESCRIPTION
        ""
routeName OBJECT-TYPE
    SYNTAX    DisplayString
    MAX-ACCESS  read-only
    STATUS      current
    DESCRIPTION
        ""
    ::= { jmxEntry 2 }
contextName OBJECT-TYPE
    SYNTAX    DisplayString
    MAX-ACCESS  read-only
    STATUS      current
    DESCRIPTION
        ""
    ::= { jmxEntry 3 }
exchangesFailed OBJECT-TYPE
    SYNTAX      INTEGER
    MAX-ACCESS  read-only
    STATUS      current
    DESCRIPTION
        ""
    ::= { jmxEntry 4 }
We've got a four column table with a visible integer index, two text columns and a numeric column.

It turns out the important thing is that getnext on jmxEntry.0 must return jmxEntry.1.1.

Of course jmxEntry.0 doesn't exist in the MIB, except for its ability to respond to getnext.

command line xslt transforms

Just putting this here because I was asked to. How to transform a file using XSLT on the command line.

Here is a really simple xsl stylesheet to remove some elements from an xml file:

<?xml version="1.0"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
 <xsl:output method="xml" indent="yes" omit-xml-declaration="no"/>
 <xsl:template match="node()|@*">
  <xsl:copy>
   <xsl:apply-templates select="node()|@*"/>
  </xsl:copy>
 </xsl:template>
 <xsl:template match="book[@name='X']"/>
</xsl:stylesheet>
</xsl:stylesheet>

Here is an xml file:

<?xml version="1.0" encoding="UTF-8"?>
<a>
<book name="X">
<title>A book</title>
</book>
<book name="Y">
<title>Another book</title>
</book>
</a>
Here is a command:

xsltproc x.xsl x.xml

And here is the output;

<?xml version="1.0"?>
<a>
<book name="Y">
<title>Another book</title>
</book>
</a>

securing jmx

JMX is cool. Lots of Java stuff has fantastic management beans that work really well.

Too well.

I have Apache ActiveMQ and Apache Camel running under Tomcat and anyone can fire up a jconsole and do what they like to it.

Now I need to mediate this stuff to SNMP because that's what our management tool uses. I could setup all the JMX security with ssl and certificates, but I really just want it so only my SNMP sub-agent can talk to it. Now you'd think telling the socket to just listen on the loopback interface would be easy.

It is.

-Djava.rmi.server.hostname=127.0.0.1

Finding that out took ages. I got it from here. But the usually reliable stackoverflow gives a way overcomplicated answer.