TheLazy.Dev
Portfolio GitHub

Blog

tutorial |bash |gdal |gnuplot

gpsdecode and ogr2ogr for AIS messages

jordanbell2357

Jordan Bell

Posted on February 14, 2024

gpsdecode and ogr2ogr for AIS messages

Access to AIS data

Kystverket (Norwegian Coastal Administration)

AIS data are also available at the following IP address: 153.44.253.27 port 5631

The AIS data stream is in standard IEC format IEC 62320-1.

See https://www.confluent.io/blog/streaming-etl-and-analytics-for-real-time-location-tracking/

OpenCPN

We start by viewing the AIS data from 153.44.253.27 port 5631 using OpenCPN.

AIS data from 153.44.253.27 port 5631 using OpenCPN

AIS and NMEA message format

https://documentation.spire.com/tcp-stream-v2/

https://www.itu.int/rec/R-REC-M.1371-5-201402-I/en

AIVDM/AIVDO protocol decoding. Eric S. Raymond. Version 1.58, 24 June 2023.

ncat

Ncat Users’ Guide. Nmap

Ncat is a feature-packed networking utility which reads and writes data across networks from the command line. Ncat was written for the Nmap Project as a much-improved reimplementation of the venerable Netcat. It uses both TCP and UDP for communication and is designed to be a reliable back-end tool to instantly provide network connectivity to other applications and users. Ncat will not only work with IPv4 and IPv6 but provides the user with a virtually limitless number of potential uses.

timeout 3600s ncat 153.44.253.27 5631 > ais_2024_02_14_01_15_UTC
Enter fullscreen mode Exit fullscreen mode

The messages look like this:

head -n 5 ais_2024_02_14_01_15_UTC
Enter fullscreen mode Exit fullscreen mode 
\s:2573555,c:1707873398*05\!BSVDM,1,1,,A,13mB7d001E1dDH0`KlKBjB?<0490,0*34
\s:2573105,c:1707873398*04\!B2VDM,2,1,4,B,53oHa402;QilhU1D00085=@v0TT000000000000t6`p>440Ht:j3lU1CcCCl,0*7A
\s:2573105,c:1707873398*04\!B2VDM,2,2,4,B,lh000000000,2*5F
\s:2573415,c:1707873398*00\!BSVDM,1,1,,B,13m9ERPP00Pp5opUivG8P?w:0<12,0*6F
\s:2573210,c:1707873398*03\!BSVDM,1,1,,B,13oQg60000PJMu`QhUB6D26v0D18,0*40
Enter fullscreen mode Exit fullscreen mode

Inspecting format of messages with xxd

We use xxd and sed.

ncat 153.44.253.27 5631 | xxd -p -c 1024 | sed 's/0d0a/&\n/g'
Enter fullscreen mode Exit fullscreen mode
  1. Uses ncat to receive the data.
  2. Pipes the data into xxd -p -c 1024 to convert it into a plain hex dump. The -c 1024 sets the number of columns to 1024, ensuring that each message up to 1024 bytes is processed as a single unit.
  3. Pipes to sed 's/0d0a/&\n\n/g' to search for 0d0a (the hex representation of \r\n) in the hex output and replace it with itself followed by a newline. 
ncat 153.44.253.27 5631 | xxd -p -c 1024 | sed 's/0d0a/&\n/g'
Enter fullscreen mode Exit fullscreen mode 
5c733a323537333438352c633a313730383138383437352a30465c21425356444d2c312c312c2c412c42315777743f5030354843733d3939756565416b57777634335030362c302a36460d0a
21425356444d2c312c312c2c422c31336d486751303030305049554f6a5264305250493251563044303b2c302a34330d0a
5c733a323537333333352c633a313730383138383437352a30335c21425356444d2c312c312c2c412c31336d3a6e6e3730303c30614256465477736d623d706d68303850652c302a34380d0a
5c733a323537333134352c633a313730383138383437352a30365c21425356444d2c312c312c2c412c31334d43306b4030303d303a446f4c504e44733864563f68303850652c302a33310d0a
5c733a323537333532352c633a313730383138383437352a30345c21425356444d2c312c312c2c412c48315777743f50606e3038754556333e32323232323167373e456c2c322a35360d0a
Enter fullscreen mode Exit fullscreen mode

This is a fun roundabout way of seeing that the messages are terminated with \r\n. We could use ncat -x to directly output the hex dump, but I didn't figure out a way nicely to parse the separate messages.

gpsdecode

gpsdecode

gpsdecode tool is a batch-mode decoder for NMEA and various binary packet formats associated with GPS, AIS, and differential-correction services. It produces a JSON dump on standard output from binary on standard input. The JSON is the same format documented by gpsd; this tool uses the same decoding logic as gpsd, but with a simpler interface intended for batch processing of data files.

All sensor-input formats known to the GPSD project can be decoded by this tool. These include: NMEA, AIVDM (the NMEA-derived sentence format used by AIS, the marine Automatic Identification System), RTCM2, and all supported GPS binary formats (notably including SiRF). See gpsd(8) for applicable standards and known limitations of the decoding logic.

You can use this tool with nc(1) to examine AIS feeds from AIS pooling services, RTCM feeds from RTCM receivers or NTRIP broadcasters.

The messages parsed using gpsdecode look like this:

head -n 5 ais_2024_02_14_01_15_UTC | gpsdecode
Enter fullscreen mode Exit fullscreen mode 
{"class":"AIS","device":"stdin","type":1,"repeat":0,"mmsi":257198000,"scaled":true,"status":0,"status_text":"Under way using engine","turn":0,"speed":8.5,"accuracy":false,"lon":23.662507,"lat":70.664715,"course":71.3,"heading":71,"second":38,"maneuver":0,"raim":false,"radio":16960}
{"class":"AIS","device":"stdin","type":1,"repeat":0,"mmsi":257054090,"scaled":true,"status":0,"status_text":"Under way using engine","turn":"nan","speed":0.0,"accuracy":true,"lon":12.253433,"lat":66.026820,"course":217.6,"heading":511,"second":37,"maneuver":0,"raim":false,"radio":49218}
{"class":"AIS","device":"stdin","type":1,"repeat":0,"mmsi":259551000,"scaled":true,"status":0,"status_text":"Under way using engine","turn":0,"speed":0.0,"accuracy":true,"lon":5.782060,"lat":58.998307,"course":161.6,"heading":67,"second":31,"maneuver":0,"raim":false,"radio":81992}
Enter fullscreen mode Exit fullscreen mode

jq

jq

cat ais_2024_02_14_01_15_UTC_decoded_with_timestamps| jq --slurp '.' > ais_2024_02_14_01_15_UTC_decoded_with_timestamps.json
Enter fullscreen mode Exit fullscreen mode 
jq '.[0]' ais_2024_02_14_01_15_UTC_decoded_with_timestamps.json
Enter fullscreen mode Exit fullscreen mode 
{
  "class": "AIS",
  "device": "stdin",
  "type": 1,
  "repeat": 0,
  "mmsi": 257198000,
  "scaled": true,
  "status": 0,
  "status_text": "Under way using engine",
  "turn": 0,
  "speed": 8.5,
  "accuracy": false,
  "lon": 23.662507,
  "lat": 70.664715,
  "course": 71.3,
  "heading": 71,
  "second": 38,
  "maneuver": 0,
  "raim": false,
  "radio": 16960,
  "timestamp": ""
}
Enter fullscreen mode Exit fullscreen mode

Longitude and latitude 

head -n 5 ais_2024_02_14_01_15_UTC |
gpsdecode |
jq -r '. | select(.lon != null and .lat != null) | "\(.lon) \(.lat)"'
Enter fullscreen mode Exit fullscreen mode 
23.662507 70.664715
12.253433 66.02682
5.78206 58.998307
Enter fullscreen mode Exit fullscreen mode 
cat ais_2024_02_14_01_15_UTC |
gpsdecode |
jq -r '. | select(.lon != null and .lat != null) | "\(.lon) \(.lat)"' > \
ais_2024_02_14_01_15_UTC.dat
Enter fullscreen mode Exit fullscreen mode

It is worth noting that | does not require \ to split across lines while > does.

Bounding box with awk 

awk '$1 >= 0 && $1 <= 35 && $2 >= 55 && $2 <= 80' ais_2024_02_14_01_15_UTC.dat > ais_2024_02_14_01_15_UTC_box.dat
Enter fullscreen mode Exit fullscreen mode 
wc -l ais_2024_02_14_01_15_UTC.dat
72999 ais_2024_02_14_01_15_UTC.dat
wc -l ais_2024_02_14_01_15_UTC_box.dat
72585 ais_2024_02_14_01_15_UTC_box.dat
Enter fullscreen mode Exit fullscreen mode

Gnuplot 

gnuplot> plot 'ais_2024_02_14_01_15_UTC_box.dat' using 1:2 with points
Enter fullscreen mode Exit fullscreen mode

Longitude is between 0 and 35 and latitude is between 55 and 80

GDAL ogr2ogr 

echo "Longitude,Latitude" > ais_2024_02_14_01_15_UTC_box.csv

awk '{print $1 "," $2}' ais_2024_02_14_01_15_UTC_box.dat >> ais_2024_02_14_01_15_UTC_box.csv
Enter fullscreen mode Exit fullscreen mode 
head -n 5 ais_2024_02_14_01_15_UTC_box.csv
Enter fullscreen mode Exit fullscreen mode 
Longitude,Latitude
23.662507,70.664715
12.253433,66.02682
5.78206,58.998307
2.966662,62.485987
Enter fullscreen mode Exit fullscreen mode

ogr2ogr

ogr2ogr -f "CSV" -lco GEOMETRY=AS_XY -t_srs "EPSG:32633" -s_srs "EPSG:4326" projected_output/ ais_2024_02_14_01_15_UTC_box.csv -oo X_POSSIBLE_NAMES=Longitude -oo Y_POSSIBLE_NAMES=Latitude -overwrite
Enter fullscreen mode Exit fullscreen mode 
head -n 5 projected_output/ais_2024_02_14_01_15_UTC_box.csv
X,Y,Longitude,Latitude
Enter fullscreen mode Exit fullscreen mode 
819151.812467396,7862821.57947704,23.662507,70.664715
375505.625008375,7325626.69347061,12.253433,66.02682
-28538.7387103818,6576427.56565985,5.78206,58.998307
-117577.293012299,6986142.22730701,2.966662,62.485987
Enter fullscreen mode Exit fullscreen mode

plot_ais.gp

set terminal pngcairo enhanced
set output 'ais_plot.png'
set title 'Ship Positions in Norwegian Waters'
set xlabel 'UTM Easting (kilometers)'
set ylabel 'UTM Northing (kilometers)'
set grid
set datafile separator ','
plot 'projected_output/ais_2024_02_14_01_15_UTC_box.csv' using ($1/1000):($2/1000) with points pt 7 ps 1 lc rgb 'blue' title 'Ships'
Enter fullscreen mode Exit fullscreen mode 
gnuplot plot_ais.gp
Enter fullscreen mode Exit fullscreen mode

Ship locations using the UTM 33N projection system, with distances measured in kilometers

Including ISO 8601 timestamps

https://www.w3.org/TR/NOTE-datetime

ISO 8601 UTC timestamp format YYYY-MM-DDThh:mm:ssZ

To pipe the output of ncat, we do the following:

ncat 153.44.253.27 5631 | tr -d '\r'
Enter fullscreen mode Exit fullscreen mode

Now we can compose this pipe with gpsdecode:

ncat 153.44.253.27 5631 | tr -d '\r' | gpsdecode
Enter fullscreen mode Exit fullscreen mode 
{"class":"AIS","device":"stdin","type":3,"repeat":0,"mmsi":257878000,"scaled":true,"status":0,"status_text":"Under way using engine","turn":"fastright","speed":6.0,"accuracy":false,"lon":9.639210,"lat":59.107997,"course":278.4,"heading":284,"second":30,"maneuver":0,"raim":false,"radio":7792}
{"class":"AIS","device":"stdin","type":1,"repeat":0,"mmsi":257021970,"scaled":true,"status":7,"status_text":"Engaged in fishing","turn":"nan","speed":0.0,"accuracy":false,"lon":17.902577,"lat":69.509332,"course":360.0,"heading":511,"second":32,"maneuver":0,"raim":false,"radio":27224}
{"class":"AIS","device":"stdin","type":1,"repeat":0,"mmsi":235103057,"scaled":true,"status":0,"status_text":"Under way using engine","turn":0,"speed":11.3,"accuracy":false,"lon":5.716548,"lat":63.704728,"course":206.7,"heading":204,"second":30,"maneuver":0,"raim":false,"radio":27224}
Enter fullscreen mode Exit fullscreen mode

Now we compose with awk and date (https://www.gnu.org/software/coreutils/manual/html_node/Examples-of-date.html):

ncat 153.44.253.27 5631 | tr -d '\r' | gpsdecode | awk -v date="$(date -u +%Y-%m-%dT%H:%M:%SZ)" '{print "{\"timestamp_ISO_8601\":\"" date "\"," substr($0, 2)}'
Enter fullscreen mode Exit fullscreen mode 
{"timestamp_ISO_8601":"2024-02-16T21:38:38Z","class":"AIS","device":"stdin","type":18,"repeat":0,"mmsi":257129980,"scaled":true,"reserved":0,"speed":0.0,"accuracy":true,"lon":23.329870,"lat":69.977463,"course":360.0,"heading":511,"second":39,"regional":0,"cs":false,"display":false,"dsc":true,"band":true,"msg22":true,"raim":true,"radio":609914}
{"timestamp_ISO_8601":"2024-02-16T21:38:38Z","class":"AIS","device":"stdin","type":1,"repeat":0,"mmsi":257482600,"scaled":true,"status":0,"status_text":"Under way using engine","turn":"nan","speed":0.1,"accuracy":false,"lon":5.209840,"lat":59.298867,"course":27.4,"heading":511,"second":39,"maneuver":0,"raim":false,"radio":49221}
{"timestamp_ISO_8601":"2024-02-16T21:38:38Z","class":"AIS","device":"stdin","type":5,"repeat":0,"mmsi":258033200,"scaled":true,"imo":0,"ais_version":1,"callsign":"JWCH","shipname":"NORDNES","shiptype":0,"shiptype_text":"Not available","to_bow":9,"to_stern":6,"to_port":2,"to_starboard":2,"epfd":1,"epfd_text":"GPS","eta":"02-14T20:00Z","draught":0.9,"destination":"NAVIGATION TRAINING","dte":0}
Enter fullscreen mode Exit fullscreen mode 
timeout 3600s ncat 153.44.253.27 5631 | tr -d '\r' | gpsdecode | awk -v date="$(date -u +%Y-%m-%dT%H:%M:%SZ)" '{print "{\"timestamp_ISO_8601\":\"" date "\"," substr($0, 2)}' > ais_2024_02_16T21_42_57Z
Enter fullscreen mode Exit fullscreen mode

We examine the contents of the file we created above:

head -n 2 ais_2024_02_16T21_42_57Z
Enter fullscreen mode Exit fullscreen mode 
{"timestamp_ISO_8601":"2024-02-16T21:43:35Z","class":"AIS","device":"stdin","type":1,"repeat":0,"mmsi":257827700,"scaled":true,"status":15,"status_text":"Not defined","turn":0,"speed":0.0,"accuracy":false,"lon":14.376283,"lat":67.284670,"course":342.6,"heading":48,"second":35,"maneuver":0,"raim":false,"radio":81943}
{"timestamp_ISO_8601":"2024-02-16T21:43:35Z","class":"AIS","device":"stdin","type":21,"repeat":1,"mmsi":992576156,"scaled":true,"aid_type":30,"aid_type_text":"Special Mark","name":"AQUACULTURE 4","lon":18.480670,"lat":69.895130,"accuracy":false,"to_bow":1,"to_stern":1,"to_port":1,"to_starboard":1,"epfd":7,"epfd_text":"Surveyed","second":60,"regional":0,"off_position":false,"raim":false,"virtual_aid":true}
Enter fullscreen mode Exit fullscreen mode

Because the interrupt from timeout 3600s did not coincide with a complete message from ncat, the tail of the file looks like:

tail -n 1 ais_2024_02_16T21_42_57Z
Enter fullscreen mode Exit fullscreen mode 
{"timestamp_ISO_8601":"2024-02-16T21:43:35Z","class":"AIS","devi
Enter fullscreen mode Exit fullscreen mode

Thus we omit the last line of the file, and then parse as JSON using jq --slurp:

head -n -1 ais_2024_02_16T21_42_57Z | jq --slurp '.' > ais_2024_02_16T21_42_57Z.json
Enter fullscreen mode Exit fullscreen mode

We inspect the first element:

jq '.[0]' ais_2024_02_16T21_42_57Z.json
Enter fullscreen mode Exit fullscreen mode 
{
  "timestamp_ISO_8601": "2024-02-16T21:43:35Z",
  "class": "AIS",
  "device": "stdin",
  "type": 1,
  "repeat": 0,
  "mmsi": 257827700,
  "scaled": true,
  "status": 15,
  "status_text": "Not defined",
  "turn": 0,
  "speed": 0,
  "accuracy": false,
  "lon": 14.376283,
  "lat": 67.28467,
  "course": 342.6,
  "heading": 48,
  "second": 35,
  "maneuver": 0,
  "raim": false,
  "radio": 81943
}
Enter fullscreen mode Exit fullscreen mode

Now we use jq to convert this to the GeoJSON format (RFC 7946).

jq '{type: "FeatureCollection", features: map({type: "Feature", geometry: {type: "Point", coordinates: [.lon, .lat]}, properties: .})}' ais_2024_02_16T21_42_57Z.json > ais_2024_02_16T21_42_57Z.geojson
Enter fullscreen mode Exit fullscreen mode

ais_2024_02_16T21_42_57Z.geojson on ArcGIS Online

Map view on ArcGIS Online

ais_2024_02_16T21_42_57Z.geojson on GitHub

Ais Decoder by Neal Arundale

Ais Decoder by Neal Arundale

Ais Decoder Options:

Ais Decoder Options

Ais Decoder main menu:

Ais Decoder main menu

Ais Decoder Nmea input:

Ais Decoder Nmea input

Ais Decoder parsed Summary:

Ais Decoder parsed Summary

Ais Decoder NMEA sentence Detail:

Ais Decoder NMEA sentence Detail

Compare to gpsdecode:

echo '!BSVDM,1,1,,B,13mL5v0001PEjC6Rsu:Pca`L0@9W,0*59,2/14/2024 6:36:17 PM' | gpsdecode | jq --slurp '.' | jq '.[0]'
Enter fullscreen mode Exit fullscreen mode 
{
  "class": "AIS",
  "device": "stdin",
  "type": 1,
  "repeat": 0,
  "mmsi": 257361400,
  "scaled": true,
  "status": 0,
  "status_text": "Under way using engine",
  "turn": 0,
  "speed": 0.1,
  "accuracy": true,
  "lon": 4.759205,
  "lat": 61.056497,
  "course": 17.4,
  "heading": 308,
  "second": 14,
  "maneuver": 0,
  "raim": false,
  "radio": 66151
}
Enter fullscreen mode Exit fullscreen mode
💖 💪 🙅 🚩
jordanbell2357
Jordan Bell

Posted on February 14, 2024

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

gpsdecode and ogr2ogr for AIS messages
tutorial gpsdecode and ogr2ogr for AIS messages

February 14, 2024