Data fra E39-prosjekt (Data collection from E39-project)

MET has a collection of survey data provided by the Norwegian road administration. This data is primarily sampled from a selection of fjords in Sunnmøre in regards to the 'ferry-less' E39 road plan.
The data is stored in a common network common dataform (netCDF) format, which enables multidimensional variables. It is a widely used data format for scientific data. Every .nc file has a myriad of variables, all of which has descriptive meta-data.

Since it is difficult to publish multi-dimensional data through MQTT, some compromises have been made. For example, the measurement of salinity is stored and accessed with regards to both time and depth. I.e to get a value for salinity, depth and time must be provided. Time can be provided through timestamp, but depth has to be set to a static variable. In this case the depth is set to be always 5 meters.

The script called XarrayBuouData.py fetches every .nc file on MET buoy data section of E39 and parses through all variables and stores their keys in an array. A separate function handles connection and publishing through MQTT. Below is a code-snippet which shows how to select and publish a variable.

json_data = json.dumps(kwargs, indent=2)

if 'adcp' in locationname:
                publishMqtt(json_data,'acdcp')
            elif 'aquadopp' in locationname:
                publishMqtt(json_data,'aquadopp')
            elif 'ctd' in locationname:
                publishMqtt(json_data, 'ctd')
            elif 'raw_wind' in locationname:
                publishMqtt(json_data, 'raw')
            elif 'wave' in locationname:
                publishMqtt(json_data, 'wave')
            elif 'wind' in locationname:
                publishMqtt(json_data, 'wind')

The 'adcp', 'aquadopp' and other tags makes sure the data gets sent to the correct data converter on Thingsboard.  A sample json package sent by the script to the server might look something like this 

E39_F_Vartdalsfjorden_wind
Json Data:  {
  "time": 1553426576000.0,
  "latitude": 62.22162247,
  "longitude": 5.89998245,
  "WindSpeed": 4.511720180511475,
  "WindGust": 7.96875,
  "WindDirection": 147.3046875,
  "deviceName": "E39_F_Vartdalsfjorden_wind"
}

The catalog

To fetch data, the script uses the system clock and date to dynamically create a URL to the MET-database containing .nc files for all the local buoy stations.  

'http://thredds.met.no/thredds/catalog/obs/buoy-svv-e39/' + year + '/' + month + '/' + 'catalog.html' 

Below is a sample screenshot of how the catalog looks at the day of writing.

Using the python package BeautifulSoup, the URL to each individual file is placed in an array. The array is then passed to xarray which opens the dataset for every url and loads all variable keys.