Data fra E39-prosjekt (Data collection from E39-project)

MET has a collection of survey data provided by the Norwegian road administration. This data is primarily sampled from a selection of fjords in Sunnmøre in regards to the 'ferry-less' E39 road plan.
The data is stored in a common network common dataform (netCDF) format, which enables multidimensional variables. It is a widely used data format for scientific data. Every .nc file has a myriad of variables, all of which has descriptive meta-data.

Since it is difficult to publish multi-dimensional data through MQTT, some compromises have been made. For example, the measurement of salinity is stored and accessed with regards to both time and depth. I.e to get a value for salinity, depth and time must be provided. Time can be provided through timestamp, but depth has to be set to a static variable. In this case the depth is set to be always 5 meters.

The script called XarrayBuouData.py fetches every .nc file on MET buoy data section of E39 and parses through all variables and stores their keys in an array. A separate function handles connection and publishing through MQTT. Below is a code-snippet which shows how to select and publish a variable.

json_data = json.dumps(kwargs, indent=2)

if 'adcp' in locationname:
                publishMqtt(json_data,'acdcp')
            elif 'aquadopp' in locationname:
                publishMqtt(json_data,'aquadopp')
            elif 'ctd' in locationname:
                publishMqtt(json_data, 'ctd')
            elif 'raw_wind' in locationname:
                publishMqtt(json_data, 'raw')
            elif 'wave' in locationname:
                publishMqtt(json_data, 'wave')
            elif 'wind' in locationname:
                publishMqtt(json_data, 'wind')

The 'adcp', 'aquadopp' and other tags makes sure the data gets sent to the correct data converter on Thingsboard. A sample json package sent by the script to the server might look something like this

E39_F_Vartdalsfjorden_wind
Json Data:  {
  "time": 1553426576000.0,
  "latitude": 62.22162247,
  "longitude": 5.89998245,
  "WindSpeed": 4.511720180511475,
  "WindGust": 7.96875,
  "WindDirection": 147.3046875,
  "deviceName": "E39_F_Vartdalsfjorden_wind"
}

The catalog

To fetch data, the script uses the system clock and date to dynamically create a URL to the MET-database containing .nc files for all the local buoy stations.

'http://thredds.met.no/thredds/catalog/obs/buoy-svv-e39/' + year + '/' + month + '/' + 'catalog.html'

Below is a sample screenshot of how the catalog looks at the day of writing.

Using the python package BeautifulSoup, the URL to each individual file is placed in an array. The array is then passed to xarray which opens the dataset for every url and loads all variable keys.

Data Variables

h3. time
standard_name: time
long_name: Time of measurement
axis: T
comment: End of the 10 minute sampling period
_ChunkSizes: 512

h3. latitude
units: degree_north
valid_max: 90.0
valid_min: -90.0
standard_name: latitude
_ChunkSizes: 6

h3. longitude
units: degree_east
valid_max: 180.0
valid_min: -180.0
standard_name: longitude
_ChunkSizes: 6

h3. WindSpeed
long_name: Wind speed
valid_range: [ 0. 60.]
standard_name: wind_speed
units: m s-1
_ChunkSizes: 6

h3. WindGust
long_name: 3 second Wind Gust
valid_range: [ 0. 80.]
standard_name: wind_speed_of_gust
units: m s-1
_ChunkSizes: 6

h3. WindDirection
long_name: Wind Direction
valid_range: [ 0. 360.]
standard_name: wind_from_direction
units: degree
_ChunkSizes: 6

h3. time
standard_name: time
long_name: Time of measurement
axis: T
comment: End of the 10 minute sampling period
_ChunkSizes: 512

h3. latitude
units: degree_north
valid_max: 90.0
standard_name: latitude
valid_min: -90.0
_ChunkSizes: 6

h3. longitude
units: degree_east
valid_max: 180.0
standard_name: longitude
valid_min: -180.0
_ChunkSizes: 6

h3. Hm0
long_name: Significant wave height estimate from spectrum
valid_range: [ 0. 25.]
standard_name: sea_surface_wave_significant_height
units: m
_ChunkSizes: 6

h3. hm0a
standard_name: sea_surface_wave_significant_height
long_name: Low frequency band Significant wave height estimated from spectrum
units: m
valid_range: [ 0. 25.]
_ChunkSizes: 6

h3. hm0b
standard_name: sea_surface_wave_significant_height
long_name: High frequency band Significant wave height estimated from spectrum
units: m
valid_range: [ 0. 25.]
_ChunkSizes: 6

h3. hmax
long_name: Maximum individual wave height from zero crossing analysis
valid_range: [ 0. 25.]
standard_name: sea_surface_wave_maximum_height
units: m
_ChunkSizes: 6

h3. tm02
standard_name: sea_surface_wave_mean_period_from_variance_spectral_density_second_frequency_moment
long_name: Mean wave period estimated from 0th and 2nd moment of spectrum
units: s
valid_range: [ 0. 30.]
_ChunkSizes: 6

h3. tm02a
standard_name: sea_surface_wave_mean_period_from_variance_spectral_density_second_frequency_moment
long_name: Low frequency mean wave period estimated from 0th and 2nd moment of spectrum
valid_range: [ 0. 30.]
units: s
_ChunkSizes: 6

h3. tm02b
standard_name: sea_surface_wave_mean_period_from_variance_spectral_density_second_frequency_moment
long_name: High frequency mean wave period estimated from 0th and 2nd moment of spectrum
units: s
valid_range: [ 0. 30.]
_ChunkSizes: 6

h3. tp
units: s
long_name: Spectral peak wave period
valid_range: [ 0. 30.]
standard_name: sea_surface_wave_period_at_variance_spectral_density_maximum
_ChunkSizes: 6

h3. thmax
long_name: Period of highest individual wave from zero crossing analysis
valid_range: [ 0. 30.]
standard_name: sea_surface_wave_zero_upcrossing_period
units: s
_ChunkSizes: 6

h3. tm01
standard_name: sea_surface_wave_mean_period_from_variance_spectral_density_first_frequency_moment
long_name: Mean wave period estimated from 0th and 1st moment of spectrum
units: s
valid_range: [ 0. 30.]
_ChunkSizes: 6

h3. thhf
long_name: Wave direction of wind sea (from)
units: degree
standard_name: sea_surface_wind_wave_from_direction
valid_range: [ 0. 360.]
_ChunkSizes: 6

h3. mdir
standard_name: sea_surface_wave_from_direction
long_name: Mean wave direction (from) calculated from directional spectrum
units: degree
valid_range: [ 0. 360.]
_ChunkSizes: 6

h3. mdira
standard_name: sea_surface_wave_from_direction
long_name: Low frequency mean wave direction (from) calculated from directional spectrum
units: degree
valid_range: [ 0. 360.]
_ChunkSizes: 6

h3. mdirb
standard_name: sea_surface_wave_from_direction
long_name: High frequency mean wave direction (from) calculated from directional spectrum
units: degree
valid_range: [ 0. 360.]
_ChunkSizes: 6

h3. thtp
long_name: Spectral peak wave direction
units: degree
valid_range: [ 0. 360.]
_ChunkSizes: 6

h3. sprtp
long_name: Spectral peak wave spreading
units: degree
valid_range: [ 0. 360.]
_ChunkSizes: 6

h3. time
standard_name: time
long_name: Time of measurement
axis: T
comment: Sampling twice every second
_ChunkSizes: 512

h3. latitude
units: degree_north
valid_max: 90.0
valid_min: -90.0
standard_name: latitude
_ChunkSizes: 480

h3. longitude
units: degree_east
valid_max: 180.0
valid_min: -180.0
standard_name: longitude
_ChunkSizes: 480

h3. RawWindSpeed
long_name: Raw Wind Speed
valid_range: [ 0. 60.]
standard_name: wind_speed
units: m s-1
_ChunkSizes: 900

h3. RawWindDir
long_name: Raw Wind Direction
valid_range: [ 0. 360.]
standard_name: wind_from_direction
units: degree
_ChunkSizes: 900

h3. time
standard_name: time
long_name: Time of measurement
axis: T
comment: End of the 10 s sampling period
_ChunkSizes: 512

h3. depth
standard_name: depth
long_name: Depth of measurement
positive: down
units: m
axis: Z
reference: sea_level
coordinate_reference_frame: urn:ogc:def:crs:EPSG::CRF 5831

h3. latitude
units: degree_north
valid_max: 90.0
valid_min: -90.0
standard_name: latitude
_ChunkSizes: 360

h3. longitude
units: degree_east
valid_max: 180.0
valid_min: -180.0
standard_name: longitude
_ChunkSizes: 360

h3. WaterTemp
long_name: Water Temperature at depth
standard_name: sea_water_temperature
units: degree_Celsius
valid_range: [ 0. 25.]
_ChunkSizes: [1 4]

h3. Salinity
standard_name: sea_water_salinity
valid_range: [10. 40.]
long_name: Salinity at depth
units: 1e-3
_ChunkSizes: [1 4]

h3. Conductivity
long_name: Conductivity at depth
units: mS cm-1
standard_name: sea_water_electrical_conductivity
valid_range: [10. 50.]
_ChunkSizes: [1 4]

h3. Pressure
units: dbar
standard_name: sea_water_pressure
long_name: Sea water pressure at depth
_ChunkSizes: [1 4]

h3. time
standard_name: time
long_name: Time of measurement
axis: T
comment: End of the 60 s sampling period
_ChunkSizes: 512

h3. latitude
units: degree_north
valid_max: 90.0
standard_name: latitude
valid_min: -90.0
_ChunkSizes: 60

h3. longitude
units: degree_east
valid_max: 180.0
standard_name: longitude
valid_min: -180.0
_ChunkSizes: 60

h3. currSp001
long_name: Current Speed at 1 m depth
standard_name: sea_water_speed
units: cm s-1
valid_range: [ 0. 185.]
_ChunkSizes: 60

h3. currDir001
long_name: Current Direction at 1m depth
units: degree
standard_name: direction_of_sea_water_velocity
valid_range: [ 0. 360.]
_ChunkSizes: 60

h3. WaterTempAqd
units: degree_Celsius
standard_name: sea_water_temperature
long_name: Water temperature at 1 m depth
valid_range: [ 0. 25.]
_ChunkSizes: 60

h3. time
standard_name: time
long_name: Time of measurement
axis: T
comment: End of the 60 s sampling period
_ChunkSizes: 512

h3. latitude
units: degree_north
valid_max: 90.0
valid_min: -90.0
standard_name: latitude
_ChunkSizes: 60

h3. longitude
units: degree_east
valid_max: 180.0
valid_min: -180.0
standard_name: longitude
_ChunkSizes: 60

h3. depth
standard_name: depth
long_name: Depth of measurement
positive: down
units: m
axis: Z
reference: sea_level
coordinate_reference_frame: urn:ogc:def:crs:EPSG::CRF 5831

h3. currSp
units: cm s-1
valid_range: [1.00e-03 1.98e+02]
standard_name: sea_water_speed
long_name: Current speed at depth
_ChunkSizes: [ 1 20]

h3. currDir
units: degree
valid_range: [ 0. 360.]
standard_name: direction_of_sea_water_velocity
long_name: Current direction (toward) at depth
_ChunkSizes: [ 1 20]

h3. currErr
long_name: Current Error at depth
standard_name: sea_water_speed
units: cm s-1
valid_range: [-999. 999.]
_ChunkSizes: [ 1 20]