Feeds:
Posts
Comments

Archive for the ‘Technology’ Category

Accessing External APIs

Introduction

In my previous post we saw how to connect to the Hive AlertMe API to track the variation of temperature in a home over time to produce a simple graph.  In this post we are going to correlate these readings with the external temperature as recorded by the UK Met Office.

DataPoint

The Met Office provides a publicly accessible API for both weather forecasts and historic meteorological observations called DataPoint.  To access the API you need to register for an account here http://www.metoffice.gov.uk/datapoint.  The account includes the provision of a personal API key which is required to call the API.

The basic service is free with quotas being imposed on the total number of calls  and the maximum frequency of these calls.  The quota is more than enough for individual usage.  DataPoint is a restful API providing data as either XML or JSON format.  The API is fully documented here, http://www.metoffice.gov.uk/datapoint/support/api-reference.  As before when exploring the Hive AlertMe API, DataPoint can be easily investigated using the Postman App within Google Chrome.

Sites, Forecasts and Observations 

The API is designed to support the access of both forecast and observation data for a number of UK locations or in DataPoint API likes parlance, sites.  The Met Office provides forecast data for around 5,000 sites and collects observations at approximately 120 sites.  Here we will be using the JSON based API.

Two API functions are provided by DataPoint for obtaining the list of available sites, one for forecasts and one for observations.  Substitute your own API key to make these work.

http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/json/sitelist?key=API-KEY
http://datapoint.metoffice.gov.uk/public/data/val/wxobs/all/json/sitelist?key=API-KEY

As we are interested in tracking historic weather conditions we will use the latter API function, which will return a result similar to the following (this have been abridged and only shows the first entry of the hundred and twenty)

{“Locations”:
    {“Location”:[
        {“elevation”:”7.0″,
        “id”:”3066″,
        “latitude”:”57.6494″,
        “longitude”:”-3.5606″,
        “name”:”Kinloss”,
        “region”:”gr”,
        “unitaryAuthArea”:”Moray”},
        …]
      }
}

You can see the basic information returned includes name, elevation (I have been unable to confirm the units but this might be in feet as wind speeds elsewhere in the API are reported in miles per hour an imperial unit) plus latitude and longitude.  We use this call to identify the sites we are interested in and in particular their ids.

A Weighty Problem

Because observation coverage is sparse when compared to forecast locations my home town is not listed which it is for forecast data.  Since we wish to track historic temperatures we will calculate the weighted average of temperature at the three nearest sites to my home town based on distance.  If there are any meteorologists reading this, I am open to suggestions on if there is a better approach given the data available?

weighted-temperature

In the above diagram we can see three separate locations, shown by grey diamonds, with the three temperatures 3.5C, 4.0C and 5.0C and the corresponding distances 10, 12 and 11 respectively.

The temperature at ‘Home’ is calculated as follows,

t1 × 1/d1 + t2 × 1/d2 + t3 × 1/d3
–––––––––––––––––––––––
1/d1 + 1/d2 + 1/d3

3.5 × 0.1 + 4.0 × 0.0833 + 5.0 × 0.0909
––––––––––––––––––––––––––
0.1 + 0.0833 + 0.0909

0.35 + 0.3333 + 0.4545
–––––––––––
0.2745

= 4.15C

Observations

The three nearest Met Office locations to my home town are Charlwood, Gravesend-Broadness and Kelly which have the site ids 3769, 3784 and 3781 respectively.

met-office-obs-4th-feb-2017

The above chart shows the variation of temperature for these three sites over a 24-hour period plus the calculated weighted temperature for my home town shown in red.  The data was obtained by calling the DataPoint API three times with the site id of each location.  The data is obtained using the following API call,

http://datapoint.metoffice.gov.uk/public/data/val/wxobs/all/json/3784?res=hourly&key=API-KEY

In the above the site id is incorporated into the URL, here 3784 for Gravesend-Broadness, and the res parameter requests that the data is provided on an hourly basis.  Currently hourly observations are the finest resolution for observational data.  Below is an example of the response provided.

{“SiteRep”:
    {“Wx”:
        {“Param”:[
             {“name”:”G”,”units”:”mph”,”$”:”Wind Gust”},
             {“name”:”T”,”units”:”C”,”$”:”Temperature”},
             {“name”:”V”,”units”:”m”,”$”:”Visibility”},
             {“name”:”D”,”units”:”compass”,”$”:”Wind Direction”},
             {“name”:”S”,”units”:”mph”,”$”:”Wind Speed”},
             {“name”:”W”,”units”:””,”$”:”Weather Type”},
             {“name”:”P”,”units”:”hpa”,”$”:”Pressure”},
             {“name”:”Pt”,”units”:”Pa\/s”,”$”:”Pressure Tendency”},
             {“name”:”Dp”,”units”:”C”,”$”:”Dew Point”},
             {“name”:”H”,”units”:”%”,”$”:”Screen  Relative Humidity”}
             ]
        },
    “DV”:
        {“dataDate”:”2017-01-29T19:00:00Z”,
         “type”:”Obs”,
         “Location”:
               {“i”:”3784″,
               “lat”:”51.464″,
               “lon”:”0.314″,
               “name”:”GRAVESEND-BROADNESS”,
               “country”:”ENGLAND”,
               “continent”:”EUROPE”,
               “elevation”:”3.0″,
               “Period”:[
                    {“type”:”Day”,
                    “value”:”2017-01-28Z”,
                    “Rep”:[
                          {“D”:”SSW”,”H”:”80.9″,”P”:”1012″,”S”:”10″,
                          “T”:”6.1″,”Pt”:”R”,”Dp”:”3.1″,”$”:”1140″},
                          {“D”:”SSW”,”H”:”83.8″,”P”:”1013″,”S”:”9″,
                          “T”:”5.7″,”Pt”:”R”,”Dp”:”3.2″,”$”:”1200″}, 
                          {“D”:”S”,”H”:”87.9″,”P”:”1014″,”S”:”8″,
                          “T”:”4.7″,”Pt”:”R”,”Dp”:”2.9″,”$”:”1260″},
                          {“D”:”SSW”,”H”:”83.7″,”P”:”1014″,”S”:”9″,
                          “T”:”5.0″,”Pt”:”R”,”Dp”:”2.5″,”$”:”1320″},
                          {“D”:”SSW”,”H”:”83.0″,”P”:”1014″,”S”:”8″,
                          “T”:”4.3″,”Pt”:”R”,”Dp”:”1.7″,”$”:”1380″}]
                    },
                    {“type”:”Day”,
                    “value”:”2017-01-29Z”,
                    “Rep”:[
                          {“D”:”S”,”H”:”86.7″,”P”:”1014″,”S”:”7″,
                         “T”:”4.0″,”Pt”:”R”,”Dp”:”2.0″,”$”:”0″},
                          {“D”:”SSW”,”H”:”87.4″,”P”:”1013″,”S”:”10″,
                         “T”:”4.9″,”Pt”:”F”,”Dp”:”3.0″,”$”:”60″},
                          {“D”:”SSW”,”H”:”85.0″,”P”:”1013″,”S”:”13″,
                         “T”:”5.3″,”Pt”:”F”,”Dp”:”3.0″,”$”:”120″},
                          {“D”:”SSW”,”H”:”86.2″,”P”:”1013″,”S”:”9″,
                         “T”:”5.3″,”Pt”:”F”,”Dp”:”3.2″,”$”:”180″},
                          {“D”:”SSW”,”H”:”88.6″,”P”:”1013″,”S”:”6″,
                         “T”:”4.8″,”Pt”:”R”,”Dp”:”3.1″,”$”:”240″},
                          {“D”:”SW”,”H”:”87.4″,”P”:”1013″,”S”:”8″,
                         “T”:”4.6″,”Pt”:”R”,”Dp”:”2.7″,”$”:”300″},
                          {“D”:”SSW”,”H”:”93.8″,”P”:”1014″,”S”:”7″,
                         “T”:”3.2″,”Pt”:”R”,”Dp”:”2.3″,”$”:”360″},
                          {“D”:”S”,”H”:”92.5″,”P”:”1014″,”S”:”7″,
                         “T”:”3.5″,”Pt”:”R”,”Dp”:”2.4″,”$”:”420″},
                          {“D”:”S”,”H”:”93.2″,”P”:”1014″,”S”:”8″,
                         “T”:”4.1″,”Pt”:”R”,”Dp”:”3.1″,”$”:”480″},
                          {“D”:”S”,”H”:”91.3″,”P”:”1014″,”S”:”8″,
                         “T”:”5.8″,”Pt”:”R”,”Dp”:”4.5″,”$”:”540″},
                          {“D”:”S”,”H”:”88.2″,”P”:”1014″,”S”:”11″,
                         “T”:”7.0″,”Pt”:”R”,”Dp”:”5.2″,”$”:”600″},
                          {“D”:”SSW”,”H”:”85.2″,”P”:”1014″,”S”:”13″,
                         “T”:”7.6″,”Pt”:”F”,”Dp”:”5.3″,”$”:”660″},
                          {“D”:”SSW”,”H”:”82.0″,”P”:”1013″,”S”:”10″,
                         “T”:”9.7″,”Pt”:”F”,”Dp”:”6.8″,”$”:”720″},
                          {“D”:”S”,”H”:”77.5″,”P”:”1013″,”S”:”11″,
                         “T”:”9.3″,”Pt”:”F”,”Dp”:”5.6″,”$”:”780″},
                          {“D”:”S”,”H”:”77.0″,”P”:”1012″,”S”:”11″,
                         “T”:”9.3″,”Pt”:”F”,”Dp”:”5.5″,”$”:”840″},
                          {“D”:”S”,”H”:”81.9″,”P”:”1012″,”S”:”13″,
                         “T”:”8.8″,”Pt”:”F”,”Dp”:”5.9″,”$”:”900″},
                          {“D”:”S”,”H”:”84.7″,”P”:”1011″,”S”:”10″,
                         “T”:”8.4″,”Pt”:”F”,”Dp”:”6.0″,”$”:”960″},
                          {“D”:”SE”,”H”:”90.7″,”P”:”1010″,”S”:”9″,
                         “T”:”7.6″,”Pt”:”F”,”Dp”:”6.2″,”$”:”1020″},
                          {“D”:”SE”,”H”:”92.6″,”P”:”1010″,”S”:”11″,
                         “T”:”7.4″,”Pt”:”F”,”Dp”:”6.3″,”$”:”1080″}]
                 }]
             }
         }
     }
}

The temperature values that we are interested in are highlighted in bold.

Pulling It All Together

Using the data gathered from the Hive AlertMe API and the Met Office DataPoint API we can see the correlation between the two over a 24-hour period.  I extended the simple Java client from last time to call the DataPoint API to generate the dataset used to produce the following graph.

internal-external-temperature-08022017

The red line at the top represents the internal temperature gathered via the Hive AlertMe API and the line at the bottom is the weighted external temperature gathered from the three nearest Met Office observation sites.  Note that no data was gathered from the Hive AlertMe API  prior to 11:00 AM on 8th February.  Also notice that the heating was switched off from around midnight to 6:00 AM where you can see the temperature fall.  Interesting to note the ripple effect from about 16:00 to 23:00 as the temperature is maintained at the 20C set value.

Over the next couple of weeks I plan to capture a much larger dataset which can be analysed for trends and correlations.

In the meantime in my next blog we will explore how to build a Graph Database on a Relational Database, some of the challenges that this provides and the benefits this gives.

Advertisements

Read Full Post »

An exploration of connected devices

Introduction

Recently chez Architect has acquired a number of Internet of Things (IoT) devices, from a couple of Sonos wireless music speakers to more recently Hive, British Gas’s wireless home heater thermostat and boiler management system.  If you are familiar with Nest you get the idea.  More than 360,000 units have been installed across the UK alone and the rate of adoption appears to be increasing according to Nina Bhatia Managing Director of Centrica’s Connected Home Unit.

Hive has a number of features for controlling the heating and hot water in your home.  It comes with a fancy mobile app (I was impressed by the UX) and website for controlling your appliances remotely.  You can even set up a schedule for when devices are on or off.  The app allows you to adjust the temperature of your heating and also reports the current temperature in your home.  This article is not intended to be a review of Hive but an exploration of the APIs that these types of ecosystems provide.

After seeing the mobile app I realised that there must be an API to support it.  It is not widely advertised, or promoted by British Gas, but there are a number of blogs that make reference to it.  The original company that developed and then operated the Hive ecosystem was called AlertMe.  AlertMe was acquired by Centrica the parent of British Gas in 2015.  Slightly out of date documentation for the API can be found here http://www.smartofthehome.com/wp-content/uploads/2016/03/AlertMe-API-v6.1-Documentation.pdf.  This is the documentation for V6.1 of the API although the most current version is V6.5.  I’ve been unable to locate the latest documentation so here is a gentle appeal to Centrica to release it.

Hive supports a Restful JSON API, so I thought that I would write a simple application that would allow me to record the historic temperature within the home using the API.

Logging into Hive

The base URL for the Hive API is, https://api-prod.bgchprod.info:443/omnia.  In order to use this interface clients, users, need to be authenticated using the user-id and password setup for the device.  Once authenticated a 24-hour access token is provided that is used for all subsequent calls.  In order to experiment with the API I would recommend using the Postman App available for use with the Google Chrome browser.

To use the restful API a number of attributes need to be set in the HTTP header. For the initial client authentication call, [POST] https://api-prod.bgchprod.info:443/omnia/auth/sessions, three header attributes need to be set with the following values.

Content-Type: application/vnd.alertme.zoo-6.5+json
Accept: application/vnd.alertme.zoo-6.5+json
X-Omnia-Client: Hive Web Dashboard

Along with setting the header attributes, the POST request requires data to be passed in the HTTP body as a JSON object.  Below is an example of the request body.

{
    “sessions”: [{
        “username”: “xxxxxx”,
        “password”: “yyyyyy”,
        “caller”: “WEB”
    }]
}

Looking at the sessions structure it looks like the call can be used to establish multiple sessions in a single call but I have not tried this as I do not have access to multiple user accounts.

Once posted, the AlertMe API will respond by returning a JSON object in the HHTP body. Below is an example response

{
  “meta”: {},
  “links”: {},
  “linked”: {},
  “sessions”: [
    {
      “id”: “…”,
      “links”: {},
      “username”: “xxxxxxx”,
      “userId”: “vvvvvvv”,
      “extCustomerLevel”: 1,
      “latestSupportedApiVersion”: “6”,
      “sessionId”: “…”
    }
  ]
}

The session id is valid for 24 hours and is used in all subsequent API calls and is added as a fourth parameter to the HTTP header with the key X-Omnia-Access-Token.

Accessing Hive

Hive provides a number of function calls such as those for accessing the topology of your devices and discovering their status, in Hive terminology devices are called nodes.

The API function [GET] https://api-prod.bgchprod.info:443/omnia/nodes returns a JSON structure with all the details of the device nodes.  It is a bit of a kitchen sink function returning many details such as the thermostat settings and the schedule settings for each node.  When a node id is specified the results of a single node can be returned.  For instance by using [GET] https://api-prod.bgchprod.info:443/omnia/nodes/ the following structure is returned (collapsed).

{
  “meta”: {},
  “links”: {},
  “linked”: {},
  “nodes”: [
    {
      “id”: “ … ”,
      “href”: “…://api.prod.bgchprod.info:8443/omnia/nodes/”,
      “links”: {},
      “name”: “Thermostat”,
      “nodeType”: “…/node.class.thermostat.json#”,
      “parentNodeId”: “ … ”,
      “lastSeen”: 1485983701626,
      “createdOn”: 1481892545442,
      “userId”: “vvvvvvv”,
      “ownerId”: “vvvvvvv”,
      “features”: {
        “transient_mode_v1”: { …. },
        “temperature_sensor_v1”: {
          “temperature”: {
            “reportedValue”: 20.35,
            “displayValue”: 20.35,
            “reportReceivedTime”: 1485983701626,
            “reportChangedTime”: 1485983461820
          },
        “featureType”: { …. },
        },
        “on_off_device_v1”: { …. },
        “heating_thermostat_v1”: { …. },
        “thermostat_frost_protect_v1”: { …. }
      }
    }
  ]
}

By examining the nodes structure it is possible to determine the currently reported temperature of the thermostats in your home.  The time values are Unix timestamps representing seconds since 1st January 1970.

Observations

AlertMe appears to sample Hive devices on a two minute interval so there is no benefit in calling the API more frequently.  Sampling is also unreliable so updates to the sample values may be missed.  I have observed periods of up to eight minutes where the same value was reported before a revised sample update was successfully received.

The precision of the thermostat value is high to two decimal places which was a surprise.  I am not convinced the temperature reading is that accurate, but it was refreshing obtaining values to a fraction of a degree.

Hive Analytics

Now the real reason for why I was interested in accessing the AlertMe API for Hive was to build a picture of how temperature varies over time.

I wrote a simple Java application that logged in and polled the state of the devices in my house on a two minute basis outputting the results to a CSV file initially.  I used the GSON JSON parser which I found very straight forward and intuitive to use.

Below is a graph of temperature variation over twenty four hours.  The left axis is temperature in Celsius and the bottom axis is date and time.  You can see where the temperature dips when the heating was switched off over night and also during the day.

hive_temperature_chart

In my next blog I will describe how these values can be correlated to historical weather temperatures provided by the Met Office through their DataPoint API.

Read Full Post »

This is a departure from the usual blog entry, but I thought I would share the findings from a little research that I had been conducting recently on tech start-ups near Silicon Roundabout.  I wanted to get a feel for the number and location of these companies.

For international readers, Silicon Roundabout, is the name given to a small and growing tech cluster that has started to mushroom around Old Street roundabout in London.  Start-ups are attracted to this area of London due to the relatively affordable office rental whilst still being in the centre of London.  A number of large international firms such as Google have also been attracted to the area to be at the centre of this cluster.  Google have recently opened their London campus here.

Silicon Roundabout Map Image

Above is an interactive map pinpointing the companies that are within a mile (1.6 km) of Silicon Roundabout.  Clicking each pin reveals details of each company including a link to its web-site.

Although the survey was not especially scientific, a couple of Google searches, I did manage to identify approximately thirty companies.  The data is here to share in the attached Excel file. Tech Companies Survey 2012.

Based on the sample the majority of companies have been created in the last four years.

Silicon Roundabout Start-ups Bar-chart

In constructing the survey I also categorised each company fairly arbitrarily, resulting in the following table showing the frequency of each category.

Analytics

6

Visualization or presentation of data.
Finance

5

Retail financial services.
Fashion

4

Fashion retail.
Service

4

Provision of an end-user service such as booking a cab.
Cloud

2

Cloud provided services.
Software

2

Software companies.
Travel

2

Travel sites.
Advertising

1

Advertising and marketing.
Food

1

Food retail.
Games

1

Games vendors.
Mobile

1

Mobile phone related services.
Retail

1

General retail.
Security

1

Computer and web security.

Although the sample is not big, Analytics and Finance are at the top accounting for a third of start-ups.

I’m unsure what can be concluded with this data, all thoughts are welcome, however I thought I would share what I had found.

Read Full Post »