THIS BLOGPOST WAS UPDATED ON DECEMBER 29, 2020, BECAUSE THE OLD API DIDN’T WORK ANYMORE. FROM NOW ON AN OFFICIAL GOOGLE API CAN BE USED. CHANGES WERE MARKED IN RED ON DECEMBER 29, 2020.

My method to create a map with the most recent dates Google captured Streetview images consists of some different steps. In short it comes down to querying Google for every point on the map if Streetview imagery exists and putting all the answers on a map. The technical explanation on how I did this, can be found below.

The streets layer

Because usually Streetview images are captured in public streets, I start with a layer of all streets in town. Luckily there are different data sources available who distribute this kind of data as open data.

I used the Road Connections Layer (Wvb) of the Grootschalig ReferentieBestand (GRB) from Informatie Vlaanderen. This layer can be downloaded free and legally from download.agiv.be. You could also use OpenStreetMap Data. More information about that you can find on wiki.openstreetmap.org.

From the download application of Informatie Vlaanderen (formerly known as AGIV) I only downloaded the data for the area I wanted to use, but with QGIS (qgis.org) you can als perform a clip on a larger data set.

A layer with points to check

A streets layer is usually a polyline layer, but for a request to Google I need XY-coordinates from a point. Therefore I needed to convert the streets layer to a points layer. To do this I used QGIS and the “Locate points along lines” plugin that can be downloaded from the official QGIS plugin repository. With this plugin I placed a point every 5 meter along all streets and saved the result as a new points layer. In this layer I added 2 columns with the X and Y coordinates in WGS84. It’s important to use a point instead of a comma for decimal numbers, because that’s what Google expects. I also added a field pointID with an unique ID number for each point.

Create a MySQL table

As database I use MySQL because I use PHP for my requests to Google and PHP and MySQL go well together. But you could also use PostgreSQL or any other database. I create a MySQL table with the fields “pointID”, “X”, “Y”, “Xgoogle”, “Ygoogle”, “DateStreetview” en “DateCheck”. The field “PointID” is the primary key and got an index.

I convert the table of the layer with points to check to a CSV table, because that’s easy to import into MySQL with phpMyAdmin. The coordinates from the database come in the fields “X” and “Y” and the unique number from the database comes in the field “pointID”.

Now I have MySQL table with loads of records for which only the fields “pointID”, “X” and “Y” have a value. All other fields are empty.

Perform checks

On December 29, 2020 I noticed that the old unofficial Google API didn’t work any more. In the mean time Google created an official API that can be used to get the same information. The downside is that you have to get an API key for the “Street View Static API” in the Google Cloud Console and you have to give Google your credit card details in the Billing module of Google Cloud Console. Fortunately it’s free to request metadata of the Streetview images. Requesting the Streetview images using the API does cost money. Because an API key is personal and linked to a credit card, I replaced my API key by MyApiKey, but if you want to use this method, you’ll have to replace that with your own API key.

In the old API an XML file was requested, but in the new API only an JSON file can be requested, so I had to change my script a little, to make it work again.

What I do is, I perform a query to the API for every point in the database. To do this, I made a PHP script that checks a point and stores the retrieved information in the MySQL database. I store the lat and lng data from the JSON file in the fields “Xgoogle” and “Ygoogle”. The information from date from the JSON file I store in the field “DateStreetview” and the current date is stored in the field “DateCheck”. By sorting on the field “DateCheck”, I make sure the point that hasn’t been checked the longest, gets checked first. In this way the data is kept up-to-date.

This API kan be queried through the follwowing url: https://maps.googleapis.com/maps/api/streetview/metadata?location=$y,$x&key=MyApiKey where I replace $x and $y with the X and Y coordinates in WGS84 (the order of X and Y in the url is important!). By going to this url, I get an JSON result with all necessary information.

This gives a JSON result like in the following example:

{
   "copyright" : "© Google",
   "date" : "2020-08",
   "location" : {
      "lat" : 51.22039286329926,
      "lng" : 4.359739876319615
   },
   "pano_id" : "KGJzmAhMdJs290cBSG7A-g",
   "status" : "OK"
}

You can do 30.000 meta data requests per minute to this API, before you would be blocked from using it. I use the script once every two minutes for 24 hours a day using a CRON job, because I don’t want to overload my webserver. This means it will take some time to check all points.

For the checks I use an non-official Google API. The advantage of this API is it gives me detailed information about the Streetview images. The big disadvantage is that you never know how long it ‘s functional, because Google can shut it down or block access without a warning and without giving a reason. It’s an API from Google itself, but it’s not publicly documented.

This API kan be queried through the follwowing url: https://maps.google.com/cbk?output=xml&ll=$y,$x where I replace $x and $y with the X and Y coordinates in WGS84 (the order of X and Y in the url is important!). By going to this url, I get an XML result with all necessary information. An example can be seen here: https://maps.google.com/cbk?output=xml&ll=51.154733,4.445684.

What I do is, I perform a query to the API for every point in the database. To do this, I made a PHP script that checks a point and stores the retrieved information in the MySQL database. I store the lat and lng data from the XML file in the fields “Xgoogle” and “Ygoogle”. The information from image_date from the XML file I store in the field “DateStreetview” and the current date is stored in the field “DateCheck”. By sorting on the field “DateCheck”, I make sure the point that hasn’t been checked the longest, gets checked first. In this way the data is kept up-to-date.

I limit the number of points per query to prevent Google blocking access to the API for me. Because it’s a non-official API, the usage limits are unknown. So you never know when your access to the API will be blocked. I use the script once a minute for 24 hours a day. This way I can check 1440 points per day. For Edegem for example, that has 17074 points to check, it takes almost 12 days to check all points.

Use the data in GIS

To use the data from the MySQL database in QGIS I convert it to geojson with usage of GeoPHP. In the database the Streetview Capture Date is stored per month, like 08-2015, but on my map I want to give it a color per year. To do this the fastest, the capture year is added to the geojson as a separate field. QGIS has the advantage it can read geojson files directly from the internet. Therefore QGIS wil always have the live data from the MySQL database. The database is kept up-to-date by continue to check even after all points have been checked. To use a geojson url in QGIS click on “Add Vector Layer”. In the menu that opens select “Protocol”, as type select “GeoJSON” and in the box “URI” fill in the geojson url.

In QGIS I symbolize the points with simple dots that I give a color per year, where the reds are the oldest and greens are the youngest capture dates. In the legend settings I use “Symbol levels” to make sure the most recent years are always shown on top. After that it’s only a matter of exporting the map with legend and titles. The result can be seen on https://stuyts.xyz/english/most-recent-google-streetview-data-edegem-may-1-2016.

EDIT:

A couple of years later I wrote a follow-up of this blog post that contains the PHP and MySQL scripts I use: https://stuyts.xyz/2020/02/27/a-map-with-the-most-recent-streetview-capture-dates-per-street-the-php-and-sql-scripts-i-use/