Author Topic: Reverse Geocoding outage  (Read 2497 times)

Offline ahoward

  • Camera Bits Staff
  • Hero Member
  • *****
  • Posts: 935
    • View Profile
Reverse Geocoding outage
« on: July 25, 2024, 01:09:25 PM »
We are currently working through an issue with our reverse geocoding service where incorrect data is being returned for various locations across the globe. If you don't care too much about the details, the most important information that we'd like to convey is that we will have to take the service down while we work to correct the problem. During this time, if you attempt to use the service, you will get an error message. While we hope to get the service up and running again quickly, the process to fix it could take up to two weeks.

Reverse geocoding is a service that was introduced in Photo Mechanic 6. There are many services that provide reverse geocoding, but most either come with a cost per request (Google Maps), or are free with a rate limit that would make it useless to Photo Mechanic users. In order to be able to provide this service at no additional cost to our users, the approach we took with this service was to pull the data from OpenStreetMap, and then mirror it on our servers where we could provide the response to any reverse geocode lookups.

OpenStreetMap follows a crowdsourced model for keeping their maps up to date: anyone in the world can update the OpenStreetMap data. This means that anyone can add correct information, but anyone can also add whatever information they want; the goal here is, of course, that with enough people actively updating the map, that if bad information is added, it is quickly corrected by someone who wants to contribute to the data being good. Our server doesn't get every single change made to the map immediately, it updates periodically, so generally, even if someone contributes bad information to the map, it won't be pulled to our server unless it has been there for a significant amount of time and hasn't been caught by folks who spends their time contributing to OpenStreetMap accurately representing our world.

Unfortunately, the data center that houses our server suffered a power outage a few weeks ago, and of course our server lost power along with it. During this time, the service was of course unavailable, but we believe that outage caused some corruption of our database, and that corruption is preventing our periodic updates of the database from completing. To make matter worse, the data that is currently sitting in the database seems to have gotten a significant amount of bad information, and for many locations it is returning completely inaccurate locations for some or all of the location fields.

Due to this, we may need to do a full rebuild of our database, which will mean taking the service offline while the server does that work. This database is huge, and building it takes some time. If we can update the database in stages, it will limit the down time, but if we have to reimport the entire database, it could take up to two weeks. We will make another post here once the work is complete and it is back up and running.

Offline PabloGG

  • Camera Bits Staff
  • Newcomer
  • *****
  • Posts: 16
    • View Profile
Re: Reverse Geocoding outage
« Reply #1 on: August 10, 2024, 11:44:57 AM »
Status update on the Reverse Geocoding database rebuild:
The rebuild completed successfully yesterday morning. We are now running what they call "catchup" task whereby it grabs the latest updates since the image that was used to do the rebuild.
My guess is that we should be able to get the service back up and running in the next few days (or sooner). We are monitoring the rebuild progress round the clock in order to get the service up and running again as soon as possible.
Your patience is much appreciated.
« Last Edit: August 11, 2024, 05:59:55 AM by Kirk Baker »

Offline PabloGG

  • Camera Bits Staff
  • Newcomer
  • *****
  • Posts: 16
    • View Profile
Re: Reverse Geocoding outage
« Reply #2 on: August 13, 2024, 09:08:29 AM »
The "catchup" task to get the latest updates took a bit longer than expected, but finally completed early this morning.
Our goal is to get the Reverse Geocoding service available to the customers by end of day today (PST time) after we complete internal verification that services are running properly.

Offline ahoward

  • Camera Bits Staff
  • Hero Member
  • *****
  • Posts: 935
    • View Profile
Re: Reverse Geocoding outage
« Reply #3 on: August 13, 2024, 03:32:14 PM »
The server is back up and running. If you continue to experience errors with Reverse Geocoding, either with connection to the server, or with the data returned from a Reverse Geocoding operation, please get in touch, either through the forums or by emailing support@camerabits.com