Mark Richards

System Outline

This is another extension of the “AWS Area Eligibility API“ project in that the data used for my analysis gets published annually to an ArcGIS data repository. This last year FY24 I was late in updating the analysis because I was not checking for new data releases, and I had almost forgotten about it. On the repository page there is a link to an RSS feed but that appears to be for a generic feed that was never set up. So, the question became how I can monitor the repo for new data without having a built in feed or relying on myself to remember to check periodically.

This led me to writing a lambda function that pulls the list of available data layers and stores it as a list inside of the NoSQL database that hold the analysis results. Each time it runs it then pulls the list from the ESRI ArcGIS Directory Service and the database and compares the two. If there are new items in the ESRI directory it will update the databse entry then use the Simple Email Service (SES) to send a notification to my Gmail account with the name of the new layer.

This is all well and good but in order to prevent me from having to remember to run this code I set up a scheduled event in the AWS EventHub. This scheduled event triggers the lambda function to run, perform its check and notification if needed, then exit until the following day when it triggers again.

Implementation Lessons

The Simple Email Service is only simple in theory. This tool is the main configuration headache in this setup but it still winds up being one of the best ways to notify me of changes. So the pain of setup may be worth it in the long run.

Event Bridge Scheduler

The EventHub is a collection of event driven tools including busses, pipes, and schedules. There are a lot of very versatile tools here but the scheduler offers options like one time scheduling, CRON scheduling, or more human readable 'rate-based' or interval scheduling. In this instance its rate based with a 24 hour interval.

Lambda IAM

There a small amount of trial and error that went into granting the lambda function role the needed access to not only the DynamoDB table for read and write, but also to SES to send an email using the correct configuration set and verified identity.

ESRI Custom Error Messages

To my surprise when you make a call to the ESRI directory service, if there is an error like an incorrect link the endpoint doesnt return a standard HTTP status. It returns 200 along with a JSON object that contains the error code and reason. This isnt a big problem but does require a little bit of a rewrite for how you handle errors vs other endpoints.

SES Sandbox

SES is clearly built with a business mass mailer in mind. It has strict rules about email health and safety and limits you to a sandbox with a maximum of 200 emails a day to verified addresses. In order to access the real production level services, you need to submit a request with your detailed plan for managing your email lists and addressing reports of spam or undelivered mail. Etc. fortunately for this system the sandbox is more than enough. This will generate at most 1 email a day and send it to my own address which I was able to verify.

Storing liste items in DynamoDB

Dynamo has a very particular way it wants you to read and write data. When writing it wants to know the data format for every piece of information it stores. In this case I wanted to store a simple list, but I also did not want to have to iterate over the list and build out data types for each item. So, a trick you can use is to convert your list into a string and store it as a string. The only trick them becomes when you read it back you need to parse the text into a list again.

Python Abstract Syntax Trees

In order to read the text from the database back into a list we can use the AST package or Abstract Syntax Trees. It uses the structure of the text to infer the datatype. This lets python know that the string really should be parsed as a list. The library has a function for ast.literal_eval() that can take the text and turn it back into a list. Which then allows for easy comparison between the stored database information and the latest ESRI directory service layer list.

Retrospective

Setting up the Simple Email Service is kind of a pain. It is something that I’m not entirely sure I would recommend without good reason. There is just a lot of information to cover as you start to learn about the service and a lot of restrictions. Most of the rules makes sense in that they seem to be in place to prevent abuse and spamming coming from the amazon servers. But it also means there are barriers to entry for the casual user.

Had I not already had some addresses and domains verified from when I had attempted to learn the service earlier I am not sure I would have spent the time to do so.

That said I think this combination of a scheduler, a lambda function, and a notification service makes for a powerful mix. This example is fairly basic you could adapt this to any number of monitoring or alerting tasks. An application I may explore next is using this architecture but connecting event bridge to a CloudWatch alarm that monitors sage maker model trainings.

Return To AWS Projects