Triggering Azure Functions with Java when a storage change occurs

Recently my azure-javadocs project has been running into trouble. This project is a simple one that uses Travis CI to clone a number of GitHub projects, generate an aggregate JavaDoc from all of them, and then uses the static hosting feature of GitHub to host that output. The problem is that for every build, all of the files in the static branch on GitHub are overwritten, and over time this has become exceedingly large. Travis CI was configured to run daily, and in the last six months or so of running this service, the .git directory in a fresh clone has grown to 4.18GB! This is clearly not going to end well 🙂

So, I set about rearchitecting this project, and now I’m using the free Visual Studio Team Services (VSTS) to do the bulk of the work, as well as Azure Functions to do a little bit of work at the end. Essentially, VSTS now acts as my git repository and build system. For every commit into the git repo, a build is triggered automatically (I commit infrequently – just whenever I want to enable / disable a project from the generated Javadoc – so VSTS is also set to run the build daily). The build steps are specified within the VSTS user interface, and run through the standard steps: clone all the external repos, run Maven tasks to install and generate javadoc. The build concludes by creating a zip file containing the javadoc output, and storing it in an ‘incoming’ container within my Azure storage account.

Now that I don’t want to host the JavaDoc within GitHub (because I don’t want to end up with a massive repo again), I make use of a new feature of Azure Storage, which is the built-in support for hosting static sites directly from a storage container (technically the feature is still in-preview, but it works well for me). All I need to do is place my static website (i.e. the JavaDoc output) into a special $web container that is created for me by Azure Storage when I enable the static sites support.

The cool part comes next: I’ve written an Azure Function (in Java, of course) that is set up to trigger on any files being added to the incoming container. When a file appears, Azure makes sure to call my function, where I can then do whatever I want. In my case, I use it as a chance to firstly empty the entire $web container, and then to unzip the new zip file into the $web container, before deleting the zip file. Once this is all done, the docs are available from java.ms/api (which, coincidentally, is another Azure Function written in Java). At present the API docs available here are just a subset of all available docs, but I plan to add the missing ones soon (pending some pull requests into various projects).

The code I wrote is below. The most notable aspect is the use of @BlobTrigger  to tell the function that I want to trigger on file changes in the ‘incoming’ container. The code is not overly complex, and mostly just deals with deleting all files and then unzipping all files, but I hope it is helpful for someone who is wanting to work out how to use Azure Functions to act as the glue between different services.

In the fullness of time it is unlikely I will need to even have this Azure Function code at all – VSTS will almost certainly have first class support for static site hosting baked into it any day now, but because static site hosting is currently a preview feature, I took it as a good opportunity to learn how to trigger an Azure Function on something different than an HTTP Trigger, and I’m publishing it here in case others run into any issues trying to achieve this for themselves.

Building a serverless url shortener with Azure Functions and Java, part two

This is the second post in a series of blog posts on building a serverless url shortener in Java and Azure Functions. Here’s a list of all the posts I (may) publish eventually (links will be added as the posts are published):

In this post, I wanted to cover the additional code required to use Azure Storage Queues as part of a function app, to reduce the amount of time that your users must wait before a function returns by offloading work that can be done asynchronously into a queue (which is triggered by a separate Azure Function).

My use case is simple: I’d like to do analysis on the use of my URL shortener – which links are popular, when are they popular, who refers them, etc. This analysis involves looking at various request headers, storing data into data storage, and maybe doing some number crunching. Additionally, I wouldn’t mind being pinged on a Slack channel whenever my URL shortener is used, just for fun… The thing is – all these tasks take time, and while I’m happy to pay Azure to do them, I don’t want them done while my users are waiting to be redirected to their desired URL.

Adding Elements to the Queue

This is a perfect use case for the queues support baked into Azure Functions. What we can do is use a queue in the redirect function that we discussed in the last post, and add into the queue the relevant data (as a string). This enables the redirect function to focus on its core task and responding to the user as soon as possible. I can then define another Azure Functions function that, instead of being triggered by an HTTP request, is triggered by a queue having an item added to it. First of all, lets update the redirect function code from the last blog post to have the queue provided as a function argument, so that we can write to it:

Note that new line in the method arguments starting with @QueueOutput. This is the queue that we will write to later in the function. To actually have this queue setup, you should refer to the Azure Functions queue documentation. The name attribute is an internal name, whereas queueName is the name the queue has been given in Azure. The connection value is a connection string – in my case I am using the standard Azure Storage that is provisioned as part of my Azure Functions deployment (in the same way I use the standard Azure Storage that is part of the Azure Functions deployment for table storage also, for every short code mapping).

Further down the redirect function code I extract out a few useful headers, then I simply concatenate all the values I want into a single string (with the pipe character as a separator), and then I call queue.setValue(...) to add this element to the queue. As far as this function is concerned, this task is now offloaded into the queue, and is no longer its concern.

Queue Processing

Now we move over to the other side of the queue, which I call the processing function. I won’t include the actual data analytics discussion here (because I haven’t done much with it, and so I will cover it at a later date when I have more to say), but what I will cover is getting the trigger, and sending notification out to Slack.

In this code you can see that instead of @QueueOutput we now have a @QueueTrigger annotation. This tells Azure that we are expecting this method to be triggered whenever the specified queue has elements added to it, with the new value that was added to the queue being set as the value of the request string argument. When this queue value is received, you can see we have some analytics code (to be covered in a future post) that takes the pipe-delimited string from earlier and turns it back into separate values, and then logs it and sends a notification to Slack.

That is remarkably simple! We’ve offloaded time-intensive work to a separate function, unblocking our redirect function and enabling it to be more responsive to visitors. At the same time, the complexity of the code is kept to a minimum.

This approach to integrating queues into Azure Functions is really useful, but you can also use queues outside of functions as well. There is a good tutorial on how to use the queues API in Java.

Sending to Slack

I like Slack a lot, and I use it for both communications and notifications. I like that it is really easy to setup and integrate with external systems. Because Slack supports incoming webhooks, I simply added a dependency in my Azure Functions to Feign, and wrote a few lines of code to allow for me to send messages to my account. Here’s the Slack interface I wrote:

I then wrote a SlackUtil class to make consuming this interface even simpler:

To use this code, you can refer back to the processing function earlier: it’s a single line call to send a message directly to the #general channel in my Slack account. The only external element in the webhook URL, which I added as an application setting in Azure, so that it isn’t part of the source code that I ship into GitHub (so people can’t spam me) 🙂 To learn more about how to bring in application settings (both when developing Azure Functions locally and also when deployed to the web), I’ve posted a separate article explaining best practices.

Summary

As with my first post on this topic – serverless programming makes server-side development really easy, even for people who are not skilled server-side developers! If you are a Java developer, you should definitely take a look at Azure Functions today – get started with the free tier and go from there! As I noted in my last post – the cost of operating these services is extremely minimal (cents per month).

Environment variables in Azure Functions with Java

It is often desirable to extract out secret information from source code for security reasons. This allows code to be published to source code repos without accidentally providing credentials to other developers. This can be achieved simply by using environment variables, both when running Azure Functions locally, and when deploying your functions to Azure.

To easily set environment variables when running Azure Functions locally, you may choose to add these variables to the local.settings.json file. If one is not present in the root directory of your function project, feel free to create one. Here is what the file should look like:

Each key / value mapping in the values map will be made available at runtime as an environment variable, accessible by calling System.getenv("<keyname>"), for example, System.getenv("AzureWebJobsStorage"). Adding additional key / value pairs is accepted and recommended practice.

Note: If this approach is taken, be sure to consider whether adding the local.settings.json file to your repository ignore file, so that it is not committed.

With your code now depending on these environment variables, you can log in to the Azure Portal to set the same key / value pairs in your function app settings, so that your code functions equivalently when testing locally and when deployed to Azure. Here’s a screenshot for reference:

Serverless programming makes development of web services and triggers really easy, even for people who are not skilled server-side developers! If you are a Java developer, you should definitely take a look at Azure Functions today – get started with the free tier and go from there! As I noted in my first post – the cost of operating these services is extremely minimal (cents per month).

Creating custom routes in Azure Functions

I’ve been working on my URL shortener project recently, but I took a few days away from it to start writing some JavaDoc for the Java APIs for Azure Functions. In doing so I learned about a cool piece of API that might not be readily apparent (although I hope it is now that I’ve written documentation!), and so in this blog post I wanted to quickly introduce the route field on the @HttpTrigger annotation.

By default, when you create a function for an HTTP trigger the function is addressable with a route of the form http://<yourapp>.azurewebsites.net/api/<functionname>. This is fine in many cases, but in some cases you want your endpoint to be parameterised. For example, we’ve all seen URLs such as reddit.com/r/java, where the java can be replaced by any value (I hope I’m not shattering illusions for anyone by informing you that these all aren’t separate HTML pages sitting on a server 🙂 ). Doing this with Azure Functions is trivial, with help from the route property. For example, we could specify a route of products/{category:alpha}/{id:int}, and this would mean that the function is now addressable at http://<yourapp>.azurewebsites.net/api/products/electronics/357.

If the only benefit was that the path was parameterised, we wouldn’t bother using this API, which is why the other half of the feature is the ability to use the @BindingName annotation to bring in these arguments into the function. For example, here is a full method signature with route and @BindingName in use (apologies for the odd line wrapping required):

As can be seen here, we are receiving the category and id values from the URL endpoint and they are being provided to us as arguments into the function itself, where they are immediately usable. The route string can be quite complex, as there are a number of configuration options available. Microsoft has published useful documentation on routing (just ignore the C# noise 🙂 ).

This has just been a quick blog post on the routing support in Azure Functions, because I thought it was neat. I will keep posting this short snippets as I discover API that delights me. As always though: playing with Azure Functions is really fun – I recommend all Java developers take it for a spin to see what they can achieve when they don’t need to worry about all the underlying infrastructure. Even better, you can get started with the free tier where you have 1,000,000 free function calls a month, and go from there! If you want more inspiration, check out my series on URL shorteners, built using Java and Azure Functions.

Building a serverless url shortener with Azure Functions and Java, part one

I’m a Java engineer who doesn’t really know the intricacies of cloud development as well as my ‘Cloud Developer Advocate‘ job title suggests that I should. That is why I’m such a fan of Azure Functions – it makes the concept of serverless programming a possibility for Java developers. Serverless is one of those odd marketing terms, but what it essentially boils down to, as my colleague Jeremy Likness likes to say, is that it is ‘less server’ – which sounds perfect for me! 🙂

To better understand Azure Functions, I set out to build a URL shortener app, so that I could take a long URL like http://docs.microsoft.com/en-us/java/azure/ and replace it with a short url like http://java.ms/docs. I’m not the first Microsoft Cloud Developer Advocate to do this, as it seems like one of those projects people like to do when they first encounter serverless programming 🙂 You can read Jeremy’s blog about how he built a URL shortener in C# (he’s put a lot more hours into features than me, so there is a lot of good ideas there to borrow 🙂 ).

This is a first post in a series. I’ll add links here when I publish new articles, but you can always follow me on Twitter to keep updated. The total topic list for this series includes:

Serverless Value Proposition

The main attraction to building a link shortener using serverless programming is that you only pay for the time your function is actually operating – you aren’t paying for 24/7/365 server uptime, which is pretty handy for a link shortener that only operates occasionally. You also don’t need to worry at all about concerns such as scaling your service – the cloud provider takes care of that for you. I wanted to build this as cheaply as possible (i.e. consume as few resources, and use the cheapest options) in Microsoft Azure, and I didn’t want to have to write code in any language other than Java, and I really didn’t want to have to worry about all that other cloud nonsense 🙂

In terms of actual costs, Jeremy stated in his blog post the following:

Although actual results will differ for everyone, in my experience, running the site for a week while generating around 1,000 requests per day resulted in a massive seven cent U.S.D. charge to my bill. I don’t think I’ll have any problem affording this!

He also updated his calculations for cost in a follow-up tweet:

I tend to agree – I think I can afford this serverless lifestyle! 🙂 So, off I set on my adventure… I started by buying two domain names – jogil.es and java.ms – which both seemed relevant to my interests. After that, I started writing code (and note: this project is all open source on GitHub)! 🙂 Let’s get into it…

Creating a Java Function App

Setting up a new Azure Function App is simple, especially with the Java / Maven tooling that is available. Firstly though, if you don’t have an Azure account, you can create one for free, and it comes with 1,000,000 Azure Function calls free per month forever (which is well in excess of what I need, so I don’t think I’ll even be spending 7 cents)! Once you have an account, you can follow the Azure Functions on Java tutorial to step through the software setup and creating your own function app. The approach works really smoothly – it’s all based around a few Maven commands that will auto-generate your first function, and you even use this to deploy to Azure! Because of this, I’m not going to dive any more deeply into getting started with Azure Functions with Java, and I’m just going to dive into some of the configuration details, and then into the code for building a link shortener.

Once you’ve done your first mvn clean package azure-functions:deploy, you can log in to the Azure Portal to see your function app, which will look something like this:

In here you’ll see a list of your functions in the left column (my app has four functions: frontend, keepAlive, redirect, and shortcode, as well as details on the URL, subscription, etc. Today we will just discuss the redirect and shortcode functions, and address the other two functions in another blog post.

By default, to access any of your functions, you simply take the base URL (in the image above, it is https://jogiles-short-url.azurewebsites.net), add /api/, and then the function name. For example, the redirect function is at http://jogiles-short-url.azurewebsites.net/api/redirect (and, low-and-behold, if you go there with a given shortcode, it should work – try http://jogiles-short-url.azurewebsites.net/api/redirect?shortcode=docs). As noted at the beginning of this post, the final URL is http://java.ms/docs, but that is achieved using an Azure Functions Proxy, which simply redirects to the full URL shown here. We will return to proxy configuration in a later blog post, to achieve this nicer URL effect.

URL Shortening Function

The key requirement of a URL shortener is to take a URL and return a shorter URL. In terms of implementation, it’s quite simple really – take a url query parameter that we want to convert, use some algorithm to create a unique short code, and store it in some persistent storage. That is basically what you see in the code below, with one additional feature: the shortcode function also supports an optional shortcode parameter. If the user specifies a preferred shortcode (like ‘docs’ above), we simply store that mapping in the persistent storage without running the shortcode algorithm. Here’s essentially the full code listing (remember all the code is on GitHub):

As can be seen, at present there is no authentication, so anyone can create shortlinks (I might rectify this some day soon) 🙂 We simply check the url is valid, and if the user wants an auto-generated shortcode or if they want to provide their own. Based on this, we go in to one of two functions. In the auto-generated case, we iterate until we find an acceptable shortcode, and then we persist that into the data store.

In terms of data storage, I’ve written a small wrapper API around the Azure Storage APIs for Java, as I originally intended to use MySQL or SQL Server (with or without JPA), but then I realised that because Azure Functions are built on top of Azure App Service, which has built-in storage available to it. Because of this, I simply piggy-back on the table store that is already available, using the Java APIs explained in the Azure Table Storage guide. I’ve written a simple AzureTableStore class to handle the read / write access to this table:

The end result of all this code is that I have a table that shows all mappings. Here is a screenshot of the excellent (and free) Microsoft Azure Storage Explorer app, looking at my table of short codes (click the image for a larger version):

You can see I simply use the first letter of the RowKey as the PartitionKey, to have an even distribution in all partition buckets. You can also see that of the four short links I have created, two are ‘custom’ shortlinks (for docs and linkedin), and two are auto-generated (00 and zC). To generate the short links, I simply have the following code I use:

Again, it’s not the prettiest or best approach, but it’ll do for now 🙂

The end result of all this code is that a short code is generated, and a short url is returned, including the URL (that is, java.ms or jogil.es, depending on which host was called to shrink the URL in the first place). That’s great, but it is only half the story – now we need to support the user actually going to that URL and it being converted to the long URL again!

Shortcode Redirection Function

The code to convert a shortcode into a full URL is even simpler, it simply gets the shortcode query parameter, looks up the long url in the data store, and does a HTTP 302 redirect to that URL (302 is the status code for a permanent redirect, as opposed to 301 which is temporary, and will therefore put more strain on the function for people who repeatedly visit the same URL). Here’s the redirect function class in full:

The only odd code is the line where I retrieve the url by calling Util.trackDependency(..). This code is simply convenience code that allows for me to more easily track application performance using Azure Application Insights, which I will cover in more detail in another blog. However, here’s the complete function listing, to make it clear that all that really is happening is we’re calling the data store to get the long url, and we’re recording how long it takes into Application Insights:

Summary

This is the core of the URL shortener, but there is a lot more still to cover, which I will cover in follow-up posts in the coming weeks. The total topic list for this series includes:

The main point for this post is that even though I’m not super-skilled in cloud development I was able to implement a simple application that provides a useful service to me, and at an extremely low cost. As I noted at the beginning, the cost to operate this service is mere cents per month! So, if you are a Java developer looking to build web services and don’t want to get bogged down in dealing with server details, you should definitely take a look at Azure Functions today – get started with the free tier and go from there!