I'm a Java engineer who doesn't really know the intricacies of cloud development as well as my 'Cloud Developer Advocate' job title suggests that I should. That is why I'm such a fan of Azure Functions - it makes the concept of serverless programming a possibility for Java developers. Serverless is one of those odd marketing terms, but what it essentially boils down to, as my colleague Jeremy Likness likes to say, is that it is 'less server' - which sounds perfect for me! :-)
To better understand Azure Functions, I set out to build a URL shortener app, so that I could take a long URL like http://docs.microsoft.com/en-us/java/azure/ and replace it with a short url like http://java.ms/docs. I'm not the first Microsoft Cloud Developer Advocate to do this, as it seems like one of those projects people like to do when they first encounter serverless programming :-) You can read Jeremy's blog about how he built a URL shortener in C# (he's put a lot more hours into features than me, so there is a lot of good ideas there to borrow :-) ).
This is a first post in a series. I'll add links here when I publish new articles, but you can always follow me on Twitter to keep updated. The total topic list for this series includes:
- Part one: Introduction to project and development of shortcode and redirect functions
- Aside One: Creating custom routes in Azure Functions
- Part two: Using queues and integrating realtime Slack notifications of events
- Aside Two: Environment variables in Azure Functions with Java
- Part three: Integrating Application Insights to learn more about performance bottlenecks, etc
- Part four: Configuring proxies and custom domains
- Part five: Building a static HTML user interface to easily create short links, and hosting it as an Azure Function
- Part six: Focusing on performance - lets pay as little as possible by making our code as performant as possible
Serverless Value Proposition
The main attraction to building a link shortener using serverless programming is that you only pay for the time your function is actually operating - you aren't paying for 24/7/365 server uptime, which is pretty handy for a link shortener that only operates occasionally. You also don't need to worry at all about concerns such as scaling your service - the cloud provider takes care of that for you. I wanted to build this as cheaply as possible (i.e. consume as few resources, and use the cheapest options) in Microsoft Azure, and I didn't want to have to write code in any language other than Java, and I really didn't want to have to worry about all that other cloud nonsense :-)
In terms of actual costs, Jeremy stated in his blog post the following:
Although actual results will differ for everyone, in my experience, running the site for a week while generating around 1,000 requests per day resulted in a massive seven cent U.S.D. charge to my bill. I don’t think I’ll have any problem affording this!
He also updated his calculations for cost in a follow-up tweet:
https://twitter.com/jeremylikness/status/957976243634307073
I tend to agree - I think I can afford this serverless lifestyle! :-) So, off I set on my adventure... I started by buying two domain names - jogil.es and java.ms - which both seemed relevant to my interests. After that, I started writing code (and note: this project is all open source on GitHub)! :-) Let's get into it...
Creating a Java Function App
Setting up a new Azure Function App is simple, especially with the Java / Maven tooling that is available. Firstly though, if you don't have an Azure account, you can create one for free, and it comes with 1,000,000 Azure Function calls free per month forever (which is well in excess of what I need, so I don't think I'll even be spending 7 cents)! Once you have an account, you can follow the Azure Functions on Java tutorial to step through the software setup and creating your own function app. The approach works really smoothly - it's all based around a few Maven commands that will auto-generate your first function, and you even use this to deploy to Azure! Because of this, I'm not going to dive any more deeply into getting started with Azure Functions with Java, and I'm just going to dive into some of the configuration details, and then into the code for building a link shortener.
Once you've done your first mvn clean package azure-functions:deploy, you can log in to the Azure Portal to see your function app, which will look something like this:
In here you'll see a list of your functions in the left column (my app has four functions: frontend, keepAlive, redirect, and shortcode, as well as details on the URL, subscription, etc. Today we will just discuss the redirect and shortcode functions, and address the other two functions in another blog post.
By default, to access any of your functions, you simply take the base URL (in the image above, it is https://jogiles-short-url.azurewebsites.net), add /api/, and then the function name. For example, the redirect function is at http://jogiles-short-url.azurewebsites.net/api/redirect (and, low-and-behold, if you go there with a given shortcode, it should work - try http://jogiles-short-url.azurewebsites.net/api/redirect?shortcode=docs). As noted at the beginning of this post, the final URL is http://java.ms/docs, but that is achieved using an Azure Functions Proxy, which simply redirects to the full URL shown here. We will return to proxy configuration in a later blog post, to achieve this nicer URL effect.
URL Shortening Function
The key requirement of a URL shortener is to take a URL and return a shorter URL. In terms of implementation, it's quite simple really - take a url query parameter that we want to convert, use some algorithm to create a unique short code, and store it in some persistent storage. That is basically what you see in the code below, with one additional feature: the shortcode function also supports an optional shortcode parameter. If the user specifies a preferred shortcode (like 'docs' above), we simply store that mapping in the persistent storage without running the shortcode algorithm. Here's essentially the full code listing (remember all the code is on GitHub):
public class ShortcodeFunction {
private static final int MIN_KEY_LENGTH = 2;
@FunctionName("shortcode")
public HttpResponseMessage<String> shortcode(
@HttpTrigger(name = "req", methods = {"post"}, authLevel = AuthorizationLevel.ANONYMOUS) HttpRequestMessage<Optional<String>> request,
final ExecutionContext context) {
String url = request.getQueryParameters().getOrDefault("url", null);
if (url == null || url.isEmpty()) {
return request.createResponse(Util.HTTP_STATUS_BAD_REQUEST, "No url query parameter provided");
}
// if there is no tracking information, add the default tracking information here
url = Util.addMicrosoftTracking(url, request);
// we allow people to request their desired short code by setting a 'shortcode' query parameter
String desiredShortcode = request.getQueryParameters().getOrDefault("shortcode", null);
if (desiredShortcode == null) {
// the user just wants an auto-generated shortcode
return useGeneratedShortcode(request, url, context);
} else {
// lets try to use their shortcode. If it fails, we return a failure.
return useProvidedShortcode(request, url, desiredShortcode, context);
}
}
private HttpResponseMessage<String> useGeneratedShortcode(HttpRequestMessage<Optional<String>> request,
String url,
final ExecutionContext context) {
context.getLogger().info("Attempting to create shortcode with url " + url + " and autogenerated shortcode");
DataStore dataStore = DataStoreFactory.getInstance();
String shortCode = "";
int collisionCount = 0;
int keyLength = MIN_KEY_LENGTH;
while (shortCode == null || shortCode.isEmpty()) {
shortCode = dataStore.saveShortCode(url, Util.generateKey(keyLength), true);
// if we are here, the proposed shortcode failed, so we count that.
// If we get too many collisions, increase the key length by one and
// keep trying
collisionCount++;
if (collisionCount > 3) {
keyLength++;
collisionCount = 0;
}
}
context.getLogger().info("Created short code: " + shortCode);
return request.createResponse(Util.HTTP_STATUS_OK, createShortUrl(request, shortCode));
}
private HttpResponseMessage<String> useProvidedShortcode(HttpRequestMessage<Optional<String>> request,
String url,
String shortcode,
final ExecutionContext context) {
context.getLogger().info("Attempting to create shortcode with url " + url + " and shortcode " + shortcode);
DataStore dataStore = DataStoreFactory.getInstance();
// we set checkForDupes to be false, so that we allow this short code to be used, even if the long URL is
// recorded elsewhere
String result = dataStore.saveShortCode(url, shortcode, false);
if (result == null) {
return request.createResponse(Util.HTTP_STATUS_CONFLICT, "Requested shortcode already in use");
}
return request.createResponse(Util.HTTP_STATUS_OK, createShortUrl(request, result));
}
private String createShortUrl(HttpRequestMessage<Optional<String>> request, String shortCode) {
return "http://" + Util.getHost(request).getHost() + "/" + shortCode;
}
}As can be seen, at present there is no authentication, so anyone can create shortlinks (I might rectify this some day soon) :-) We simply check the url is valid, and if the user wants an auto-generated shortcode or if they want to provide their own. Based on this, we go in to one of two functions. In the auto-generated case, we iterate until we find an acceptable shortcode, and then we persist that into the data store.
In terms of data storage, I've written a small wrapper API around the Azure Storage APIs for Java, as I originally intended to use MySQL or SQL Server (with or without JPA), but then I realised that because Azure Functions are built on top of Azure App Service, which has built-in storage available to it. Because of this, I simply piggy-back on the table store that is already available, using the Java APIs explained in the Azure Table Storage guide. I've written a simple AzureTableStore class to handle the read / write access to this table:
public class AzureTableStore implements DataStore {
private static final String TABLE_NAME = "shortcodes";
private static final String storageConnectionString = System.getenv("AzureWebJobsStorage");
private CloudTable cloudTable;
@Override
public synchronized String getLongUrl(String shortCode) {
try {
TableOperation lookupLongUrl = TableOperation.retrieve(Util.getPartitionKey(shortCode), shortCode, ShortCodeRecord.class);
TableResult result = getTable().execute(lookupLongUrl);
if (result == null) return null;
ShortCodeRecord record = result.getResultAsType();
return record.getLongUrl();
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
@Override
public synchronized String getShortCode(String longUrl) {
TableQuery<ShortCodeRecord> query = TableQuery
.from(ShortCodeRecord.class)
.where("LongUrl eq '" + longUrl + "'");
try {
Iterable<ShortCodeRecord> results = getTable().execute(query);
for (ShortCodeRecord record : results) {
return record.getShortCode();
}
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
@Override
public synchronized boolean persistShortCode(String longUrl, String shortCode) {
try {
getTable().execute(TableOperation.insertOrReplace(new ShortCodeRecord(longUrl, shortCode)));
return true;
} catch (Exception e) {
e.printStackTrace();
return false;
}
}
private CloudTable getTable() throws Exception {
if (cloudTable == null) {
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
CloudTableClient tableClient = storageAccount.createCloudTableClient();
cloudTable = tableClient.getTableReference(TABLE_NAME);
cloudTable.createIfNotExists();
}
return cloudTable;
}
}The end result of all this code is that I have a table that shows all mappings. Here is a screenshot of the excellent (and free) Microsoft Azure Storage Explorer app, looking at my table of short codes (click the image for a larger version):
You can see I simply use the first letter of the RowKey as the PartitionKey, to have an even distribution in all partition buckets. You can also see that of the four short links I have created, two are 'custom' shortlinks (for docs and linkedin), and two are auto-generated (00 and zC). To generate the short links, I simply have the following code I use:
public static String generateKey(int length) {
StringBuilder key = new StringBuilder();
Random random = new Random();
for (int i = 0; i < length; i++) {
int type = random.nextInt(3);
switch (type) {
case 0: key.append((char)(random.nextInt(10) + 48)); break; // 0-9
case 1: key.append((char)(random.nextInt(26) + 65)); break; // A-Z
case 2: key.append((char)(random.nextInt(26) + 97)); break; // a-z
}
}
return key.toString();
}Again, it's not the prettiest or best approach, but it'll do for now :-)
The end result of all this code is that a short code is generated, and a short url is returned, including the URL (that is, java.ms or jogil.es, depending on which host was called to shrink the URL in the first place). That's great, but it is only half the story - now we need to support the user actually going to that URL and it being converted to the long URL again!
Shortcode Redirection Function
The code to convert a shortcode into a full URL is even simpler, it simply gets the shortcode query parameter, looks up the long url in the data store, and does a HTTP 302 redirect to that URL (302 is the status code for a permanent redirect, as opposed to 301 which is temporary, and will therefore put more strain on the function for people who repeatedly visit the same URL). Here's the redirect function class in full:
public class RedirectFunction {
@FunctionName("redirect")
public HttpResponseMessage<String> redirect(
@HttpTrigger(name = "req", methods = {"get"}, authLevel = AuthorizationLevel.ANONYMOUS) HttpRequestMessage<Optional<String>> request,
final ExecutionContext context) {
String shortCode = request.getQueryParameters().getOrDefault("shortcode", null);
String url;
if (shortCode == null || shortCode.isEmpty()) {
Util.TELEMETRY_CLIENT.trackEvent("No shortcode provided, returning default url instead");
url = Util.getHost(request).getDefaultURL();
} else {
String _shortCode = shortCode.toLowerCase();
if (_shortCode.equals(Util.ROBOTS_TXT)) {
context.getLogger().info("Request for robots.txt ignored");
return request.createResponse(Util.HTTP_STATUS_OK, Util.ROBOTS_RESPONSE);
}
url = Util.trackDependency(
"AzureTableStorage",
"Retrieve",
() -> DataStoreFactory.getInstance().getLongUrl(shortCode),
proposedUrl -> proposedUrl != null && !proposedUrl.isEmpty());
if (url == null) {
url = Util.getHost(request).getDefaultURL();
}
}
HttpResponseMessage response = request.createResponse(Util.HTTP_STATUS_REDIRECT, url);
response.addHeader("Location", url);
return response;
}
}The only odd code is the line where I retrieve the url by calling Util.trackDependency(..). This code is simply convenience code that allows for me to more easily track application performance using Azure Application Insights, which I will cover in more detail in another blog. However, here's the complete function listing, to make it clear that all that really is happening is we're calling the data store to get the long url, and we're recording how long it takes into Application Insights:
/**
* Convenience method to log telemetry data into application insights.
*/
public static <T> T trackDependency(String dependencyName, String commandName, Supplier<T> task, Function<T, Boolean> success) {
long start = System.currentTimeMillis();
T result = task.get();
long end = System.currentTimeMillis();
Util.TELEMETRY_CLIENT.trackDependency(dependencyName, commandName, new Duration(end - start), success.apply(result));
return result;
}Summary
This is the core of the URL shortener, but there is a lot more still to cover, which I will cover in follow-up posts in the coming weeks. The total topic list for this series includes:
- Part one: Introduction to project and development of shortcode and redirect functions
- Aside One: Creating custom routes in Azure Functions
- Part two: Using queues and integrating realtime Slack notifications of events
- Aside Two: Environment variables in Azure Functions with Java
- Part three: Integrating Application Insights to learn more about performance bottlenecks, etc
- Part four: Configuring proxies and custom domains
- Part five: Building a static HTML user interface to easily create short links, and hosting it as an Azure Function
- Part six: Focusing on performance - lets pay as little as possible by making our code as performant as possible
The main point for this post is that even though I'm not super-skilled in cloud development I was able to implement a simple application that provides a useful service to me, and at an extremely low cost. As I noted at the beginning, the cost to operate this service is mere cents per month! So, if you are a Java developer looking to build web services and don't want to get bogged down in dealing with server details, you should definitely take a look at Azure Functions today - get started with the free tier and go from there!
Thoughts on “Building a serverless url shortener with Azure Functions and Java, part one”