Improving Log Aggregation & Search for Application Services

Leave a reply

When my team and I first took on responsibility for modernising our business’ backend application services, we found it was hard to obtain a full picture of the events that were being logged by services, in a timely manner. Logging of events are an essential source of data for monitoring how a service is operating, and also to support troubleshooting of problems e.g. the no. and rate of failed API requests, and supporting failure diagnostics. Therefore, one of the first tasks we prioritised as a (devops) team was to improve this situation by provisioning a modern log aggregation and search solution. This post outlines the problems we addressed, the architectural pattern we used as the basis of the solution, the tools we used to implement the pattern, and the resulting benefits.

Continue reading →

Designing message consumer error handler for Amazon SQS

Leave a reply

amazon-sqs-logo

When building a message consumer you need to handle the various errors that will inevitably occur during message processing. If you’re using Amazon SQS as your message broker it provides some built-in support for error handling that you can utilise. But, if you want to handle all types of errors efficiently you should also add your own custom error handler. This post explains how you can do both these things, including how to classify message processing errors.

Continue reading →

JSON Schema Part 2 – Automating JSON validation tests

Leave a reply

In my first post on the topic of JSON Schema I outlined the limitations of specifying data-formats by example, described how schema languages help address this, and provided an introduction to the not so well known schema language for JSON – JSON Schema.

json-schema.org

In that first article I stated my intention to document non-trivial JSON data-format I specified in the future using JSON Schema in addition to using specification-by-example, whether it be for an API resource or a message payload. When you adhere to this you quickly realise that while producing a JSON Schema is valuable, to increase the value you’ll additionally want to extend the tests for your JSON producing components to automate the following:

  • Test the schema remains valid – i.e. it’s well-formed and conforms to the JSON Schema spec.- in order to avoid publishing a broken or invalid schema.
  • Test your app produces JSON that conforms to the published schema. This goes beyond just asserting that the component produces an expected lump of JSON, to additionally check that the generated JSON conforms to any validation constraints in the schema.

This follow-up, second post on JSON Schema, provides example code for implementing the above such tests using Java, JUnit and an open-source library for JSON Schema Validation.

Continue reading →

Measuring API Throughput using JMeter

Leave a reply

As an online business grows in popularity it can highlight scalability problems in your architecture. Certain hardware and/or software components can struggle to keep up with increased demand and it may not be easy or cost-effective to address this by extending them in their present form, especially when the demand is ‘spiky’ in nature. My team was recently faced with such an issue. We decided to engineer our way out of the problem by designing a more scalable replacement solution. Key technical requirements for the new solution included the throughput and latency of a public web API. In this post I share details of how you can measure these performance characteristics of an API using Apache JMeter, and in the process I identify a very useful supporting JMeter plugin.

jmeter-throughput-per-sec-graph

JMeter – Throughput per second graph

Continue reading →

Amazon API Gateway – What’s it all about?

Leave a reply

amazon-api-gateway-logo

My team and I design and build a lot of REST APIs during the course of our work, to expose functionality and data to client systems, and to integrate our back-end services with one another. I was therefore interested in learning more about Amazon’s recently launched API Gateway service. This is a short post summarising the role and major features of Amazon API Gateway based on my own preliminary research and trialling. I also briefly cover what the service is not designed to support, and consider possible future use-cases.

Continue reading →

Selecting a modern stack for building & deploying microservices on the JVM

Leave a reply

The success of an online service with millions of users worldwide, and an interactive audience of several thousand, depends on a well architected and engineered software platform. Our engineering team has recently updated the software stack we use to build and deploy back-end (JVM-based) services for such an online service. In this post I share the choices we made, as well as some of the background (motivation behind the project, benefits we were looking to gain). This might help if you’re undertaking a similar strategic review and update of the software stack and deployment approach for your back-end services in the near future.

Continue reading →

Designing and building an open source REST API client

I’ve recently designed and built an open source API client for the REST API of a major enterprise video and webcast service, for use by its customers. It was a rewarding exercise that has helped me gain some valuable additional practical experience of emerging technology and tools for building and testing apps. This post outlines the value of the API client, its design, the technologies and tools I used, and how I rate them. I’ve open sourced the code to allow customers, and others, to view and reuse what I’ve built.

Continue reading →

Java 8 – Examples of the major, new language features

Leave a reply

Over the course of the last month or so, I’ve invested some time familiarising myself with the major new features in Java 8, learning how to apply them, and understanding the benefits which can be gained from using them. As part of this exercise I’ve produced some code examples (see below) for the following features:

  • Lambda expressions
  • Streams (aka bulk or aggregate data operations)
  • Default methods
  • Date/time API
  • Optional type

Continue reading →