Building a REST API with Finch and Finagle
Table of Contents
A while ago I wrote about using Finagle for an API endpoint. I now have another need to write a simple API in Scala and this time I’m going to use Finch on top of Finagle.
Finch is a thin layer over Finagle that makes it easier to do things like routing and serialisation. Interestingly it also heavily promotes a functional programming style, which can help build very high perfoming APIs in fewer lines of code.
This is an introduction to Finch and Finagle. The full code can be found on Github.
Hello world
Let’s start with a simple Hello World. When we call /hello, it will return with { "message": "Hello, world!" }.
First, we’ll create a simple model to represent the returned message, which is just a case class.
package com.andrewjones.models
case class Message(message: String)
Then we’ll put our application logic in the services package. This simply returns a message populated with Hello, world!.
package com.andrewjones.services
import com.andrewjones.models.Message
import com.twitter.util.Future
class ExampleService {
val message = Message("Hello, world!")
def getMessage(): Future[Message] = {
Future.value(message)
}
}
One thing to note here is the use of a future. Futures are used heavily by Finch and Finagle, so it’s worth knowing a bit about what they are. See the Twitter Scala School for a good introduction.
Our main application class looks like this.
package com.andrewjones
import com.andrewjones.models.Message
import com.andrewjones.services.ExampleService
import com.twitter.app.Flag
import com.twitter.finagle.Http
import com.twitter.server.TwitterServer
import com.twitter.util.Await
import io.circe.generic.auto._
import io.finch.{Endpoint, _}
import io.finch.circe._
object ExampleApplication extends TwitterServer {
val port: Flag[Int] = flag("port", 8081, "TCP port for HTTP server")
val exampleService = new ExampleService
def hello: Endpoint[Message] = get("hello") {
exampleService.getMessage().map(Ok)
}
val api = (hello).handle {
case e: Exception => InternalServerError(e)
}
def main(): Unit = {
log.info(s"Serving the application on port ${port()}")
val server =
Http.server
.withStatsReceiver(statsReceiver)
.serve(s":${port()}", api.toServiceAs[Application.Json])
closeOnExit(server)
Await.ready(adminHttpServer)
}
}
You can see we’re initialising our service and then defining our /hello endpoint, which calles the getMessage function. As it returns a future, we use map to make a synchronous call to the Ok function, which will return a 200 OK response to the user with our message as the payload.
Next we define an api to handle our endpoint. This can also do some exception handling for us.
Finally the main function starts the server. We’re also adding stats and starting the admin server. This gives us Finagle’s admin web UI on port 9990 which has a load of useful information.
You can start the server with sbt run and test it with curl -s http://localhost:8081/hello.
Unit tests
To unit test the endpoint, we can use something like the following:
package com.andrewjones
import com.andrewjones.models.Message
import io.finch.Input
import org.scalatest.{FlatSpec, Matchers}
class ExampleSpec extends FlatSpec with Matchers {
import ExampleApplication._
behavior of "the hello endpoint"
it should "get 'Hello, world!'" in {
hello(Input.get("/hello")).awaitValueUnsafe() shouldBe Some(Message("Hello, world!"))
}
}
This is pretty self-explanatory. We make a call to the hello endpoint, wait for the future to complete, and ensure we get our message back. Run the tests with sbt test.
Alternatively we could test the service functions directly, which I’ll show later.
Accepting POST
Let’s expand the example a bit and accept a POST request with a JSON payload. For now, we’ll just accept the same kind of message that hello returned, and send it back.
Accepting a POST request isn’t that obvious, which is why I’ve added it to this example. First, you create an endpoint that accepts the JSON body. In our main application object, add the following:
def acceptedMessage: Endpoint[Message] = jsonBody[Message]
Then create another endpoint that accepts the acceptedMessage as part of the input:
def accept: Endpoint[Message] = post("accept" :: acceptedMessage) { incomingMessage: Message =>
exampleService.acceptMessage(incomingMessage).map(Ok)
}
Finally the code in ExampleService looks like this:
def acceptMessage(incomingMessage: Message): Future[Message] = {
Future.value(incomingMessage)
}
We’re just taking the incoming message and returning it as is. Start your server up and use the following curl command to test:
curl -d '{"message":"heres some post"}' -H "Content-Type: application/json" -X POST http://localhost:8081/accept
Finally the unit test, which again is pretty straightforward:
it should "post our message" in {
val input = Input.post("/accept")
.withBody[Application.Json](Buf.Utf8("{\"message\":\"heres some post\"}"), Some(StandardCharsets.UTF_8))
val res = accept(input)
res.awaitValueUnsafe() shouldBe Some(Message("heres some post"))
}
To accept some other JSON, you would simply create a new model describing the data and change the endpoint to accept it.
Calling a downstream API
For our last endpoint, let’s imagine we are calling a downstream API, maybe a micro-service, and presenting the results of that to the user. This will show off some of the benefits of using futures.
We’ll be calling the Github repositories API to get a list of repos for a particular user, and their size, then return that to the user.
Again, we’ll start with a model to represent our repositories.
package com.andrewjones.models
case class GithubRepositories(full_name: String, size: Long)
We’ll put the logic in a new service, as shown below. There’s quite a lot going on here, but we’ll go through it in a bit.
package com.andrewjones.services
import cats.data.EitherT
import com.andrewjones.exceptions.RateLimitException
import com.andrewjones.models.GithubRepositories
import com.twitter.finagle.http.{Request, Response}
import com.twitter.finagle.{Http, Service}
import com.twitter.logging.Logger
import com.twitter.util.Future
import io.catbird.util._
import io.circe.generic.auto._
import io.circe.parser.parse
class GithubService(
client: Service[Request, Response] = Http.client.withTls("api.github.com").newService(s"api.github.com:443")
) {
private val log = Logger.get(getClass)
def getRepositories(username: String): Future[List[GithubRepositories]] = {
val requestUrl = s"/users/$username/repos"
val req = Request(requestUrl)
req.contentType = "application/json"
req.userAgent = "andrew-jones.com-example"
req.accept = "application/vnd.github.v3+json"
// This for comprehension is just shorthand for calling flatMap
(for {
response <- EitherT.right(client(req))
rawJson <- EitherT
.fromEither[Future](parse(response.getContentString()))
.leftMap(failure => {
// TODO: we could do some error handling here and provide a more useful response to the user
log.error(failure, "error parsing JSON from content string")
List.empty[GithubRepositories]
})
repos <- EitherT
.fromEither[Future](rawJson.as[List[GithubRepositories]])
.leftMap(failure => {
log.error(failure, "error parsing JSON to GithubRepositories")
// TODO: we could do more error handling here and provide a more useful response to the user
if (response.statusCode == 403) {
response.headerMap.get("X-RateLimit-Remaining").foreach(remaining =>
if (remaining == "0") {
val exception = new RateLimitException(response.headerMap.get("X-RateLimit-Reset").get.toLong)
log.warning(exception.getMessage)
throw exception
}
)
}
List.empty[GithubRepositories]
})
} yield repos).merge
}
}
This new service takes an optional client parameter, which it will use to talk to Github. This will be useful for testing, as we’ll see later. By default, we set up a client to talk to the live Github API.
The getRepositories is going to be called by our endpoint. It takes a username parameter, which it will then use in the call to Github.
After setting up the request, we use the client to make the call (client(req)). This is where it gets interesting, as everything here on in is dealing with futures.
First off we’re using the for comprehension as shorthand for calling flatMap on each of the following futures (the left hand side of the arrows).
Then you will have noticed a lot of Either calls. This comes from the cats library, which provides abstractions for functional programming. The left hand side of the Either is an error, and called with leftMap, and the right hand side is success. For more information, see the documentation.
So if we start off with response <- EitherT.right(client(req)), we are making the request with the client and getting the response back, which is a future of type com.twitter.finagle.http.Response.
The next lines are parsing the JSON content of that response.
rawJson <- EitherT
.fromEither[Future](parse(response.getContentString()))
.leftMap(failure => {
// TODO: we could do some error handling here and provide a more useful response to the user
log.error(failure, "error parsing JSON from content string")
List.empty[GithubRepositories]
})
If the call to parse is successful, we get the raw JSON. Otherwise, we log the error, and return an empty list of repositories to the user. Obviously we could do more here, maybe returning an exception.
The next block of code is much the same. We deserialise the JSON in to a list of GithubRepositories objects, and again handle the failure. We also check for the X-RateLimit-Remaining header and return a useful exception if we’ve come up against the Github rate limit.
Finally we yield the list of repos and return it.
The key here is that every action is being called on a future and is being ran asynchronously. While we wait for a response from Github, we’re not blocking the execution of the application, which is free to handle other requests. Only when we get a response from Github do we execute the next block of code.
Finally in the ExampleApplication object we define our endpoint and add it to our api.
val githubService = new GithubService
def repositories: Endpoint[List[GithubRepositories]] = get("repositories" :: string) { username: String =>
githubService.getRepositories(username).map(Ok)
}
val api = (hello :+: accept :+: repositories).handle {
case e: Exception => InternalServerError(e)
}
Note how we can accept the string parameter and pass the username to the service, similar to how we accepted the post data. To test it, run sbt run and then curl -s http://localhost:8081/repositories/andrewrjones.
Testing with a mock Github response
When tesing this endpoint, we don’t want to make live calls to Github. The data might change, or Github might be down, and that shouldn’t fail our tests. So instead we’ll mock the service to return a known response to our code.
We’ll create a new test spec, GithubServiceSpec, which tests the service itself, rather than the endpoint. The code is shown below.
package com.andrewjones.services
import com.andrewjones.models.GithubRepositories
import com.twitter.finagle.Service
import com.twitter.finagle.http.Response.Ok
import com.twitter.finagle.http.{Request, Response}
import com.twitter.util.{Await, Future}
import org.mockito.Matchers.any
import org.mockito.Mockito.when
import org.scalatest.mockito.MockitoSugar
import org.scalatest.{FlatSpec, Matchers}
class GithubServiceSpec extends FlatSpec with MockitoSugar with Matchers {
behavior of "the github repositories service"
it should "get repositories" in {
val mockedService = mock[Service[Request, Response]]
val githubService = new GithubService(mockedService)
// mock the reply from GitHub
val reply = new Ok()
reply.contentString = scala.io.Source.fromFile("src/test/resources/repos.json").mkString
when(mockedService(any[Request])).thenReturn(Future.value(reply))
val response = Await.result(githubService.getRepositories("andrewrjones"))
response shouldBe
List(
GithubRepositories("andrewrjones/perl5-App-MP4Meta", 115),
GithubRepositories("andrewrjones/perl5-AtomicParsley-Command", 6624),
GithubRepositories("andrewrjones/perl5-Dist-Zilla-Plugin-Test-Fixme", 116)
)
}
}
Again, pretty obvious what’s going on here. We create our mocked server and pass that to GithubService, so it will make calls against this rather than Githubs API. We then tell mockedService what to respond with, in this case some JSON from a file. Finally we invoke the service and check the response.
Summary
This hopefully provided an overview of how we can use Finch and Finagle to build a JSON based API. We can accept requests over GET or POST and return either static JSON or a transformed result from another service. We’ve also added some unit tests to test the application.
The full code can be found on Github.
Cover image from Unsplash.