This project implements an api service that gets all books by a specified author that have appeared on the New York Times Bestsellers list.
It sources this information from the New York Times Books api.
A high level overview of the flow of requests and responses is shown:
The components of this service include:
- Finch server with endpoints serving client requests
- Finagle client executing requests to the New York Times api
- Service that processes responses from the New York Times api to match the domain model
- Configuration loader
- In-memory cache, caching responses which expire after a specified duration.
- Logging middleware
- Request-meter middleware restricting the number of calls to the New York Times API to avoid rate limited responses
- Admin server that provides an interface with variety of tools for diagnostics, profiling, monitoring e.t.c
This server is built using finch, it runs on specified port, the default is 8087
.
An admin server as defined above is also started, this server runs on port 9990
.
- get all books from a specified author that have appeared on the New York Times bestsellers list
GET :8087 /books/author=<author>&year=<year>&year=<year>
{
"author": "cal newport",
"books": [
{
"name": "A WORLD WITHOUT EMAIL",
"publisher": "Portfolio",
"publicationYear": "2021"
},
{
"name": "DIGITAL MINIMALISM",
"publisher": "Portfolio/Penguin",
"publicationYear": "2019"
}
]
}
author
& year
params are validated by the server. author
is a required param while year
is optional.
author
param is expected to be non-empty and year
param is expected to match ^(\\d{4})$
which represents YYYY
format.
- admin interface with default port
:9990
- health check endpoint
:9990/health
The service depends on a finagle client that executes requests to the New York Times api. Its functionalities include:
- jittered back off retry for non
rate limitation
(429
) errors - session pool so connections can be reused for multiple requests
- requests with timeout so request wait forever
Responses from the New York Times api are cached for a default time (3 minutes
). Subsequent requests for related
queries are served from the cache and thereby faster and less risk for the requests to the New York Times api to be rate
limited.
Request metering is implemented as the New York Times api dependency
implements rate limits on calls. There is a rate limit of 500
requests per
day and 5
requests per minute.
This implies, calls to this service have to be metered to avoid hitting the rate limits. Meters are implemented for both the minute and daily limit. If a request to the service has a previous New York Times api response cached it is not metered.
To start the service, export your New York Times api key and run the command
export NYT_API_KEY=<key>
sbt run
By default, the api server runs on :8087
and the admin server runs on :9990
.
Docker images can be generated and published for this service. Running sbt docker:publishLocal
generates a local
docker image. To publish to a docker repository, please update the build file.
sbt test
- production instances will have the admin server port for internal requests only.
- new york times api doesn't actually provide the publication year of a book, it is assumed that the year of publication is similar to the year the book appear on a bestseller list.
This implementation can be improved in these areas:
- more metrics
- pagination of responses to clients
- more unit tests for components
- integration testing
- improve request metering