Working as a web freelancer I am interested in how different frameworks and technologies perform, but the majority of the benchmarks found over the internet are considering just the Hello World example.
When you are building a real life application there are more aspects to take into consideration, so I decided to run a complete benchmark between the most popular frameworks and technologies.
Alongside the performance, I was interested in how easy is to achieve specific tasks in each framework and what is the cost of scaling the application performance.
Who are the candidates
- Laravel 5, PHP 7.0, Nginx
- Lumen 5, PHP 7.0, Nginx
- Express JS 4, Node.js 8.1, PM2
- Django, Python 2.7, Gunicorn
- Spring 4, Java, Tomcat
- .NET Core, Kestrel
What are we testing
We are interested in the number of requests per seconds each framework achieves on different server configurations alongside how clean or verbose the code looks like.
How server configurations look
We are also interested in how each framework scales its performance and what is the price of achieving such performance. This is why we’ll test them on 3 different server configurations using DigitalOcean:
- 1 CPU, 512 MB – $5 / month
- 4 CPU, 8 GB – $80 / month
- 12 CPU, 32 GB – $320 / month
What are we building
We want to test a real life application example so we’ll basically build a Web REST API that exposes 4 endpoints, each one having a different complexity:
- Hello World – simply respond with a JSON containing Hello World string.
- Computation – compute the first 10.000 Fibonacci numbers.
- Simple Listing – we have a MySQL database containing a countries table and we’ll list all the countries.
- Complex Listing – we add a users table alongside a many-to-many mapping between users and countries and we want to list all users that visited France, alongside all countries each one visited.
For building the last two endpoints we’ll use the tools each framework provides to achieve our goal in the easiest possible way.
How are we testing them
For testing them we’ll use both wrk and ab HTTP benchmarking tools in order to check if we get similar results and also variate the concurrency of the requests so each technology can achieve its maximum potential.
These tools will run on their own droplet created on DigitalOcean so they don’t compete on server resources with the actual API application.
Also, the server used for placing the test requests and the one used to run the application are linked using their private IP so there won’t be any noticeable network latency.
Benchmark results
Below you can see the results grouped by each endpoint and also you can check on a single chart how each framework scaled on different server configurations.
How the API is built
Below are the actual controllers used for each framework to make an idea about how the code looks like. You can also check the whole code which is available on Github.
Laravel and Lumen with PHP
<?php
namespace App\Http\Controllers;
use Illuminate\Routing\Controller as BaseController;
use App\User;
use App\Country;
class Controller extends BaseController
{
public function hello()
{
return response()->json(['hello' => 'world']);
}
public function compute()
{
$x = 0; $y = 1;
$max = 10000 + rand(0, 500);
for ($i = 0; $i <= $max; $i++) {
$z = $x + $y;
$x = $y;
$y = $z;
}
return response()->json(['status' => 'done']);
}
public function countries()
{
$data = Country::all();
return response()->json($data);
}
public function users()
{
$data = User::whereHas('countries', function($query) {
$query->where('name', 'France');
})
->with('countries')
->get();
return response()->json($data);
}
}
Express JS with Node.js
const Country = require('../Models/Country');
const User = require('../Models/User');
class Controller
{
hello(req, res) {
return res.json({ hello: 'world' });
}
compute(req, res) {
let x = 0, y = 1;
let max = 10000 + Math.random() * 500;
for (let i = 0; i <= max; i++) {
let z = x + y;
x = y;
y = z;
}
return res.json({ status: 'done' })
}
async countries(req, res) {
let data = await Country.fetchAll();
return res.json({ data });
}
async users(req, res) {
let data = await User.query(q => {
q.innerJoin('UserCountryMapping', 'User.id', 'UserCountryMapping.userId');
q.innerJoin('Country', 'UserCountryMapping.countryId', 'Country.id');
q.groupBy('User.id');
q.where('Country.name', 'France');
})
.fetchAll({
withRelated: ['countries']
})
return res.json({ data });
}
}
module.exports = new Controller();
Django with Python
from django.http import JsonResponse
from random import randint
from models import Country, User, UserSerializer, CountrySerializer
def hello(req):
return JsonResponse({ 'hello': 'world' })
def compute(req):
x = 0
y = 1
max = 10000 + randint(0, 500)
for i in range(max):
z = x + y
x = y
y = z
return JsonResponse({ 'status': 'done' })
def countries(req):
data = Country.objects.all()
data = CountrySerializer(data, many=True).data
return JsonResponse({ 'data': list(data) })
def users(req):
data = User.objects.filter(usercountrymapping__countries__name='France').all()
data = UserSerializer(data, many=True).data
return JsonResponse({ 'data': list(data) })
Spring with Java
package app;
import java.util.List;
import java.util.Random;
import java.util.concurrent.atomic.AtomicLong;
import org.hibernate.Criteria;
import org.hibernate.SessionFactory;
import org.hibernate.criterion.Restrictions;
import org.json.JSONException;
import org.json.JSONObject;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import model.Country;
import model.User;
@RestController
public class Controller {
private SessionFactory sessionFactory;
public Controller(SessionFactory sessionFactory) {
this.sessionFactory = sessionFactory;
}
@RequestMapping(value = "/hello", produces = "application/json")
public String hello() throws JSONException {
return new JSONObject().put("hello", "world").toString();
}
@RequestMapping(value = "/compute", produces = "application/json")
public String compute() throws JSONException {
long x = 0, y = 1, z, max;
Random r = new Random();
max = 10000 + r.nextInt(500);
for (int i = 0; i <= max; i++) {
z = x + y;
x = y;
y = z;
}
return new JSONObject().put("status", "done").toString();
}
@RequestMapping(value = "/countries", produces = "application/json")
@Transactional
public List<Country> countries() throws JSONException {
List<Country> data = (List<Country>) sessionFactory.getCurrentSession()
.createCriteria(Country.class)
.list();
return data;
}
@RequestMapping(value = "/users", produces = "application/json")
@Transactional
public List<User> users() throws JSONException {
List<User> data = (List<User>) sessionFactory.getCurrentSession()
.createCriteria(User.class)
.createAlias("countries", "countriesAlias")
.add(Restrictions.eq("countriesAlias.name", "France"))
.list();
return data;
}
}
.NET Core
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.EntityFrameworkCore;
using dotnet_core.Models;
using dotnet_core.Data;
namespace dotnet_core.Controllers
{
public class MyController : Controller
{
private readonly ApplicationDbContext context;
public MyController(ApplicationDbContext context)
{
this.context = context;
}
[HttpGet]
[Route("/hello")]
public IEnumerable<string> hello()
{
return new string[] { "hello", "world" };
}
[HttpGet]
[Route("/compute")]
public IEnumerable<string> compute()
{
int x = 0, y = 1, z, max;
Random r = new Random();
max = 10000 + r.Next(500);
for (int i = 0; i <= max; i++) {
z = x + y;
x = y;
y = z;
}
return new string[] { "status", "done" };
}
[HttpGet]
[Route("/countries")]
public IEnumerable<Country> countries()
{
return context.Country.ToList();
}
[HttpGet]
[Route("/users")]
public object users()
{
return context.UserCountryMapping
.Where(uc => uc.country.name.Equals("France"))
.Select(uc => new {
id = uc.user.id,
firstName = uc.user.firstName,
lastName = uc.user.lastName,
email = uc.user.email,
countries = uc.user.userCountryMappings.Select(m => m.country)
})
.ToList();
}
}
}
Conclusions
Having in mind that in a real world application almost all the requests interact with the database, none of the choices are bad and all of them could handle the requirements of most web applications.
However, the performance of the Node.js with Express JS is quite remarkable. It competes with technologies such as Java and .NET Core or even outperforms them and combined with the simplicity of Javascript ES6 which you can natively use with Node.js 8, it provides so much power.
Regarding application scalability, the best performance per cost was gained on the middle size server. Adding 12 cores and 32 GB of memory didn’t help too much. Maybe, in this case, the bottle neck is somewhere else or it requires fine tunings to unlock the full server potential.
What do you think?
If you found the results interesting please click the 💚️ button so other can see them, too. You can find all the source code here: