Simple Messaging System with Redis and Node.js

Hey all, been a while since I posted anything constructive, I’ve been so busy wasting my time creating games for Facebook Instant Games Messenger (I will do a proper write up with my analysis and final findings / thoughts on this very soon). Not all has been lost working on Instant Games however. Two of my games required an instant messaging system that enables me to send real-time messages between players which ironically the Instant Games SDK doesn’t cater for. So I created one using Node.js and Redis (these two bad boy pieces of tech together are like sweet music). You can grab the code from Github here.

Note that you will need to install this to your own server, I like to run most of my node modules using PM2. So to get the messaging system up and running just run pm2 msys. The server looks to the /msys endpoint, but you can change this in msys.js if you need it to go elsewhere. No, I don’t answer questions on how to set up servers and mess with Apache config files because I hate all that junk, it gets in the way of my actual coding which I do enjoy :). If you cannot do this stuff yourself then you probably should be paying someone else do this for you.

Oh word of warning, any messages sent will time out after 7 days (this is to keep Redis memory usage down), but you can extend this to whatever time limit you like. Messages are queued, when you collect the pending messages it collects them all and deletes them from the database.

Ok, how to use client side? Here is a simple class (erm I mean collection of functions) with an awesome original name that I ripped out of one of my games for you guys to use:

[sourcecode language=”js”]
var Backend = {};

Backend.SendMessage = function(data, done_callback)
{
var url = "https://yourdomain.com/msys?c=s&t=<your token>";
url += "&g=1";
url += "&u=" + data.to_id;
url += "&d=" + encodeURIComponent(JSON.stringify(data));
b5.Utils.SendGetRequest(url, function(response) {
if (done_callback !== undefined)
done_callback(response);
})
}

Backend.SendMessageMulti = function(recipients, data, done_callback)
{
var users = "";
var len = recipients.length;
for (var t = 0; t < len; t++)
{
users += recipients[t];
if (t < (len – 1))
users += "_";
}
var url = "https://yourdomain.com/msys?c=s&t=<your token>";
url += "&g=1";
url += "&m=" + users;
url += "&d=" + encodeURIComponent(JSON.stringify(data));
b5.Utils.SendGetRequest(url, function(response) {
if (done_callback !== undefined)
done_callback(response);
})
}

Backend.GetMessages = function(done_callback)
{
var url = "https://yourdomain.com/msys?c=g&t=<your token>";
url += "&g=1";
url += "&u=" + Social.GetPlayerID();
b5.Utils.SendGetRequest(url, function(response) {
if (response.status == 200)
{
var data = decodeURIComponent(response.responseText);
var obj = JSON.parse("[" + data + "]");

if (done_callback !== undefined)
done_callback(obj);
}
else
{
if (done_callback !== undefined)
done_callback();
}
})
}
[/sourcecode]

There are a few functions in here that you will need to implement yourself:

  • Social.GetPlayerID() – Replaced with your players user ID, if you are using Facebook Instants SDK then use FBInstant.player.getID()
  • b5.Utils.SendGetRequest() – Performs a GET request, e.g:

[sourcecode language=”js”]
b5.Utils.SendGetRequest = function(url, callback)
{
var req = new XMLHttpRequest();
req.onreadystatechange = function()
{
if (callback != undefined && req.readyState == 4)
callback(req);
}
req.open("GET", url, true);
req.send();
}
[/sourcecode]

Anyway, that’s it, hope you find more useful than I did. Happy coding :).

Creating a Unity leaderboard using node.js and redis

I’ve recently added a new node.js based leaderboard system for Unity to Github. I’m going to use something like this in my next game which will feature global leaderboards as a major focus of the game.You can grab the code from Github.

Let start by taking a look at what the repo contains:

Node.js Server

The server side code is written in JavaScript using Node.js with Redis for super fast storage. To use this code you will need a web server that allows you to host your own node.js scripts. There are plenty of super cheap VPS providers around these days so if you go that route take a look at my article on installing node on your own VPS here. You will also need Redis, take a look at my article on installing Redis here.

Once you have everything installed and setup, run main.js located in the Server folder to set the leaderboards server off going:


node main.js

Once running, the server will be listening for connections from Unity on the default port which is set in server.js.

The entire leaderboards service consists of 3 files:

  • server.js – The web server which interacts with the Uniy app
  • leaderboard.js – The leaderboard code which communicates with Redis to store an retrieve persistent data
  • main.js – A main file to create the server and set it off running

The code in server,js is straight forward and much of it was already covered in this article here. The main changes include dealing with data hiding, we now pass the data base64 encoded via the d parameter:

[sourcecode language=”js”]
var vars = body.split("=");
if (vars[0] == "d")
{
var data = Buffer.from(vars[1], "base64");
console.log("Data: " + data);
this.processCommand(data.toString(), res);
}
[/sourcecode]

Here we decode the base 64 string data then pass it on to be processed. This enables me to encrypt the data at the other end then decrypt it when it arrives at this end (not covered in this article), this will allow me to hide what is sent and minimise the chance that someone spams my leaderboards with fake data.

The next major change is to handle commands that are passed to the server, to allow the client to perform different actions such as submit a score or get the users rank etc..

leaderboard.js was added to take care of all actions that are related to dealing with the actual leaderboard data, such as chatting to Redis about what to do with the data. The leaderboard class handles all of this including pushing multiple commands to Redis in one go instead of sending them separately to speed things up. For example sending scores to multiple leaderboards at the same time:

[sourcecode language=”js”]
setUserScores(userName, scores, cb)
{
var multi = this.client.multi();
for (var t = 1; t < this.MAX_BOARDS + 1; t++)
{
var name = this.boardName + ":" + t;
if (scores[t – 1] > 0 && scores[t – 1] < this.MAX_SCORE)
multi.zadd([name, scores[t – 1], userName]);
}
multi.exec((err, replies) =>
{
if (cb) cb(err);
});
}
[/sourcecode]

Here we issue many zadd commands to redis at the same time.

Unity Client

The Unity client code is implemented in Leaderboards.cs located in the Client folder. Lets take a quick look at the code:

[sourcecode language=”csharp”]
using UnityEngine;
using UnityEngine.Networking;
using System.Collections;
using System;
using System.Text;

public class Leaderboards : MonoBehaviour
{
public const int RANK_INVALID = -1; // The rank is invalid

private static string _Url = "http://localhost";
private static string _Port = "8080";

// Submits users score to the server.
//
// @param which Leaderboard index to submit score to
// @param score Score to submit
// @param userName Name of user to submit score for
// @param OnScoreSubmitted Callback to call when operation completes or fails, a bool is passed to the callback which
// is true if an error occurred or false if not
//
public void SubmitScore(int which, int score, string userName, Action<bool> OnScoreSubmitted)
{
StartCoroutine(SubmitScoreToServer(which, score, userName, OnScoreSubmitted));
}

private IEnumerator SubmitScoreToServer(int which, int score, string userName, Action<bool> OnScoreSubmitted)
{
Debug.Log("Submitting score");

// Create a form that will contain our data
WWWForm form = new WWWForm();
StringBuilder sb = new StringBuilder("m=score");
sb.Append("&w=");
sb.Append(which.ToString());
sb.Append("&s=");
sb.Append(score.ToString());
sb.Append("&n=");
sb.Append(userName);
byte[] bytes = Encoding.UTF8.GetBytes(sb.ToString());
form.AddField("d", Convert.ToBase64String(bytes));

// Create a POST web request with our form data
UnityWebRequest www = UnityWebRequest.Post(_Url + ":" + _Port, form);
// Send the request and yield until the send completes
yield return www.Send();

if (www.isError)
{
// There was an error
Debug.Log(www.error);
if (OnScoreSubmitted != null)
OnScoreSubmitted(true);
}
else
{
if (www.responseCode == 200)
{
// Response code 200 signifies that the server had no issues with the data we sent
Debug.Log("Score send complete!");
Debug.Log("Response:" + www.downloadHandler.text);
if (OnScoreSubmitted != null)
OnScoreSubmitted(false);
}
else
{
// Any other response signifies that there was an issue with the data we sent
Debug.Log("Score send error response code:" + www.responseCode.ToString());
if (OnScoreSubmitted != null)
OnScoreSubmitted(true);
}
}
}

// Submits a collection of scores to the server.
//
// @param scores Scores to submit, once score per leaderboard
// @param userName Name of user to submit score for
// @param OnScoresSubmitted Callback to call when operation completes or fails, a bool is passed to the callback which
// is true if an error occurred or false if not
//
public void SubmitScores(int[] scores, string userName, Action<bool> OnScoresSubmitted)
{
StartCoroutine(SubmitScoresToServer(scores, userName, OnScoresSubmitted));
}

private IEnumerator SubmitScoresToServer(int[] scores, string userName, Action<bool> OnScoresSubmitted)
{
Debug.Log("Submitting scores");

WWWForm form = new WWWForm();
StringBuilder sb = new StringBuilder("m=scores");
sb.Append("&a=");
for (int t = 0; t < scores.Length; t++)
{
sb.Append(scores[t].ToString());
if (t < scores.Length – 1)
sb.Append(",");
}
sb.Append("&n=");
sb.Append(userName);
byte[] bytes = Encoding.UTF8.GetBytes(sb.ToString());
form.AddField("d", Convert.ToBase64String(bytes));

UnityWebRequest www = UnityWebRequest.Post(_Url + ":" + _Port, form);
yield return www.Send();

if (www.isError)
{
Debug.Log(www.error);
if (OnScoresSubmitted != null)
OnScoresSubmitted(true);
}
else
{
if (www.responseCode == 200)
{
Debug.Log("Scores sent complete!");
Debug.Log("Response:" + www.downloadHandler.text);
if (OnScoresSubmitted != null)
OnScoresSubmitted(false);
}
else
{
Debug.Log("Scores sent error response code:" + www.responseCode.ToString());
if (OnScoresSubmitted != null)
OnScoresSubmitted(true);
}
}
}

// Gets the user rank from the server.
//
// @param which Leaderboard index to submit score to
// @param score Score to submit
// @param userName Name of user to submit score for
// @param OnScoreSubmitted Callback to call when operation completes or fails, an int is passed to the callback which
// represents the users rank
//
public void GetRank(int which, string userName, Action<int> OnRankRetrieved)
{
StartCoroutine(GetRankFromServer(which, userName, OnRankRetrieved));
}

private IEnumerator GetRankFromServer(int which, string userName, Action<int> OnRankRetrieved)
{
Debug.Log("Getting rank");

WWWForm form = new WWWForm();
StringBuilder sb = new StringBuilder("m=rank");
sb.Append("&w=");
sb.Append(which.ToString());
sb.Append("&n=");
sb.Append(userName);
byte[] bytes = Encoding.UTF8.GetBytes(sb.ToString());
form.AddField("d", Convert.ToBase64String(bytes));

UnityWebRequest www = UnityWebRequest.Post(_Url + ":" + _Port, form);
yield return www.Send();

if (www.isError)
{
Debug.Log(www.error);
if (OnRankRetrieved != null)
OnRankRetrieved(RANK_INVALID);
}
else
{
if (www.responseCode == 200)
{
Debug.Log("Get rank complete!");
Debug.Log("Response:" + www.downloadHandler.text);
if (OnRankRetrieved != null)
{
int rank = RANK_INVALID;
int.TryParse(www.downloadHandler.text, out rank);
OnRankRetrieved(rank);
}
}
else
{
Debug.Log("Get rank error response code:" + www.responseCode.ToString());
if (OnRankRetrieved != null)
OnRankRetrieved(RANK_INVALID);
}
}
}

// Gets the users ranks from the server.
//
// @param userName Name of user to submit score for
// @param OnRanksRetrieved Callback to call when operation completes or fails, callback has an array of ints, with
// each element representing a leaerboard rank
//
public void GetRanks(string userName, Action<int[]> OnRanksRetrieved)
{
StartCoroutine(GetRanksFromServer(userName, OnRanksRetrieved));
}

private IEnumerator GetRanksFromServer(string userName, Action<int[]> OnRanksRetrieved)
{
Debug.Log("Getting ranks");

WWWForm form = new WWWForm();
StringBuilder sb = new StringBuilder("m=ranks");
sb.Append("&n=");
sb.Append(userName);
byte[] bytes = Encoding.UTF8.GetBytes(sb.ToString());
form.AddField("d", Convert.ToBase64String(bytes));

UnityWebRequest www = UnityWebRequest.Post(_Url + ":" + _Port, form);
yield return www.Send();

if (www.isError)
{
Debug.Log(www.error);
if (OnRanksRetrieved != null)
OnRanksRetrieved(null);
}
else
{
if (www.responseCode == 200)
{
Debug.Log("Get ranks complete!");
Debug.Log("Response:" + www.downloadHandler.text);
if (OnRanksRetrieved != null)
{
string[] sranks = www.downloadHandler.text.Split(‘,’);
int[] ranks = new int[sranks.Length];
int i = 0;
foreach (string s in sranks)
{
ranks[i] = RANK_INVALID;
int.TryParse(sranks[i], out ranks[i]);
i++;
}
OnRanksRetrieved(ranks);
}
}
else
{
Debug.Log("Get ranks error response code:" + www.responseCode.ToString());
if (OnRanksRetrieved != null)
OnRanksRetrieved(null);
}
}
}
}
[/sourcecode]

This class contains four methods at present:

  • SubmitScore – Submits a score to the server
  • SubmitScores – Submits a collection of scores to the server
  • GetRank – Gets the users rank from the server
  • GetRanks – Gets all of the users ranks from the server

Internally each request to the server is called via a coroutine and the supplied callback that we pass is called when a response from the server comes in, e.g.:

[sourcecode language=”csharp”]
Leaderboards lbds = GetComponent<Leaderboards>();
lbds.GetRank(which, (int rank) =>
{
// Do something with the result
});
[/sourcecode]

Its also worth mentioning that when we send data to the server we make some effort to hide it which makes it a little harder for someone to spam the server with false scores, e.g.:

[sourcecode language=”csharp”]
// Create a form that will contain our data
WWWForm form = new WWWForm();
StringBuilder sb = new StringBuilder("m=score");
sb.Append("&w=");
sb.Append(which.ToString());
sb.Append("&s=");
sb.Append(score.ToString());
sb.Append("&n=");
sb.Append(userName);
byte[] bytes = Encoding.UTF8.GetBytes(sb.ToString());
form.AddField("d", Convert.ToBase64String(bytes));
[/sourcecode]

Note how we firstly encode the data to bytes then convert it to a base 64 string, this enables us to add a step between, where we can encrypt the byte data using whatever means we like.

Whats coming next….

I plan to extend the server to allow queries of the data, so I can for example view all of the leaderboards from a website and possibly on device.

Installing Redis to Windows / Linux

I recently began adding support for a global leaderboard system to my latest Unity game and decided that instead of the huge overhead related to going the RDBMS SQL route that i would have a crack at implementing back-end storage using a NoSQL database instead, which whilst much faster than traditional RDBMS is much more restrictive in terms of query. In fact Redis is for all intents and purposes almost impossible to query in any kind of depth. That said, I am not particularly interested in its relational query features but more in its ability to store and run basic queries on data very quickly, which Redis does extremely well.

Note that I am writing this article with a view to using Redis from Node.js, I will cover using Redis from .NET Core in a later article. If you missed my article on installing node.js then look here.

What is Redis

Redis is a fast in memory key/value store that can be used like a database. It differs aainly to SQL in that complex queries cannot be ran on the data, but if you do not need to carry out complex queries on the data (for example a persistent game world map or a leaderboard) then something like redis is ideal for storing the data.

Installing Redis

Windows

For Windows download the MSI release from here then install it. Redis will be installed as a service. You can take a look in Task Manager to ensure that Redis is running. To confirm the install open up command prompt and run Redis command line interface redis-cli.

Linux

Installing redis on Linux is not so easy because you need to build the latest stable release, firstly lets get the tools you are going to need to build Redis:


sudo apt-get update
sudo apt-get install build-essential
sudo apt-get install tcl8.5

Download and untar the latest stable build:


wget http://download.redis.io/releases/redis-stable.tar.gz
tar xzf redis-stable.tar.gz

Now lets build Redis:


cd redis-stable
make
make test
sudo make install

Run Redis as a background daemon (a script has been provided to do this), note that you will be asked to set some options, just choose the defaults for the moment:


cd utils
sudo ./install_server.sh

The above script creates a script in etc\init.d called redis_6379 (the numbers represent the port that you selected during install).

To get Redis to run automatically on boot:


sudo update-rc.d redis_6379 defaults

Installing Redis client library for Node.js

This is actually very simple, run the following:


npm install redis

We can now reference Redis from Node.js by importing Redis in our Node.js code:


var redis = require("redis");

Testing Redis with Node.js

Create a new file called redis_test.js and add the following code:


var redis = require("redis");

// Create the redis client
var client = redis.createClient();
client.on("error", function (err)
{
console.log("Redis error: " + err);
});

// Try out a simple command
client.set("my_key", "some value");

For more information on Redis see the Redis website

Getting Unity 3D and node.js talking

I’m working on a Unity 3D game at the moment that needs a global leaderboard system that works across platform and not tied into the likes of Google Play or Game Centre. After taking a look at various technologies including my old favourite .NET (specifically thew newish .NET Core) I decided to use node.js because a) I want to learn it and b) .NET is a bit of overkill I think.

So I installed node.js then wrote a small server:

[sourcecode language=”js”]
"use strict";

var http = require("http");

class Server
{
constructor()
{
this.port = 8080;
this.ip = "localhost";

this.start();
}

start()
{
this.server = http.createServer((req, res) =>
{
this.processRequest(req, res);
});

this.server.on("clientError", (err, socket) =>
{
socket.end("HTTP/1.1 400 Bad Request\r\n\r\n");
});
console.log("Server created");
}

listen()
{
this.server.listen(this.port, this.ip);
console.log("Server listening for connections");
}

processRequest(req, res)
{
// Process the request from the client
// We are only supporting POST
if (req.method === "POST")
{
// Post data may be sent in chunks so need to build it up
var body = "";
req.on("data", (data) =>
{
body += data;
// Prevent large files from benig posted
if (body.length > 1024)
{
// Tell Unity that the data sent was too large
res.writeHead(413, "Payload Too Large", {"Content-Type": "text/html"});
res.end("Error 413");
}
});
req.on("end", () =>
{
// Now that we have all data from the client, we process it
console.log("Received data: " + body);
// Split the key / pair values and print them out
var vars = body.split("&");
for (var t = 0; t < vars.length; t++)
{
var pair = vars[t].split("=");
var key = decodeURIComponent(pair[0]);
var val = decodeURIComponent(pair[1]);
console.log(key + ":" + val);
}
// Tell Unity that we received the data OK
res.writeHead(200, {"Content-Type": "text/plain"});
res.end("OK");
});
}
else
{
// Tell Unity that the HTTP method was not allowed
res.writeHead(405, "Method Not Allowed", {"Content-Type": "text/html"});
res.end("Error 405");
}
}

}

module.exports.Server = Server;
[/sourcecode]

Create a file called server.js and add the above code.

Note that the above code is designed as a module so you can import it into your own code, e.g.:

[sourcecode language=”js”]
var server = require(‘./server’);

var httpServer = new server.Server();
httpServer.listen();
[/sourcecode]

Create a file called main.js and add the above code. Run the server by running node main.js

In the server code we create an instance of a HTTP client then we begin listening for connections. When a connection comes in we read the POST data and print out the key / value pairs that are sent.

On the Unity side we write the following code to test out our mini server:

[sourcecode language=”csharp”]
using UnityEngine;
using UnityEngine.Networking;
using System.Collections;

public class Leaderboards : MonoBehaviour
{
private static string _Url = "http://localhost";
private static string _Port = "8080";

public void SubmitScore(int which, int score)
{
StartCoroutine(SubmitScoreToServer(which, score));
}

private IEnumerator SubmitScoreToServer(int which, int score)
{
Debug.Log("Submitting score");

// Create a form that will contain our data
WWWForm form = new WWWForm();
form.AddField("which", which.ToString());
form.AddField("score", score.ToString());

// Create a POST web request with our form data
UnityWebRequest www = UnityWebRequest.Post(_Url + ":" + _Port, form);
// Send the request and yield until the send completes
yield return www.Send();

if (www.isError)
{
// There was an error
Debug.Log(www.error);
}
else
{
if (www.responseCode == 200)
{
// Response code 200 signifies that the server had no issues with the data we went
Debug.Log("Form sent complete!");
Debug.Log("Response:" + www.downloadHandler.text);
}
else
{
// Any other response signifies that there was an issue with the data we sent
Debug.Log("Error response code:" + www.responseCode.ToString());
}
}
}
}
[/sourcecode]

In the above code we create a form, add the data that we wish to send then create a web request (UnityWebRequest) to send the form to our test server. We send the request then yield the coroutine until the send completes. We finally test the response code to ensure that our data was sent ok,

To test, add the component to an object and then call SubmitScore(1, 100), you should see the following displayed from node:

Server created
Server listening for connections
Received data: which=1&score=100
which:1
score:100

And in the Unity console you should see:
Submitting score
Form sent complete!
Response:OK

At the moment this is still pretty much test code and doesn’t handle security other than to check for idiots sending really POST data. You should really encrypt the data sent to the server to prevent even bigger idiots from spamming your server with false information.

Next I will be looking at redis as a persistent store for my leaderboard data. I will be extending the node.js server code to include this functionality. I may do a blog on redis soon and then one another that includes details on how to use redis to create a leaderboard.

Installing and running Node.js on a VPS

Introduction

I recently had the requirement to implement a global none app store specific leaderboard system that can track scores and players for a mobile game that I am developing in Unity3D. After much investigation node.js seems to be the technology to use to create a server to handle it. I want it to be as quick as possible without having to resort to C++, plus I’ve been looking for an excuse to play with node.js in more detail. Ok, so I went to ovh.net and purchased one of their super cheap VPS (Virtual private server). I opted for Debian 8 LAMP (this is the Debian 8 Linux OS with, Apache web server, MySQL and PHP). After reading up the set up instructions, I managed to get logged into my VPS using PUTTY on Windows.

Installing node.js on a VPS

First thing to do is install node.js to the VPS. In the Putty terminal type the following:

Download and run the Nodesource PPA (personal package archive) installer script:

cd ~
curl -sL https://deb.nodesource.com/setup_6.x -o nodesource_setup.sh
sudo bash nodesource_setup.sh

Install node.js:

sudo apt-get install nodejs

Install the build essentials for some packages that require compilation

sudo apt-get install build-essential

Testing out the node.js installation

Ok, so now we have node.js installed, its time to create a hello world and run it to ensure that it works. In terminal create hello.js:

cd ~
nano hello.js

If you do not have the nano text editor installed then you can install it as follows:

sudo apt-get install nano

Once the file is open add the following code:

[sourcecode language=”js”]
#!/usr/bin/env nodejs
var http = require(‘http’);
http.createServer(function (req, res)
{
res.writeHead(200, {‘Content-Type’: ‘text/plain’});
res.end(‘Hello World!\n’);
}).listen(8080, ‘localhost’);
console.log(‘Server is running at http://localhost:8080/’);
[/sourcecode]

Make the script executable:

chmod +x ./hello.js

Run the script:

./hello.js

Note that the script is blocking because it is sat in an infinite loop waiting for connections, so you can no longer type anything into terminal. To test, run another instance of terminal (Putty on Windows) and enter the following to test the script:

curl http://localhost:8080

You should see “Hello World” printed out which is the response from the hello.js script.

Installing and using Process Manager (PM2) for node.js

The next issue we need to look at is how to make our script run in the background so it does not block. To do that we need install a tool called PM2 (process manager for node.js):

sudo npm install -g pm2

Once installed we can set our script off running as a background process with:
pm2 start hello.js

And to make PM2 re-run when the server reboots:

pm2 startup systemd

Note that when PM2 restarts it will restart all of its running processes.

Getting node.js to work with Apache

The general idea is to run the node.js server on a different port and forward requests to a specific url to this port using a reverse proxy. To do this we need to update an Apache config file httpd.conf. Note that if you do not find this file in the /etc/apache2/ directory then you will need to create it and add the following text:

ProxyPass /node http://localhost:8000/
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so

And add the following line to /etc/apache2/apache2.conf to ensure that the file gets included

Include /etc/apache2/httpd.conf

Note that it is likely that the proxy module is not enabled on Apache, in which case enable it using:

a2enmod proxy_http

Now requests to http://www.yourdomain.com/node will be forwarded to localhost:8000, adjust the original hello world node.js script to listen on port 8000 and give it a test.