Building a Yar TCP service with PHP Sockets中文EN

Yar supports two kinds of Transporter: HTTP and TCP. The HTTP one is based on CURL; in PHP, Yar defaults to using the HTTP Transporter — this should be familiar to everyone. The TCP-based one is probably used less.

In fact, six years ago I also wrote a C Yar server framework called Yar-c, the code is at Yar-C at Github. It provides service startup, worker process management, the Yar packaging protocol, and so on. At the time we used this framework to implement high-performance services like the Weibo whitelist, for the PHP side to call via the Yar Client.

It's just that Yar C requires the Handle to be written in C, which might be a little unfamiliar to quite a few PHPers. So today we'll try writing a TCP Server in PHP, to introduce how to handle the Yar RPC protocol. This example can be conveniently combined with async PHP frameworks like Swoole to build a high-performance Yar TCP Server. Along the way, it'll give you a look at Yar's RPC communication protocol, and incidentally some socket programming.

Today we'll still use the "whitelist" service as the example. We provide an interface that accepts requests from RPC clients; the parameter is a user ID, and it returns a bool indicating whether the user is on the whitelist:

function query(int $id) : bool;

First, we create a file yar_server. To make it directly executable, we write the following at the top of the file:

#!/bin/env php7
<?php
class WhiteList {
}

Then, we use chmod a+x to add execute permission to this file.

The first step is to handle the service startup arguments. We accept a parameter S indicating the IP and port to listen on; the value's format is host:port. We use PHP's getopt function to handle the command-line arguments:

class WhiteList {
    protected $host;

    public function __construct() {
        $options = getOpt("S:");
        if (!isset($options["S"])) {
            $this->usage();
        }
    }

    protected function usage() {
        exit("Usage: yar_server -S hostname:portn");
    }
}

This way, when the user starts yar_server without specifying the S parameter, we exit and print the Usage. We also need another configuration: pointing to a word-list file, where each line in the word list is a user ID on the whitelist. We use F for this:

class WhiteList {
    protected $host;
    protected $dicts;

    public function __construct() {
        $options = getOpt("S:F:");
        if (!isset($options["S"]) || !isset($options["F"])) {
            $this->usage();
        }
        $this->host = $options["S"];
        $this->dicts = $options["F"];
    }

    protected function usage() {
        exit("Usage: yar_server -F path_to_dict -S hostname:portn");
    }
}

OK, now the startup argument handling is done. Of course, for simplicity, I've omitted validity checks on the input parameters.

Next, we need to complete two functions. The first reads the word-list file specified by -F, loading all user IDs into an array. Since our service is long-running, we don't need to worry about performance; it only processes this word-list file during the startup phase:

protected function loadDict() {
	$this->ids = array();

	$fp = fopen($this->dicts, "r");
	while (!feof($fp)) {
		$line = trim(fgets($fp));
		if ($line) {
			$this->ids[$line] = true;
		}
	}
	fclose($fp);
	echo "Loading dict successfully, ", count($this->ids), " loadedn";

	return $this;
}

Since the user IDs are integers, we use them as Hashtable keys, so that later lookups using isset will be very efficient. Note that since file handling isn't the focus of today's post, we've omitted checks for file existence, readability, and validity.

OK, here comes the key part. We need to start an IPv4 TCP Socket service, listening at the address specified by $host. To make the Socket API easy to understand, we don't use PHP's Stream family of functions, but instead use the Socket-family API that PHP wraps directly. First we use socket_create to create a Socket socket:

protected function listen() {
	$socket = socket_create(AF_INET, SOCK_STREAM, SOL_TCP);
	if ($socket == false) {
		throw new Exception("socket_create() failed: reason: " . socket_strerror(socket_last_error()));
	}
}

Then, we need to use socket_bind to bind this Socket to the address we want to listen on, and use socket_listen to listen for requests:

protected function listen() {
	$socket = socket_create(AF_INET, SOCK_STREAM, SOL_TCP);
	if ($socket == false) {
		throw new Exception("socket_create() failed: reason: " . socket_strerror(socket_last_error()));
	}
	list($hostname, $port) = explode(":", $this->host);
	if (socket_bind($socket, $hostname, $port) == false) {
		throw new Exception("socket_bind() failed: reason: " . socket_strerror(socket_last_error()));
	}
	if (socket_listen($socket, 64) === false) {
		throw new Exception("socket_listen() failed: reason: " . socket_strerror(socket_last_error()));
	}
	echo "Starting Yar_Server at {$this->host}nPresss Ctrl + C to quitn";

	$this->socket = $socket;
	return $this;
}

OK, if everything's fine, we can now use socket_accept to listen for requests. The default socket is in blocking mode; if there's no request, the process will keep blocking and waiting. For a high-performance service, it's best to adopt a non-blocking + select or epoll model to handle multiple requests simultaneously. But our example is mainly to introduce Yar's protocol, so we still use the simple blocking mode.

Next, we write the actual RPC-handling part. First we accept a request via accept, then read the request's content, parsing the Yar RPC Header info in the request header. The Yar RPC protocol header is defined as follows:

typedef struct _yar_header {
    uint32_t       id;            // transaction id
    uint16_t       version;       // protocl version
    uint32_t       magic_num;     // default is: 0x80DFEC60
    uint32_t       reserved;
    unsigned char  provider[32];  // reqeust from who
    unsigned char  token[32];     // request token, used for authentication
    uint32_t       body_len;      // request body len
}

Among these, magic_num is a special value used to verify the validity of a request; legal Yar RPC requests all set this value to 0x80DFEC60 (I'd really like to tell you why this value is this, but I genuinely don't remember why I used this number back then). This header is 82 bytes. Some folks might ask, wait, doesn't this Struct look like it shouldn't be 82? That's because when the header is declared it uses pack mode, i.e. no alignment, so it really is 82 bytes.

pack("H*", "80DFEC60");

provider is a string that indicates the client's name. For example, for the Yar extension's Yar_Client it's "Yar PHP Cient-x.x.x".

token was originally designed for API key verification, but ended up not being used, because most are intranet applications where there are many ways to guarantee the legitimacy of the request source.

id is a unique request id, this is for troubleshooting request issues. version defaults to 0 or 1; since I haven't upgraded the protocol header, we don't need to worry about it for now. reserved can be used to pass some request parameters, e.g. the client can indicate whether to keep the connection.

body_len is what we need to care about; this field indicates how large the request body of this request is in total (not including the Yar protocol header).

All of these numbers are transmitted in network byte order. We use PHP's unpack function for handling binary streams to parse the binary stream read in:

protected function parseHeader($header) {
   return
     unpack("Nid/nversion/Nmagic_num/Nreserved/A32provider/A32token/Nbody_len", $header);
}

This function returns an array corresponding to the header struct described above.

Correspondingly, we also need to use pack to implement the method for generating a Yar Header:

const YAR_MAGIC_NUM = 0x80DFEC60;
protected function genHeader($id, $len) {
	$bin = pack("NnNNA32A32N", $id, 0, self::YAR_MAGIC_NUM, 0, "Yar PHP TCP Server", "", $len);
	return $bin;
}

As mentioned earlier, before accepting a request we need to verify the request's validity:

protected function validRequest($header) {
	if ($header["magic_num"] != self::YAR_MAGIC_NUM) {
		return false;
	}
	return true;
}

So the overall logical framework for request handling is roughly:

protected function accept() {
	while (($conn = socket_accept($this->socket))) {
		$buf = socket_read($conn, self::HEADER_SIZE, PHP_BINARY_READ);
		if ($buf === false) {
			socket_shutdown($conn);
			continue;
		}

		if (!$this->validHeader($header = $this->parseHeader($buf))) {
			$output = $this->response(1, "illegal Yar RPC request");
			goto response;
		}

		$buf = socket_read($conn, $header["body_len"], PHP_BINARY_READ);
		if ($buf === false) {
			$output = $this->response(1, "insufficient request body");
			goto response;
		}

		if (!$this->validPackager($buf)) {
			$output = $this->response(1, "unsupported packager");
			goto response;
		}

		$buf = substr($buf, 8); /* skip the 8 bytes of packager info */
		$request = $this->parseRequest($buf);
		if ($request == false) {
			$this->response(1, "malformed request body");
			goto response;
		}

		$status = $this->handle($request, $ret);

		$output = $this->response($status, $ret);
response:
		socket_write($conn, $output, strlen($output));

		socket_shutdown($conn); /* close writing */
	}
}

Now the overall framework is complete; we just need to finish the handle and response methods. handle calls the specified method based on the m in the user's request.

protected function handle($request, &$ret) {
	if ($request["m"] == "query") {
		$ret = $this->query(...$request["p"]);
	} else {
		$ret = "unsupported method '" . $request["m"]. "'";
		return 1;
	}
	return 0;
}

Now let's implement the query method itself. This is simple — just check whether the id is in the whitelist array:

protected function query($id) {
	return isset($this->ids[$id]);
}

OK, next we need to complete the response method. This method packs a return body that conforms to the Yar protocol, including the 82-byte header, 8 bytes of packager info, and the serialized response body. We need to choose whether to set the r or e field in the response body based on the status:

protected function response($status, $ret) {
	$body = array();

	$body["i"] = 0;
	$body["s"] = $status;
	if ($status == 0) {
		$body["r"] = $ret;
	} else {
		$body["e"] = $ret;
	}

	$packed = serialize($body);
	$header = $this->genHeader(0, strlen($packed) + 8);

	return $header . str_pad("PHP", 8, "") . $packed;
}

OK, we're almost done. Let's finish the startup method and destructor (close the socket):

public function run() {
	$this->loadDict()->listen()->accept();
}
public function __destruct() {
	if ($this->socket) {
		socket_close($this->socket);
	}
}

Now everything's ready. Finally, we add the following at the end of the file:

(new Whitelist)->run();

Before testing, we first prepare a test word list, e.g. ids from 1 to 1000:

seq 1 1 10000 > user_id.dict

Then start the service, listening on port 9000 of the local machine:

$ ./yar_server -F user_id.dict -S127.0.0.1:9000
Loading dict successfully, 1000 loaded
Starting Yar_Server at 127.0.0.1:9000
Presss Ctrl + C to quit

Good, the service started successfully. Then we use the Yar extension to write a client (you need to have the Yar extension installed first), and test the calling effect of user ids 999 and 99999:

<?php
$yar = new Yar_Client("tcp://127.0.0.1:9000");
var_dump($yar->query("999"));
var_dump($yar->query("99999"));
?>

Different from calling an HTTP Yar service, here we should use tcp:// as the address scheme to indicate this is a TCP service.

Come on, let's run it and see:

php7 client.php
bool(true)
bool(false)

Looks good, as expected!

You can also try deliberately constructing some error scenarios, e.g. calling a non-existent method, to see how the server reacts. You can find the code for this example here.

That concludes my introduction on how to write a Yar TCP service using PHP. You should be able to conveniently modify and refine this example into whatever format you want, or embed it into Swoole (you can refer to what the Swoole author wrote: here).

I should say again: since the main purpose of this article is to introduce the Yar RPC communication protocol, the service-management side isn't very polished. For example, socket_accept, socket_read/write all default to blocking mode, there's no timeout design, and there's only one service process. If you really want to use this as an actual service, some homework is still needed — but I'm confident that if you're interested, you can make it work. 🙂

Of course, the simplest option is that you can directly use the Yar-C service framework to write a C Yar TCP service.

There's also a Yar-C Server example here: yar_server in C.

enjoy!