{"id":3254,"date":"2019-10-22T03:19:47","date_gmt":"2019-10-22T03:19:47","guid":{"rendered":"https:\/\/www.migenius.com\/?p=3254"},"modified":"2020-12-16T21:48:21","modified_gmt":"2020-12-16T21:48:21","slug":"cost-effective-rendering-with-amazon-sqs-and-s3","status":"publish","type":"post","link":"https:\/\/www.migenius.com\/articles\/cost-effective-rendering-with-amazon-sqs-and-s3","title":{"rendered":"Cost Effective Rendering with Amazon SQS and S3"},"content":{"rendered":"\n

Even though RealityServer is great at streaming fully interactive, server-side rendering directly to your browser, not every use case requires this level of interactivity. RealityServer has recently introduced a new feature called the Queue Manager which integrates with popular message queue services to manage rendering and other RealityServer tasks. In this article we will dive into the details on how to get up and running with this great new feature using Amazon SQS and S3 services.<\/p>\n\n\n\n\n\n\n\n

\"Queue<\/figure>\n\n\n\n

Why Queuing?<\/h3>\n\n\n\n

RealityServer will normally run all requests it is sent in parallel. Often times this is exactly what you want, particularly in a multi-user or collaborative application. However what about use cases where you don’t need full interactivity or live streaming and want to carefully manage your spend on GPU server resources? Or cases where you expect to get a large number of simultaneous requests, more than can be handled at once.<\/p>\n\n\n\n

The normal solution to this problem is queuing. You implement a job queue in front of RealityServer that takes in those jobs and then runs them in a controlled manner instead of all at once. This is such a common practice in web applications (e.g., for tasks like resizing images or transcoding video) that various services have sprung up to help with this.<\/p>\n\n\n\n

Enter the message queue<\/a>. Basically a message queue allows two programs to interact in an asynchronous manner. In our case, jobs are submitted to the queue (either by RealityServer itself or externally) and then at some later time they are retrieved by a listening RealityServer to execute. Since each listening RealityServer will only process a single job at a time, the queue effectively controls the flow of jobs to RealityServer.<\/p>\n\n\n\n

\n
\n

This also makes scaling out trivial. You just add more listening RealityServer instances and they will automatically pull jobs from the queue that others are not working on. This makes job distribution a piece of cake. All you need to do is start another RealityServer if you want to get through the jobs quicker.<\/p>\n\n\n\n

In this article we are going to talk specifically about using Amazon SQS as the underlying message queue and Amazon S3 for storing the results. If you have a particular message queue or storage platform you’d like to see supported let us know<\/a>.<\/p>\n<\/div>\n\n\n\n

\n
\"\"<\/figure>\n\n\n\n

<\/p>\n<\/div>\n<\/div>\n\n\n\n

Amazon SQS<\/h3>\n\n\n\n

Amazon SQS<\/a> or Simple Queue Service is a robust implementation of a message queue, provided as a turn-key service by Amazon Web Services<\/a> (AWS). This is the first external queuing service we are offering support for in RealityServer. We’ve implemented the Queue Manager in a modular way and anticipate adding additional integrations over time.<\/p>\n\n\n\n

SQS is easy to setup directly from the AWS web based console (though you can also do it through the AWS API if needed). We provide some instructions on setting up your SQS queues for use with RealityServer here<\/a>. The rest of this guide assumes you have already setup your queue.<\/p>\n\n\n\n

Amazon S3<\/h3>\n\n\n\n

It’s very likely you’ve already heard of Amazon S3<\/a> or at least downloaded files hosted there. S3 is a service for storing objects in the cloud. You can simply think of it as a place to store files. A big advantage of S3 is that you can make these files immediately accessible through a simple URL, without the need for a separate web server. You can also integrate S3 storage with Content Delivery Networks like Amazon CloudFront<\/a>.<\/p>\n\n\n\n

In this article we’ll use S3 as a place to store the results of the jobs we are going to queue and run. We also provide a generic HTTP Postback<\/a> task you can use if S3 isn’t to your liking but we’ll stick with the AWS theme here.<\/p>\n\n\n\n

Terminology<\/h3>\n\n\n\n

Let’s get a few basic terms out of the way before we jump into setting this up. Since some of these are quite similar and easy to confuse (particularly job and task) we thought we better make them clear.<\/p>\n\n\n\n

Queue Module<\/h4>\n\n\n\n

The queue module provides access to a specific type of queue platform (e.g., Amazon SQS). You’ll configure a particular queue module depending on what service you’re using. Right now we’ll just cover SQS.<\/p>\n\n\n\n

Queue<\/h4>\n\n\n\n

A queue is the place where jobs are stored. The queue manages the life cycle of a job (see below). The queue is externally hosted and not part of RealityServer.<\/p>\n\n\n\n

Command<\/h4>\n\n\n\n

The RealityServer command that will be called by a job to perform the actual work required. Jobs call a single command and this is just a regular RealityServer command (e.g., a custom V8 command you have written).<\/p>\n\n\n\n

Parameters<\/h4>\n\n\n\n

When calling a command in a job you usually will want to pass in parameters to the command, just like when calling a command directly. The parameters are included in the job and also stored in the queue.<\/p>\n\n\n\n

Task<\/h4>\n\n\n\n

The action you want to be performed after a queued job completes. This might be something like uploading an image rendered by the job to a server.<\/p>\n\n\n\n

Job<\/h4>\n\n\n\n

A job is a combination of a command, parameters and tasks. The job is what is inserted into the queue and what gets pulled off the queue when work is to be done.<\/p>\n\n\n\n

Configuration<\/h3>\n\n\n\n

Most of the work required to get going with the queue manager configuration. The queue manager and AWS services we support have a set of new configuration directives you can use in realityserver.conf<\/em> to control how they work. For these examples we’re going to make the following assumptions. Obviously you’ll need to modify for your configuration.<\/p>\n\n\n\n