Be-Delphi is organizing their first (of many) Delphi Developer Day on November 17th in Edegem near Antwerp. That day will be completely dedicated to Delphi and Prism.
At Be-Delphi, Devia will be holding a talk about the new LiveBindings in Delphi XE2, so be sure to grab a hold of me and say hello !
Setting up a Virtual Cluster to speed up Compressorwritten by Stefaan Lesage on 16/01/2009
In the previous article we had a look at two ways to get your Sequence from Final Cut Pro into Compressor and we had a look at the pro's and con's of each work-flow. We also discovered that using an intermediate QuickTime movie allowed us to use Final Cut Pro while Compressor was doing it's job. In this article we will have a look at how we can increase performance on the Compressor part of the work-flow by setting up a Virtual Cluster.
Final Cut Studio Machine
In the previous article 'Exporting from FCP to Compressor, or use a QuickTime Movie instead ?', we had a look at two ways to get your Sequence from Final Cut Pro into Compressor and we had a look at the pro's and con's of each work-flow. We also discovered that using an intermediate QuickTime movie allowed us to use Final Cut Pro while Compressor was doing it's job. In this article we will have a look at how we can increase performance on the Compressor part of the work-flow by setting up a Virtual Cluster.
But in today's world our Mac's have multiple processors and / or cores. Our machines have a lot of Raw Processing Power, but I was under the impression that we wern't using the full power of our MacPro Octo.
In my previous tests, I was using an average of 35% CPU power across all 8 cores in my MacPro. There must be a way to use more of it, and the solution is to setup and use a QuickCluster or so called Virtual Cluster !
Setting up the Virtual Cluster
If you have installed Final Cut Studio, you have a whole package of tools and utilities that can assist you in your daily Video Editing and Compression Tasks. One of those utilities is called Apple QMaster. QMaster can be used to make a Compressor Cluster consisting of a combination of Shared Storages, XServers, Macs and fibre chanel which are all connected to a Network. QMaster will split and distribute a compression job for you. But even if you have just a single MacPro, you can still use QMaster to create a Virtual Render Cluster, which can significantly boost compression speeds.
Open up the System Preferences and select the Apple QMaster option listed under Other. You should get a the QMaster Preferences pane which looks like this :
QMaster Preferences pane
In order to set up a Virtual Render Cluster, we have chosen the QuickCluster with services option under Share this computer as. Under Services, you should normally find 2 entries. One which is labeled Shake and other shell command processing, and the other one called Distributed processing for Compressor. Since we are looking on ways to speed up Compressor, we need to take a closer look at the Distributed processing for Compressor service. If you select that service, you will notice Selected Service on (1 Instance) next to the options button, and that is exactly where there is room for improvement.
This actually means that we have one instance of compressor which will be used to process the Compressor job. Since we have 2x4 cores in our MacPro we can safely increase the number of instances compressor will use. I have been experimenting a bit, and from what I read you should get best performance with 1 compressor instance for every 2 cores. So if you have a Quad Core, you can use 2 instances, if you have an Octo you can use 4 cores. It is also a general rule of thumb to have at least 1 Gb of memory for each Compressor instance. Depending on your system you can play with the number of instances. I discovered that on my machine, using 5 instances for compressor gave me the best performance. Compressor finishes the job a lot faster, while I'm still able to do some other things on my machine.
So, select Distributed processing for Compressor from the list of Services, check the Share check-box (no idea what the Managed on is for yet, but you can check that one too if you want). Next, click on the Options for selected service... button and enter the number of compressor instances you want to use.
Setting up 5 Compressor Instances
The only thing we now have to do is enter a name for our Quick Cluster or Virtual Cluster. Fill in the name under Identify this QuickCluster as:. On our Virtual Cluster we used MPCluster as the name. Once everything is set up correctly, click the Start Sharing button. There you go, you have a Virtual Cluster ready.
Using the Virtual Cluster
If we have a look at our previous article, we know we had two possible work-flows. We could export from FCP to Compressor, or we could Export a QT Movie and bring that into compressor. Sadly using a Virtual Cluster, you can only use the last technique, which means you will have to export a QT Movie first. And here it really depends which QT Movie you use as the source in order to get most our of the Virtual Cluster.
Lets try the same approach as we used before. We exported a 3 minutes 41 seconds HDV1080i50 clip to a Self Contained QT Movie. Now we can bring that into compressor, add the two targets, and set the destination.
Submit the Compressor Job to our QuickCluster / Virtual Cluster
Clicking on the Submit button brings up a new window, and you will notice that you can now choose your Virtual Cluster from the list of clusters. You should now have 2 entries there This Computer and in our setup also MPCluster which is our Virtual Render Cluster. Choose your Virtual Cluster and hit Submit.
Hitting Submit again will now send the job to compressor, and that will try to use the available Compressor instances to finish it's task. In our case the job had 2 targets and each Target got split into 2 segments (one for audio and one for video), this is (probably) due to the fact that our Sample was just too short to split it up even more.
Time needed to finish the job
Compressor took 9 minutes and 50 seconds to finish off the task, which is about 5 minutes faster than our test without the Virtual Cluster. It still only used about 65% average CPU power on all 8 cores, but again, that might be because the compression job was just too small.
It is indeed a good start, but the technique really starts to shine if you add more targets or have bigger compression jobs. Our current job was just too small for Compressor to split it into enough segments which it can process simultaneously. I needed a bigger sample in order to demonstrate it.
Just to demonstrate that I did another test run.
Some more test results
This time I wanted a bigger Compression job so I created a new Sequence and nested my old sample sequence into it 5 times in a row. This gave me a Sequence with the same settings as before, but which is now 5 times the duration for a total op 18 minutes and 26 seconds.
I went ahead and exported the sequence into a QT Movie. Brought it into compressor and submitted the job without using the Virtual Cluster. It took Compressor 1 hour 33 minutes 44 seconds to complete the task.
The next test was to see how long it would take to process the same Job, but this time using our Virtual Cluster. Soon after submitting the job to the Virtual Cluster, the MacPro split each target into 10 different segments, and started processing 5 segments at the same time. Processor power on all 8 cores averaged between 95 and 100%, but the machine wasn't getting overloaded. Compressor finished the job in 49 minutes 3 seconds.
For those interested, I've included a little table with the results of all 4 Tests :
|Source & Target||This Computer||Virtual Cluster||Difference|
|HDV1080i50 clip (3 min 41 sec) to Apple TV and iPod Preset||14 minutes 40 seconds||9 minutes 50 seconds||approx. 5 minutes|
|HDV1080i50 clip (18 min 26 sec) to Apple TV and iPod Preset||1 hour 33 minutes 44 seconds||49 minutes 3 seconds||44 minutes 41 seconds|
|Apple ProRes 422 720p25 (20 min 59 sec) to Apple TV and iPod Preset||32 minutes 59 seconds||16 minutes 52 seconds||16 minutes 7 seconds|
When OSX 10.5 came out, I had serious issues setting up my Virtual Cluster and actually using it. From time to time, it would just stop working, or my Virtual Cluster wouldn't show up anymore. I tried a lot of things, including reinstalling Final Cut Studio, even reinstalling my MacPro completely, but still my Virtual Cluster would only work 'occasionally'. Finally I found a post on the web suggesting to turn of 'Back to my Mac' in the System Preferences. This problem might be fixed by now, but I actually never turned Back to my Mac on again. So if you are having issues where your Virtual Cluster doesn't show up in Compressor once you started sharing it, you might want to try by turning off 'Back to my Mac'.