Apple: Mac OS X Server Caching

These days, new services get introduced in OS X Server during point releases. OS X now has a Software Caching server built to make updates faster. This doesn’t replace Apple’s Software Update Server mind you, it supplements. And, it’s very cool technology. “What makes it so cool” you might ask, given that Software Update Server has been around for awhile. Namely, the way that clients perform software update service location and distribution with absolutely no need (or ability) for centralized administration.

Let’s say that you have 200 users with Mac Minis and an update is released. That’s 200 of the same update those devices are going to download over your Internet connection, at up to 2 to 3 gigs per download. If you’re lucky enough to have eaten at the Varsity in Atlanta, just imagine trying to drink one of those dreamy orange goodnesses through a coffee stirrer. Probably gonna’ be a little frustrating. Suck and suck and suck and it’ll probably melt enough to make it through that straw before you can pull it through. For that matter, according to how fast your Internet pipe is, there’s a chance something smaller, like an update to Expensify will blow out that same network, leaving no room for important things, like updates to Angry Birds!

Now, let’s say you have an OS X Server running the new Caching service. In this case, the first device pulls the update down and each subsequent device uses the WAN address to determine where the nearest caching service is. If there’s one on the same subnet, provided the subnet isn’t a Class B or higher, then the client will attempt to establish a connection to the caching service. If it can and the update being requested is on that server then the client will pull the update from the server once the signature of the update is verified with Apple (after all, we wouldn’t want some funky cert getting in the way of our sucking). If the download is stopped it will resume after following the same process on a different server, or directly from Apple. The client-side configuration is automatic so provides a seamless experience to end users.

Pretty cool, eh? But you’re probably thinking this new awesomeness is hard as all heck to install. Well, notsomuch. There are a few options that can be configured, but the server is smart enough to do most of the work for you. Before you get started, you should:

• Be running Mountain Lion with Server 2.2 or better.
• Install an APNS certificate first, described in a previous article I wrote here.
• Have an ethernet connection on the server.
• Have a hard drive with at least 50GB free in the server.
• The server must be in a Class C or smaller LAN IP scheme (no WAN IPs can be used with this service, although I was able to multihome with the WAN off while configuring the service)

Once all of the requirements have been met, you will need to install the actual Caching Service. To do so, open Server.app from the /Applications directory and connect to the server with which you would like to install the Caching service.

Click on Caching from the SERVICES section of the Server sidebar. Here, you have 3 options you can configure before starting the service. The first is which volume with which to place updates. This should typically be a Pegasus or other form of mass storage that is not your boot volume. Use the Edit… button to configure which volume will be used. By default, when you select that volume you’ll be storing the updates in the Library/Server/Caching/Data of that volume.

The next button is used to clear out the cache currently used on the server. Click Reset and the entire contents of the aforementioned Data directory will be cleared.

Next, configure the Cache Size. Here, you have a slider to configure about as much space as you’d like, up to “Unlimited”. You can also use the command line to do some otherwise unavailable numbers, such as 2TB.

Once you’ve configured the correct amount of space, click on the ON button to fire up the service. Once started, grab a client from the local environment and download an update. Then do another. Time both. Check the Data folder, see that there’s stuff in there and enjoy yourself for such a job well done.

Now, let’s look at the command line management available for this service. Using the serveradmin command you can summon the settings for the caching service, as follows:

sudo serveradmin settings caching

The settings available include the following results:

caching:ReservedVolumeSpace = 25000000000
caching:SingleMachineMode = no
caching:Port = 0
caching:SavedCacheSize = 0
caching:CacheLimit = 0
caching:DataPath = "/Volumes/Base_Image/Library/Server/Caching/Data"
caching:ServerGUID = "FB78960D-F708-43C4-A1F1-3E068368655D"
caching:ServerRoot = "/Library/Server"

Don’t change the caching:ServerRoot setting on the server. This is derived from the root of the global ServerRoot. Also, the ServerGUID setting is configured automatically when connecting to Apple and so should not be set manually. When you configured that Volume setting, you set the caching:DataPath option. You can make this some place completely off, like:

sudo serveradmin settings caching:DataPath = "/Library/Server/NewCaching/NewData"

Now let’s say you wanted to set the maximum size of the cache to 800 gigs:

sudo serveradmin settings caching:CacheLimit = 812851086070
To customize the port used:

sudo serveradmin settings caching:Port = 6900

The server reserves a certain amount of filesystem space for the caching service. This is the only service I’ve seen do this. By default, it’s about 25 gigs of space. To customize that to let’s say, ‘around’ 50 gigs:

sudo serveradmin settings caching:ReservedVolumeSpace = 50000000000
To stop the service once you’ve changed some settings:

sudo serveradmin stop caching

To start it back up:

sudo serveradmin start caching

Once you’ve started the Caching service in OS X Server and familiarized yourself with the serveradmin caching options, let’s look at the status options. I always use fullstatus:

sudo serveradmin fullstatus caching

Returns the following:

caching:Active = yes
caching:state = "RUNNING"
caching:Port = 57466
caching:CacheUsed = 24083596
caching:TotalBytesRequested = 24083596
caching:CacheLimit = 0
caching:RegistrationStatus = 1
caching:CacheFree = 360581072384
caching:StartupStatus = "OK"
caching:CacheStatus = "OK"
caching:TotalBytesReturned = 24083596
caching:CacheDetails:.pkg = 24083596

The important things here:

• An Active setting of “yes” means the server’s started.
• The state is “STARTED” or “STOPPED” (or STARTING if it’s in the middle).
• The TCP/IP port used 57466 by default. If the caching:Port setting earlier is set to 0 this is the port used by default.
• The CacheUsed is how much space of the total CacheLimit has been used.
• The RegistrationStatus indicates whether the server is registered via APNS for the service with Apple.
• The CacheFree setting indicates how much space on the drive can be used for updates.
• The caching:TotalBytesRequested option should indicate how much data has been requested from clients while the caching:TotalBytesReturned indicates how much data has been returned to clients.

Look into the /Library/Server/Caching/Config/Config.plist file to see even more information, such as the following:

<key>LastConfigURL</key>
<string>http://suconfig.apple.com/resource/registration/v1/config.plist</string>
<key>LastPort</key>
<integer>57466</integer>
<key>LastRegOrFlush</key>
<date>2012-12-16T04:33:13Z</date>

There are also a number of other keys that can be added to the Config.plist file including CacheLimit, DataPath, Interface, ListenRanges, LogLevel, MaxConcurrentClients, Port and ReservedVolumeSpace. These are described further at http://support.apple.com/kb/HT5590.

As you can see, this provides the host name of the server and path on that server that the Caching server requires access to, the last port connected to and the last date that the contents were flushed.

In the Data directory that we mentioned earlier is a SQLite database, called AssetInfo.db. In this database, a number of files are mentioned. These are in a file hierarchy also in that Data directory. Client systems access data directly from that folder.

Finally, the Server app contains a log that is accessed using the Logs option in the Server app sidebar. If you have problems with the service, information can be accessed here (use the Caching Service Log to access Caching logs).

The Caching Service uses the AssetCache service, located at

/Applications/Server.app/Contents/ServerRoot/usr/libexec/AssetCache/AssetCache

then starts as the new user _assetcache user. It’s LaunchDaemon is at

/Applications/Server.app/Contents/ServerRoot/System/Library/LaunchDaemons/com.apple.AssetCache.plist

When a supported client device (OS X 10.8.2+, iOS 7+) checks for updates, it first checks with Apple as it would without an available Caching server. This makes sense as Caching does not replicate a full catalog of updates from Apple. When an update is requested, the device requests it directly from the Caching server, not knowing if the Caching server has a local copy or not. If Caching doesn't have a local copy it starts to download the update from Apple, and almost immediately starts sending what it has downloaded thus far back to the device. Otherwise, the local copy is returned to the device.

I first guessed that devices would request updates from Apple and be redirected either with standard HTTP redirection or a properiority payload. But given that the device goes directly to the Caching server for content, I believe that when the device checks for updates with Apple, Apple matches the public IP and supported local address range in the Caching server registration with the device's addresses, and provides URLs for updates that point directly to the Caching server.

A real example of a URL used when requesting an update from the Caching server looks this:
http://192.168.78.101:50856/us/r1000/010/Purple/v4/1f/9c/e1/1f9ce1e6-67e3-3bdd-0842-6ff3a0f515cd/mzps4900982890482523744.pkg?source=a1775.phobos.apple.com

Let's break this down:
http - Even although Server.app generates a self-signed certificate, Caching uses HTTP not HTTPS for serving content
192.168.78.101 - The local IP address of the Caching server
50856 - The TCP port that Caching server is using for HTTP
/us/r1000/010/Purple/v4/1f/9c/e1/1f9ce1e6-67e3-3bdd-0842-6ff3a0f515cd/mzps4900982890482523744.pkg - The URI reference to the content on the Caching server. I'm guessing it mirrors URIs on Apple's CDN servers. (This URI will be important in the next section.)
source=a1775.phobos.apple.com - Lastly, Apple appends a query string to the URL to let Caching know which Apple server to pull the content from if it needs to.


References:
1. The New Caching Service In OS X Server
http://krypted.com/mac-security/the-new-caching-service-in-os-x-server/

2.  Caching Server 2
http://fraserhess.blogspot.com/2013/11/caching-server-2.html

3.  A power user’s guide to OS X Server, Mavericks edition
http://arstechnica.com/apple/2013/12/a-power-users-guide-to-os-x-server-mavericks-edition/6/