A Fractional Key System for Memcache.
Fragmented or as I prefer to call them fractional keys provide a relatively trouble free way of managing the inter-dependencies of cached data.
At its heart fractional keys are simply tag based keys. Where they differ from the standard implementation of tag based keys is that each tag has an associated version number. This allows us to invalidate all keys dependent on a specific tag automatically by incrementing its associated version.
For example say cached data A, and B are, among other things, pegged to a specific user entry. We embed a user entry tag in the key for A and B so that when the underlying User entry is updated our keys for A and B are updated as well.
Key of A = keya_UserId:1234@version5_OtherField:FieldId@version1
Key of B = keyb_UserId:1234@version5_OtherField:FieldId@version1_OtherField3:FieldId@version1
When we modify UserId:1234 we increment its key version.
$kg = new KeyGroup("UserId", 1234); $kg->incrementVersion();
As a result the keys generated for cached item A and B are modified. On the next request for A or B no cached data will be found and the data will be regenerated according to the rules applicable to the cached item A or B.
Key of A = keya_UserId:1234@version6_OtherField:FieldId@version1
Key of B = keyb_UserId:1234@version6_OtherField:FieldId@version1_OtherField3:FieldId@version1
The code below is a rewrite of the fractional key systems I’ve used previously with Kaplan, Great Non Profits, Seekopolis and Cartoon Doll Emporium. I generally tie my keygroups to a database persistence layer provided I can guarantee high memcache availability (you don’t want a downed memcache server to exasperate database performance by also resulting in heavy tag-version requests).
I also tend to implement what I call a KeyRings that works as a facade pattern for managing various keys and their associated groups. A KeyRing, for instance, may automatically append a few keygroups on every key it generates, or provide a simple name for each key:
$usernameKey = $UserKeyRing->generateUsernameKeyRing($userId);
Even with out a KeyRing class the keyrings are fairly easy to work with:
getGroupTag() . "\n\n";
$key->incrementVersion();
echo "And after an increment the GroupTag is . . . " . $key->getGroupTag() . "\n\n";
$key2 = new StandardKeyGroup("User", 101);
$key3 = new StandardKeyGroup("Magic", 102);
$theKey = new StandardKey("ThisIsAKey", "" , array($key,$key2,$key3));
echo "Finally our composite key is " . $theKey->getKey() . "\n";
$memcache = MemcacheSingleton::getInstance();
$memcache->set($theKey->getKey() , 123456);
$resp = $memcache->get($theKey->getKey());
echo " I just retrieve $resp\n";
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 | connect('localhost', 11211); } return self::$memcache; } } /** * The Key Group is the basic tag + version_info node which our fractional * key system relies on. */ interface FractionalKeyGroup { public function __construct($group, $index, $version); public function getVersion(); public function setVersion($version,$update); public function getGroupTag(); public function getGroupName(); public function IncrementVersion(); public function ResetVersion(); public function DelegateMemcacheQuery(); } /** * The various key groups are merged together to form our final cache key. */ interface FractionalKey { public function __construct($key, $groupId, array $keyGroups); public function AddKeyGroup(FractionalKeyGroup $keyGroup); public function getKey(); } /** * This is the Standard Implementation of a KeyGroup. * Additional Implementations are: * - Database or File Backed KeyGroups - Commonly you will want to persist versioning information on a layer that * is less volatile than memcache. Memcache will perform garbage collection and could remove a keygroup that underpins a large set of data. * If this keygroup is not persisted to a database or file persistence layer this garbage collection could result in a unnecessarily large swatch of * cache in-validations. * - The Constant KeyGroup (A keygroup where the version number is static) * - The Time Delayed KeyGroup (a keygroup that uses a time delay when incrementing keygroup version. For example you may only want data up-to-date within * 10-30 minutes for some uses and within 1-2 minutes for other uses. A time delayed key group tracks multiple versions itself set to expire at different time intervals.). * allowing the end user to specify how stale of data they are willing to work with. */ class StandardKeyGroup implements FractionalKeyGroup { protected $groupName; protected $groupIndex; protected $version; protected $memcache; static protected $VERSION_SEPERATOR = ":v"; static protected $INDEX_SEPERATOR = "_"; /** * This Control function is used to determine if the fractionalkey that contains this keygroup may multiget memcache for its version value. * For items like Static Keys we set this value to false to reduce strain on the memcache server albeit at increased php processing time to check this value. */ public function DelegateMemcacheQuery() { return true; } /** * * @param string $group - name of group ("user", "day_of_week", "username", etc) * @param string $index - index for group (unique id of group record in question) * @param int $version - group version can be manually set if desired. * @param bool $update - group version will not actually be updated in memcache unless specified. */ public function __construct($group, $index = "na", $version = null, $update = false) { $this->groupName = $group; $this->groupIndex = $index; if(!empty($version)) { $this->version = $version; if($update){ $this->_StoreVersion(); } } $this->memcache = MemcacheSingleton::getInstance(); } /** * returns the current version value for this group. * @return int */ public function getVersion() { return isset($this->version) ? $this->version : $this->_getVersion(); } /** * Modify the version used for this group. * @param int $version * @param bool $update - Update Memcache/Persistant store. */ public function setVersion($version, $update) { $this->version = $version; if($update) { $this->_StoreVersion(); } } /** * Retrieve this groups tag (Unique GroupName Plus Versioning Info ); * @return */ public function getGroupTag() { return $this->getGroupName() . self::$VERSION_SEPERATOR . $this->getVersion(); } /** * Retrive this groups name. (For example USER_123423); * @return */ public function getGroupName() { return $this->groupName . self::$INDEX_SEPERATOR . $this->groupIndex; } /** * Increment version number. */ public function IncrementVersion() { if($this->version == null) { $this->ResetVersion(); } else { $this->version += 1; $this->_StoreVersion(); } } /** * Reset version number. We use a microtime stamp to insure this value is * always unique and will not result in a pull of invalidated data. */ public function ResetVersion() { $this->version = intval(microtime(true) * 1000); $this->_StoreVersion(); } protected function _getVersion() { if($this->version == null) { $result = $this->memcache->get($this->getGroupName()); if(empty($result)) { $this->ResetVersion(); } return $result; } else { return $this->version; } } protected function _StoreVersion() { $this->memcache->set($this->getGroupName(),$this->version); } } /** * This is a standard implementation of a fractional key, One obvious extensions would deal with the nesting of keys within keys. */ class StandardKey { protected $key; protected $groupId; protected $memcache; protected $keyGroups = array(); static protected $TAG_SEPERATOR = ":t"; static protected $INDEX_SEPERATOR = "_"; public function __construct($key, $groupId, array $keyGroups = array()) { $this->key = $key; $this->groupId = $groupId; foreach($keyGroups as $keyGroup) { $this->keyGroups[$keyGroup->getGroupName()] = $keyGroup; } $this->memcache = MemcacheSingleton::getInstance(); } public function AddKeyGroup(FractionalKeyGroup $keyGroup) { $this->keyGroups[$keyGroup->getGroupTag()] = $keyGroup; } public function getKey() { $key = $this->key . self::$INDEX_SEPERATOR . $this->groupId . self::$TAG_SEPERATOR . implode(self::$TAG_SEPERATOR, $this->gatherTags()); return $key; } /** * While it would be architecturally cleaner to gather group versions from a KeyGroup function call * the use of memcache multiget helps us avoid some bottle-necking produced by the increased number of key-versions we * need to look-up when using this fractional key system. */ protected function GatherGroupVersions() { $group_tags = array_keys($this->keyGroups); foreach ($group_tags as $group_tag) { if($this->keyGroups[$group_tag]->DelegateMemcacheQuery() == false) { unset($group_tag); } } $tags = $this->memcache->get($group_tags); foreach($this->keyGroups as $key => &$group) { if(array_key_exists($key, $tags)) { $group->setVersion($tags[$key],false); } else { $group->ResetVersion(); } } } /** * * @return array tag strings of all tags included within this key. */ protected function GatherTags() { $this->GatherGroupVersions(); $tags = array(); foreach($this->keyGroups as $group) { $tags[] = $group->getGroupTag(); } return $tags; } } |
Enjoy ^_^.

Categories
Tag Cloud
Blog RSS
Comments RSS
Last 50 Posts
Back
Void « Default
Life
Earth
Wind
Water
Fire
Light 
great post as usual!
This is pretty much an encapsulation of my original memcached FAQ entry about simulating namespaces. http://code.google.com/p/memcached/wiki/FAQ#Simulating_Namespaces_with_key_prefixes
Not exactly,
Tieing memcache to strings like “UserId_BrianMoon” or “User_1234″ to insure uniqueness of tag names is a fairly common practice. The above code does do this but the primary focus is on the the addition of a versioning token for hierachal cache invalidation.
We are not just name spacing our keys we are versioning the components that build up our keys. For example we are using keys like UserId_BrainMoon_July7th instead of UserId_BrainMoon and additionally providing a mechanism for incrementing that version token so that we can easily invalidate our various composite keys that are tied to that specific namespace+version key. Thus if we need to invalidate any data tied to UserId_BrainMoon we can simply increment our tag to UserId_BrainMoon_July8th and in turn invalidate a large swath of dependent data with out knowing who the dependents of that namespace are in advance.
This is handy when you have expensive to compute data that needs to be updated if and only if a member of some infrequently changing lists of inputs has been updated. For example a pdf list of faculty at a university and a stored array of members could be tied to FacultyLastUpdated. As new faculty onboard they are added through some front end tool that updates this tag which in turns invalidates any cached entries specifically tied to this attribute.