That Time We Burned Down Players’ Houses in Ultima Online
Ultima Online is celebrating its 25th anniversary! You know what that means, right? Story time!
In the spirit of #gamedev and inspired by the classic UO postmortem video by Raph Koster, et al, I thought it’d be fun to share a few stories from the middle years of Ultima Online between my time at EA Redwood Shores and Mythic Fairfax.
Today’s tale: a technical teardown of burning down cheaters’ houses after finding a “criminal” ring of illegal item dupers.
Duping: UO’s Zero-Day Exploit
When Ultima Online first launched we, the players, found some bugs.
By some, I mean, a lot.
And the most valuable, diabolical, worth-building-a-secret-web-forum-to-trade-them sort of bug was: duping.
Being able to duplicate an item, or better, a bag full of items, was the most advantageous exploit possible in the game. Better than speed-hacking, better than direct damage passthrough — even better than pre-lockdown home invasions.
Some of the beta testers had a secret. Not all of us, of course, but there were certain seedy elements of the beta testing crowd that, rather than follow the virtue of Honesty, chose to hoard their knowledge of exploits rather than report them to the development team.
Once UO’s beta ended and the game launched, well, suddenly the economy was hit by a wave of item duping.
What was this “carefully guarded” exploit?
Abusing areaserv boundaries.
Rubberbanding For Fun and Profit
I still remember the first time I saw it for myself.
It was 1997 and I was running around north of the orc encampment near the town of Cove when I saw some randos doing the weirdest thing ever: they were running back and forth over this annoyingly laggy patch of the game map while dropping little chests on the ground.
Now, I knew this particular area as being… the worst.
You see, there was this truly annoying part of the map on that dirt path between the mountains and the orc fortress that was pretty unresponsive and jerky when you walked over it.
In fact, if you *ran* across it you might get “rubberbanded” — finding yourself losing a few seconds of gameplay time as you bounced back to a previous position before the server “caught up.”
(Remind me to tell you all about UO’s server/client interpolation and tick-rate someday!)
This was especially bad if a monster, like an inconveniently placed Orc Captain directly south of you, was chasing you and whacked you to death during your pathetically laggy attempt to flee.
Imagine my surprise when these two players started excitedly yelling to each other in text (which floated over their heads) that “it worked! omgz!”
Yup, they had managed to, as they excitedly bragged, figure out a trick to drop a chest on one side of the “laggy patch” while trying to pick-it-up/hand-it-over to the other player as they were both crossing from one side to the other and now each of them had a copy of the same chest: and its contents.
Dupers!
Seamless Server Boundaries are Hard
What I had seen was an exploit based on the “areaserv” boundaries that split up the playable gameworld of Ultima Online.
UO never did anything the easy way; after all, its first generation of designers were trying things — inventing entirely new things — that later online games would simply shake their heads at and say “nah, that’s too hard.”
For instance: how to load-balance thousands of players in a huge 29,360,128 meter² game map in 1997?
Instead of having different “zones” with loading screens or “long, foggy mountain passes”, the UO developers just invented rectangular, seamless “sub-maps” with a (fairly!) invisible method that just let you walk from one side to the other, all while things on the other side were still visible and updating in real-time.
Thus, each game server (or “shard”) you played on was actually divided into “areaservs” and there was some incredibly well-written “mirror” code that handled communicating gamestate, object conditions, and events from one side of the border to the other.
But crossing those borders as a player, sometimes, was noticeably laggy.
Aside: If you look at the areaserv map above you may notice that Britannia should’ve been divided up a bit more carefully so that areaserv boundaries didn’t cut through important/populated gameplay areas… look at poor Buccaneer’s Den: there’s a reason those hidden, underground tunnels were never fun to PvP in.
The areaservs were essentially copying your player character and sending over a bundled message between themselves containing all your info. Once a player crossed the boundary it destroyed the old copy of the player on the initial side.
And it wasn’t just players: autonomous mobile objects (mobs) like monsters, animals, and NPCs could cross them as well.
(Note: for future exploits those monsters and animals also had their own backpacks/inventories players could piggyback exploit attempts on; lots of llama drama.)
Naturally, any given bug with the areaserv code represented the greatest probable source of “duping” exploits, especially when coupled with purposefully manipulations of a player’s gamestate before the server backup/shutdown sequence each morning.
We Felt Like We Were Playing Whack-a-Mole
And after I joined the UO team I learned a few things:
- The areaserv code is brilliant. It’s been patched and rewritten many times, but by my later years it had gotten pretty “smart” about the way it anticipated player state and pre-serialized objects before boundary transfers even happened.
- Yet, no matter how good the code became, we developers had introduced so much complexity into the game that sources of dupes abounded regardless — and more often than not they weren’t just areaserv issues anymore. It didn’t help that the game designers could actually write production-level code in Wombat (our own event-based scripting language) that had tons of wrappers for the C++ object management code.
It really felt like we were constantly on the backfoot; we couldn’t be very proactive about dupes and depended on player reports & customer service to identify egregious exploits.
Then, one day I had this weird idea.
UO Didn’t Have a Database
There are a couple of things you have to know about UO as it was in the mid-2000’s.
- Each shard ran a shutdown/backup sequence at a certain time in the early morning hours. (Great for not having to fix memory leaks!)
- The complete gamestate of each areaserv was dumped from memory into a big binary backup file — by the time I had joined the team they were approaching about 4gb in size.
- Once the areaservs finished the backups they shut down, restart, and went into standby while the “gameserv” did the same thing. You can think of the gameserv as a server dedicated to coordinating all of the areaservs and passing along logged in players to the right areaserv.
- The gameserv would shut down, restart, and then instruct each areaserv to load its last known-good backup.
- Each areaserv would load the binary backup and recreate the previously saved state of the game; it would also execute any triggers/hooks in the scripting code for objects/mobiles that said “Do X when the server loads.”
- [A bunch of other stuff here, including spawning new mobs or daily rares]
- The areaservs would report all was well and inform the gameserv, and the gameserv would re-announce itself to the login servers that it was available for play.
#2 was the biggest problem with trying to find dupes: UO’s gamestate wasn’t in a “database” — there was no way to query for player possessions or objects to find illicit goods. It was 4gbs of binary blob; the data only made sense when loaded back into the game itself.
I tried though: I *wanted* to read the binary blobs. I *wanted* to give Customer Service tools to find duped or stolen items. It just wasn’t possible with the tools of the time.
Then I remembered #5, and it gave me an idea for a different line of attack.
A Global Hash Registry
Every dynamic object in UO— whether a mobile, player, or item — is capable of storing both data and scripts on themselves.
Scripts don’t hold state once code execution ends, so we would store needed data in “objvars” and attach them to the same objects the scripts were attached to.
The scripts themselves contained tons of “triggers” which were event handlers tied to game conditions. As I mentioned in #5 above, one of them had a name that went something like “beforeServerLoad” (heck if I remember the actual name) that specifically executed during the backup loading stage of the areaserv startup process.
I thought: “What if we just *marked* the most valuable items in the game whenever they loaded from backup?”
I went to the lead engineer, Supreem, and asked for just one addition to the C++ code and an associated mapping to a “Wombat func” for it: a hashing function (hello crypto!).
I wanted to make the gameserv store a running list of “marked” items during the areaserv loading process to identify dupes.
A global hash registry.
A Plan Unfolds: Invisible Dye
Here’s how it worked:
- Every object in the game, when it loaded, had a hook for a generic “preload” script that would attach itself, execute, and detach itself before it allowed the object to execute any other scripts. This was really useful for cleaning up deployment errors if we did something wrong and it broke any objects in the game during a patch.
- I added a trigger in the preload script with an “beforeServerLoad” event that checked the item’s type: if it was in a predetermined list of the most expensive/rare types of items, it continued with the plan.
- If the item didn’t have a “tag” on it, it was created now: an objvar of “string” type that contained a hash of the current time, areaserv, and gameserv id (and some other unique data). This was a really long, random looking piece of text (larger than a 32bit integer) like “29bb546a415ff874e5129549fe8064249e8f1b2996fa2e7d52879d2ec24e06fd”.
- Once it had a tag (or if it already did), it sent a message to the gameserv (remember, there was only one of these per shard) and asked the gameserv to put it in a lookup registry. If the *tag* already existed in the registry then the item was permanently marked as “I AM DUPED” (again, with an objvar) in plaintext — because Customer Service reps in the god client could see it.
- The lookup registry only persisted until the areaservs all loaded up, then it cleared out to save memory.
The idea was that if a valuable item (worthy of duping) had a unique hash stored on itself as an objvar, then the duping process — whatever it may be — would copy that same objvar’s value. Then there would be two unique objects both holding the same hash.
The best time to check for this (in bulk) was during server loads, and that system marked all the objects with an “I AM DUPED” objvar.
(Note: my memory may be fuzzy here, and we may have had to tie the script to the onPlayerLoad event as well; I just don’t remember if players were all loaded with the areaserv or whether they were fetched from the backup data on-demand).
It was a lot like stamping the rare items with some sort of invisible ink that only fluoresced under certain conditions.
So we released the updated code and let it run globally for a few weeks.
And then the hunt began.
Mwaha. Mwahaha. MWAHAHAHAHAHA!
The Corporate Reaction
Within a couple of weeks we realized the first phase had worked: we had found a ton of duped rare items, and they were mostly held by the same set of player accounts across multiple game servers.
Despite our excitement, this is where we paused: what do we do now?
We sat down to talk with Management, Customer Service, and our Community Manager.
Management
Our producer was stoked to explain what we’d accomplished to the studio general manager. I remember sitting in the meeting with a smile on my face, pleased as punch (as only an arrogant young gamedev can be), just to hear something I wasn’t expecting:
“Mmm, I don’t think deleting them all [the duped items] is a good idea, you’ll hurt too many players.”
I hadn’t considered that, actually.
At all.
I was too excited about having accomplished my long held goal to “catch some dupers.”
I wanted to chafe and say something, maybe assert that he just didn’t understand, but reined myself in and after the meeting I went back to the team and told them that we were forbidden from auto-deleting all the dupes.
Customer Service
So we went to the CS team for data.
It was true. It would’ve been a terrible idea.
The dupes spread so quickly once they were created that if we just deleted them all out from under all the players who had bought them (with their hard-earned gold) from the dupers we would be affecting a significant portion of the playerbase.
Sure, some of them would be fine with the “morality” of our action — but on the scale of hundreds or thousands of affected players (per shard) we were just asking for the frustration to cause a wave of quiet quitting.
I really shouldn’t have been surprised by this: our studio’s general manager has been running MMOs for a lot longer than I ever had.
To make matters more complicated the Customer Service team started asking us hard questions, like: “Exactly how many duped runic valorite hammers should a person have before we ban them?”
Wait, we were supposed to assign an arbitrary value on how many dupes being held by one person was suspicious? Oh. Oh boy.
This was harder than code. It was community design, not just game design.
Community Manager
Thankfully, we had a level-headed community manager who was used to our over-exuberance. Honestly, they were just used to managing me whenever I got too passionate about something.
“CS is going to ban them [the actual dupers] anyways. Why not make it an event?” was about how that conversation went.
Well, I was the lead live events designer, so… we plotted.
Fire! Heh Heh, Fire!
We identified the dupers themselves and their storage depos: they had homes full of their duped items and NPC vendors selling them to the players.
The “duping ring” stretched across multiple servers, comprised of distinct groups not necessarily working together. They had all evolved the same behaviors though: making tons of UO gold from selling dupes and then selling the UO gold on secondary markets for hard cash.
So Adida and I wrote a script, that, once attached to a house, would:
- Delete the house and all of its contents. All of it. Instantly. Recursively. *poof*
- Spawn a bunch of immovable “housing rubble” in a predetermined rectangular area that fit the same dimensions the house existed in. It was colored dark black to look like it had soot all over it.
- Spawn a bunch of eternal “fire fields” amongst the rubble.
- Create a straw dummy labeled “An Effigy of a Traitor” to place in the middle of the burning rubble.
Then, we chose a day, and struck.
Customer Service mass banned the dupers on a timer, right before their associated server came online.
Then we paused the gameserv/loginserv connection to those same servers and Adida and I ran around (teleported, really) to each housing location, attached the script, watched the fires erupt with joy, and moved on.
We did this in batches of related servers so that each “duping ring” didn’t have time to notice they were banned and try to log in on alt-accounts to empty their houses before we could get to them
The Community Reacts
Confusion! Chaos! Joy! Laughter!
Discussion threads popped up on Stratics and MMORPG.com almost instantly, full of wonderings and guesses — and a few false suggestions like that a CS employee had been caught up in the banning (false!).
Well That’s Neat, But Don’t Do It Again
Dozens of homes had been destroyed across the entire multiverse of Ultima Online, and the flames licking the sooty rubble were a visible testimony to our team’s determination to deal with cheaters.
It felt fantastic!
And we were told not to do it again.
Lol.
We took some flak for doing something so daringly public to cheating players — even though we made sure not to identify them to the other players directly — but still just barely skated by with upper-upper-upper management.
Customer Service was instructed to use their discretion to deal with “I AM DUPED” items from then on, and honestly, by now I don’t know if anyone even remembers that the system exists (if it even does).
Some Lessons Learned (Satoshi Surprise?)
Readers who’ve been exposed to NFTs might notice the eerie resemblance of us marking these rare items (which were non-fungible!) with unique hashes to how blockchain-based Non-Fungible Tokens (NFTs) work today.
Ironically, one thing that our method didn’t work on was *stacks* of objects (fungible items like gold coins).
But Satoshi Nakamoto solved that problem too! The first Bitcoin paper describes a perfect method (online or offline) for validating timestamped events like currency creation. It would’ve helped prevent dupes *and* ensured internal bad actors (with item creation powers) couldn’t just generate a bunch of gold to sell outside the game.
Not that I’m saying MMOs need blockchains; but it turns out Proof-of-Work ledger technology had a use case here! (No wonder Amazon now offers a similar service: QLDB).
Finally, one of the biggest things we learned was having to deal with the economics *after* the fire: especially when players wanted to compete for the now available, very premium, housing spots.
But that’s another story.
About the Author
Tim “Draconi” Cotten was the lead game designer of Ultima Online: Stygian Abyss, and when he’s not writing up stories from the wild days of online gaming is working on applying all those hard-won lessons to Metaverse design.
Follow him on Twitter for more Ultima related stories!
