what
StructDiff - a zero-dependency, field-level delta compression crate which generates serializable diffs
where
github / crates
why
I’ve been poking at building a multiplayer game for a couple of years now, and one of the big issues that I immediately ran in to was the necessary amount of data transfers. For an MMO-style game, there’s an incredible amount of data that needs to be tracked by the server and also available for the clients in case it’s needed.
things to track for an MMO
- State of Self
- Health
- Location
- Speed
- Abilities
- Inventory
- Chat Messages
- Reputations
- State of Other (times number of nearby characters)
- Proxy value for most of the Self stats
- State of World
- Relevent events
- Prevailing conditions
This is clearly a ton of data, to the point that it will probably be impractical to serialize the entire state and package it each time if you have enough clients connected. This leaves us with two options for communicating changes in state (that I’m aware of):
Obviously a purely event-driven architecture with deterministic resultion of effects would be the ideal solution here. However, this requires that either a full mirror of the server is run by each client (and each client has all of the necessary information to run this server), or that a completely different set of events be used internally on the server and externally to update client state. Depending on how deterministic your physics engine is, this may or may not be possible. It certainly is a good solution, the 2nd type of system was more appealing to me: one based on differences. (As a sidenote, this has been a huge rabbit trail, I do not recommend writing libraries if you ever hope to actually release a game)
Delta compression at the binary level is an extremely common tactic to reduce the amount of data transfer over the network, but it doesn’t really help with the problem of reducing (de)serialization load, and also discards a large amount of the semantic data that is readily available prior to serialization. The pre-serialization delta-compression approach takes advantage of the fact that it is possible to know which fields have changed and only send those particular updates to a client. There are a couple of excellent struct-diffing libraries that existed already (serde-diff, DiffStruct), but there was still a niche there (in my opinion, at least) for a library that operated pre-serialization, did not pull syn/quote/proc_macro2 into the build tree, and generated serializable diffs. If any of those requirements don’t concern you, one of those libraries should probably be used instead.
how
My first step was to come up with a trait definition I liked. I eventually landed on the following (after an immediate flop of an attempt to RFC anonymous associated types).
pub trait StructDiff {
type Diff: Clone;
type DiffRef<'target>: Clone + Into<Self::Diff>
where
Self: 'target;
/// Generate an owned diff between two instances of a struct.
fn diff(&self, updated: &Self) -> Vec<Self::Diff>;
/// Generate an unowned diff between two instances of a struct.
fn diff_ref<'target>(&'target self, updated: &'target Self) -> Vec<Self::DiffRef<'target>>;
/// Apply a single-field diff to a mutable self ref
fn apply_single(&mut self, diff: Self::Diff);
/// Apply a full diff to a mutable self ref
fn apply_mut(&mut self, diffs: Vec<Self::Diff>) {
for diff in diffs {
self.apply_single(diff);
}
}
/// ... skipping the by-value and by-ref where Self: Clone impls
}
which allows generating a delta to be as simple as:
use structdiff::{Difference, StructDiff};
#[derive(Debug, PartialEq, Clone, Difference)]
struct Example {
field1: f64,
field2: String,
}
let first = Example {
field1: 0.0,
field2: "Hello".to_string()
};
let second = Example {
field1: 3.14,
field2: "Hello".to_string()
};
let diffs: Vec<<Example as StructDiff>::Diff> = first.diff(&second);
assert_eq!(diffs.len(), 1);
let diffed = first.apply(diffs);
assert_eq!(diffed, second);
Notice how the user of the trait never has to interact with the actual type used for the StructDiff::Diff (or StructDiff::DiffRef<'_>) associated type. Using the Diff type .clone()s its way through the non-matching fields, while the DiffRef<'_> type attempts to take everything by reference. There’s obviously a ton of boileplate code required with this approach, and I really did not want to have to manually implement this trait for every single struct in my game. Really wish rust had compile-time reflection, but a proc-macro gets it done for now.
big annoying trait impl
Deriving diffence on this struct:
#[derive(Difference)]
#[difference(setters)] // this attribute is explained later in the post
pub struct Test {
pub test1: i32,
pub test2: String,
pub test3: Vec<i32>,
pub test4: f32,
pub test5: Option<usize>,
}
generates the following code
const _: () = {
use structdiff::collections::*;
/// Generated type from StructDiff
#[derive(Clone)]
pub enum __TestStructDiffEnum {
test1(i32),
test2(String),
test3(Vec<i32>),
test4(f32),
test5(Option<usize>),
}
/// Generated type from StructDiff
#[derive(Clone)]
pub enum __TestStructDiffEnumRef<'__diff_target>
where
Self: '__diff_target,
{
test1(&'__diff_target i32),
test2(&'__diff_target String),
test3(&'__diff_target Vec<i32>),
test4(&'__diff_target f32),
test5(&'__diff_target Option<usize>),
}
impl<'__diff_target> Into<__TestStructDiffEnum> for __TestStructDiffEnumRef<'__diff_target> {
fn into(self) -> __TestStructDiffEnum {
match self {
__TestStructDiffEnumRef::test1(v) => __TestStructDiffEnum::test1(v.clone()),
__TestStructDiffEnumRef::test2(v) => __TestStructDiffEnum::test2(v.clone()),
__TestStructDiffEnumRef::test3(v) => __TestStructDiffEnum::test3(v.clone()),
__TestStructDiffEnumRef::test4(v) => __TestStructDiffEnum::test4(v.clone()),
__TestStructDiffEnumRef::test5(v) => __TestStructDiffEnum::test5(v.clone()),
}
}
}
impl StructDiff for Test {
type Diff = __TestStructDiffEnum;
type DiffRef<'__diff_target> = __TestStructDiffEnumRef<'__diff_target>;
fn diff(&self, updated: &Self) -> Vec<Self::Diff> {
let mut diffs = vec![];
if self.test1 != updated.test1 {
diffs.push(Self::Diff::test1(updated.test1.clone()))
};
if self.test2 != updated.test2 {
diffs.push(Self::Diff::test2(updated.test2.clone()))
};
if self.test3 != updated.test3 {
diffs.push(Self::Diff::test3(updated.test3.clone()))
};
if self.test4 != updated.test4 {
diffs.push(Self::Diff::test4(updated.test4.clone()))
};
if self.test5 != updated.test5 {
diffs.push(Self::Diff::test5(updated.test5.clone()))
};
diffs
}
fn diff_ref<'__diff_target>(
&'__diff_target self,
updated: &'__diff_target Self,
) -> Vec<Self::DiffRef<'__diff_target>> {
let mut diffs = vec![];
if self.test1 != updated.test1 {
diffs.push(Self::DiffRef::test1(&updated.test1))
};
if self.test2 != updated.test2 {
diffs.push(Self::DiffRef::test2(&updated.test2))
};
if self.test3 != updated.test3 {
diffs.push(Self::DiffRef::test3(&updated.test3))
};
if self.test4 != updated.test4 {
diffs.push(Self::DiffRef::test4(&updated.test4))
};
if self.test5 != updated.test5 {
diffs.push(Self::DiffRef::test5(&updated.test5))
};
diffs
}
#[inline(always)]
fn apply_single(&mut self, diff: Self::Diff) {
match diff {
Self::Diff::test1(__0) => self.test1 = __0,
Self::Diff::test2(__1) => self.test2 = __1,
Self::Diff::test3(__2) => self.test3 = __2,
Self::Diff::test4(__3) => self.test4 = __3,
Self::Diff::test5(__4) => self.test5 = __4,
}
}
}
}
As I said in my requirements, I’ve been attempting to keep syn/quote/proc_macro2 out of the build tree of my core gameplay/server library, partly for the compile times and partly because I don’t like monocultures. Nanoserde has been my replacement for the serde ecosystem, and so far it’s actually been very pleasant. My clean build time for my core game library is <6s, and incremental builds are <1s. Nanoserde uses a custom parser as a replacement for syn (which has been unpopular ), which compiles with its proc-macros in ~1s. It doesn’t parse literally everything in rust, but it’s a significant subset, and I’ve manually implemented the (de)ser traits in the few places where it doesn’t do what I want. When I realized that none of the delta-compression crates fit what I was looking for, lifting the nanoserde parser was an obvious first step (ty open source).
I’d never really written or interacted with a parser at before, so one month-long training montage later, I finally had something that could parse most rust struct definitions (on stable). Most of the heavy lifting in this crate is done in the proc macro (the trait implementation is pretty straightforward), so at this point I just had to figure out how to handle all of the bounds introduced by different features in the crate. Optional bounds that are introduced by different features include Debug, serde::Serialize/serde::DeserializeOwned/serde::Deserialize, and nanoserde::DeBin/nanoserde::SerBin, and each of the generated types needs to be set up appropriately to handle them.
As of the most recent release (v0.6.0, which added the difference-by-reference functionality) I’m finally happy with the API of the crate, and am planning to start diving a bit deeper into performance of diffs between collections, both ordered and unordered. There’s some basic support for diffing collections of hashable objects, but I’m unimpressed with the performance of the methods so far and plan to explore some Levenshtein-distance implementations at some point to try and improve things.
One bright spot on the performance front, however, is the generated_setters feature of the crate. To use it, you just have to add the #[difference(setters)] attribute on a struct you derive Difference on, and a setter will be generated for each field. This setter returns an Option<<Self as StructDiff>::Diff>, allowing you to modify data structures and record the differences to send simultaneously. Using the same example struct as before, the following code is generated when this feature is enabled:
impl Test {
/// Setter generated by StructDiff. Use to set the test1 field and generate a diff if necessary
pub fn set_test1_with_diff(&mut self, value: i32) -> Option<<Self as StructDiff>::Diff> {
if self.test1 == value {
return None;
};
let diff = <Self as StructDiff>::Diff::test1(value.clone());
self.test1 = value;
return Some(diff);
}
// ... repeat for other fields
}
…which (because Option is IntoIterator) means that the following code can be used when working with objects that you know will need to be synced:
let mut player = Test::default();
let mut diffs = vec![];
diffs.extend(player.set_test1_with_diff(1));
assert_eq(diffs.len(), 1);
// diffs now contains the changed field
You can also rename the setter using the #[difference(setter_name = {})] field attribute to avoid collisions. There’s probably room to optimize this more, but honestly at this point it’s fast enough that I haven’t noticed it.
That was a lot more words than I expected, but basically StructDiff is an attempt (for my specific inclinations) to strike a balance between minimal deps, reasonable performance (both run- and compile-time), and reduced network usage, all with as usable/simple of an API as I can come up with. I wouldn’t go so far as to call it battle-tested, but I’ve been using it in my gamedev project for quite a while now and I am definitely much more limited by my ability to understand linear algebra than I have been by problems with the crate. If you try it out and encounter issues, please let me know! I’m planning to maintain this for the forseeable future, so opening PRs and filing issues is greatly appreciated.
]]>
0.2
